!!!LOAD EXTERNAL DATA
!!Processing Information

There are two methods of loading external data using the LMTD function; Table Data Loader logic (no interface) and DIF Data Loader logic (with interface). 

!Loading External Data without an Interface
The technology used by LMTD when an interface format has not been defined is known as Table Data Loader. This technology reads table data from a comma-delimited text file and inserts or updates valid records in the file into the destination table in the database.

To run the Table Data Loader logic using LMTD, values for the source file name, destination table and character delimiter should be provided. The interface code field should be left blank. When loading simple, single tables without the need for Foreign Key resolution, this method is recommended.

The drawback of not using an interface format method is that the Table Data Loader technology is not able to transfer data from one database to another as conflicting ID values cause foreign keys to not be found. If foreign keys are involved in the data you wish to load, it is recommended to use the LMTD interface format method instead. This method is described below.

The following will review the necessary set up and process flow for running the LMTD to load data using a source file to load directly into a table.

;Step 1 - Create File
;:The first line of the file must contain the header of column names that identifies what is to be loaded. The order of data within the file must match the order in the header record. The column names can be in lower or upper case.

;Step 2 - Define Site Preferences in [IMST]
;:Within the IMST screen, the administrator must define where the Extract and Load files are located and where the load functions are to write the exception files to. These locations are used by the functions to create and retrieve information. The customer can modify these values to represent any directory on their system, which is convenient. 

;:The extract directory is defined using the [EXTRCT DATA FLDR] preference. The load directory is defined using the [LOAD DATA FLDR] preference. The [ATTACHMENTS FLDR] must be specified so the load functions know where to write the report exceptions log to.

;:If the full path is not specified here, then the path used by the HL Application Directory will apply. The user who is running the LMTD must have Read/Write/Create access to this directory on the server.

;Step 3 - Run LMTD to load the file

;Step 4 - Review Report and Exception File for Results
;:During the load process, all non-fatal exceptions as well as the number of insertions and updates (by table) are logged in the execution run logs. After the load process completes, a report is written to a text file indicating the number of records read, number of records bypassed, number of records processed, number of records that were bypassed because an identical one already existed in the table, and number of records updated/inserted by table. The report will display all non-fatal business exceptions encountered during the load with a record number indicating for which record the exception occurred. The report will also display the amount of time that was taken for the load process to complete.

;:There are 2 tiers of exceptions that can be generated during the load process. The first tier involves extracting and building data from the source file and the second involves processing the extracted data. However, there will only be ONE exception for each failed load. Basically, if the record fails to build data correctly, the program will not bother loading it so you will never see 2 exceptions for a single failed record load.

;:Exceptions generated while extracting and building data from the source file will indicate that the exception occurred while "reading data from file". Exceptions generated while processing the extracted data will indicate that the exception occurred while "loading data into the DB".

;:When a Null Pointer Exception (NPE) occurs during the load process, the entire stack trace to the NPE is captured and displayed as part of the exception message in the load report.

;:The report text file will be located in the directory which was specified in the [IMST] preference [LOAD DATA FLDR].

;Step 5 - Verify Data has been loaded
;:If there are no exceptions in the report, you should navigate to the table where the data was loaded and verify that the load was done correctly.


!Load External Data with an Interface
The technology used by LMTD when an Interface format is used is called Data Transfer Facility (DTF). This is a sophisticated program, which enables you to transfer data between different databases by extracting data from one database and loading it into another. The DTF is fully capable of multiple table load/extracts and foreign key resolution.

The DTF is comprised of two main components: the Data Extractor and the Data Loader. The Data Extractor, as the name suggests, extracts table data based on a pre-defined extract configuration from a database and writes the data to either a binary or human-readable XML file. The Data Loader processes these extracted files and loads the table data by inserting or updating records into the appropriate tables in the database.

The DIF Data Loader uses the Data Loader component of the DTF in order to take advantage of its sophisticated multiple table load/extract and foreign key resolution capabilities. The DIF Data Loader emulates the functionality of the DTF Data Extractor. Instead of extracting table data from a database, the DIF Data Loader extracts table data, based on an interface format, from a source file and passes the data onto the DTF Data Loader to be processed.

The following will review the necessary steps required to use LMTD to load data using a source file based on an interface format.

;Step 1 - Define Interface Format in IDIF
The source file layout needs to be defined as an interface format. The interface format is created and modified in the IDIF screen. Each line of the source file is read and divided into field values based on the field information table in this screen.

For this method of LMTD, only the 'Rec#', 'Field#', 'Name', 'Beg Posn' and 'End Posn' columns are used. The 'Name' column of the table specifies the table and column (or foreign key column) while the 'Beg Posn' and 'End Posn' columns determine which characters of the line in the file represent the field value.

[LMTD INTERFACE FORMAT]

;Step 2 - Define Site Preferences in [IMST]
;:Within the IMST screen, the administrator must define where the Extract and Load files are located and where the load functions are to write the exception files to. These locations are used by the functions to create and retrieve information. The customer can modify these values to represent any directory on their system, which is convenient. 

;:The extract directory is defined using the [EXTRCT DATA FLDR] preference. The load directory is defined using the [LOAD DATA FLDR] preference. The [ATTACHMENTS FLDR] must be specified so the load functions know where to write the report exceptions log to.

;:If the full path is not specified here, then the path used by the HL Application Directory will apply. The user who is running the LMTD must have Read/Write/Create access to this directory on the server.

;Step 3 - Run LMTD to load the file

;Step 4 - Review Report and Exception File for Results
;:During the load process, all non-fatal exceptions as well as the number of insertions and updates (by table) are logged in the execution run logs. After the load process completes, a report is written to a text file indicating the number of records read, number of records bypassed, number of records processed, number of records that were bypassed because an identical one already existed in the table, and number of records updated/inserted by table. The report will display all non-fatal business exceptions encountered during the load with a record number indicating for which record the exception occurred. The report will also display the amount of time that was taken for the load process to complete.

;:There are 2 tiers of exceptions that can be generated during the load process. The first tier involves extracting and building data from the source file and the second involves processing the extracted data. However, there will only be ONE exception for each failed load. Basically, if the record fails to build data correctly, the program will not bother loading it so you will never see 2 exceptions for a single failed record load.

;:Exceptions generated while extracting and building data from the source file will indicate that the exception occurred while "reading data from file". Exceptions generated while processing the extracted data will indicate that the exception occurred while "loading data into the DB".

;:When a Null Pointer Exception (NPE) occurs during the load process, the entire stack trace to the NPE is captured and displayed as part of the exception message in the load report.

;:The report text file will be located in the directory which was specified in the [IMST] preference [LOAD DATA FLDR].

;Step 5 - Verify Data has been loaded
;:If there are no exceptions in the report, you should navigate to the table(s) where the data was loaded and verify that the load was done correctly.



----

!!Report Parameters 

||Report Parameters||
|Server File| This field provides the name of the file that has been placed on the server in the \\directory specified in the [IMST] record [LOAD DATA FLDR].
|Local File| This field allows you to upload a local source file for loading into the database \\(similar to that of WW Reports in EP). To do this, you must populate the 'Local File'\\ field by selecting the local source file using the file chooser. When selecting a local\\source file to load, the 'Server File' field MUST be blank.
|Allow Update|Lexicon value of YES or NO\\ Selecting "NO" disallows the updating of existing records in the database. If there \\are new records in the load file, the program will still perform an insertion of these \\records into the destination table. In short, selecting "NO" for Allow Update does \\not allow you to run the program in trial as it only prevents existing records in the\\database from being updated. \\Selecting "YES" will update existing records in the database with changes found in \\the load file.
|Allow Insert| Lexicon value of YES or NO\\By selecting "NO", LMTD will not load any new records into the specified table.\\By selecting "YES", new records found in the load file will be inserted into the\\specified table.
|Exception Level| Lexicon value of 0 to 9. \\This field allows you to specify the level of trace you wish to see in the report.
|Interface Code| If using the Table Data Loader logic, leave this blank. \\If using the DIF Data Loader logic an interface must be specified to load a source file.\\You can select multiple interfaces if you have to run a particular source file multiple\\times using different interfaces. If the source file has thousands of records\\suggested to run the LMTD multiple times selecting only one interface code each\\time otherwise there may be an impact on performance.
|File Format| If using the Table Data Loader the field defaults to Character-delimited as the Table\\ Data Loader method only supports Character-delimited files. \\If using the DIF Data Loader the user may select either Character-delimited or Fixed Position.
|Destination Table| This field identifies the table that will be inserted or updated. This is a mandatory \\parameter for the Table Data Loader method.\\When loading a file using the DIF Data Loader (interface format) the destination \\table MUST be left BLANK.
|Delimiter |Indicates the delimiter in the file if it is not a Fixed Format.