This page (revision-31) was last changed on 26-Nov-2021 10:22 by Kevin Higgs

This page was created on 26-Nov-2021 10:22 by JEscott

Only authorized users are allowed to rename pages.

Only authorized users are allowed to delete pages.

Page revision history

Version Date Modified Size Author Changes ... Change note
31 26-Nov-2021 10:22 13 KB Kevin Higgs to previous
30 26-Nov-2021 10:22 14 KB Karen Parrott to previous | to last
29 26-Nov-2021 10:22 14 KB Karen Parrott to previous | to last
28 26-Nov-2021 10:22 13 KB Kevin Higgs to previous | to last
27 26-Nov-2021 10:22 13 KB kparrott to previous | to last
26 26-Nov-2021 10:22 14 KB kparrott to previous | to last
25 26-Nov-2021 10:22 13 KB kparrott to previous | to last
24 26-Nov-2021 10:22 13 KB kparrott to previous | to last
23 26-Nov-2021 10:22 13 KB kparrott to previous | to last
22 26-Nov-2021 10:22 13 KB kparrott to previous | to last
21 26-Nov-2021 10:22 12 KB kparrott to previous | to last

Page References

Incoming links Outgoing links
LMTD
LMTD

Version management

Difference between version and

At line 40 added 43 lines
!Load External Data with an Interface
The technology used by LMTD when an Interface format is used is called Data Transfer Facility (DTF). This is a sophisticated program, which enables you to transfer data between different databases by extracting data from one database and loading it into another. The DTF is fully capable of multiple table load/extracts and foreign key resolution.
The DTF is comprised of two main components: the Data Extractor and the Data Loader. The Data Extractor, as the name suggests, extracts table data based on a pre-defined extract configuration from a database and writes the data to either a binary or human-readable XML file. The Data Loader processes these extracted files and loads the table data by inserting or updating records into the appropriate tables in the database.
The DIF Data Loader uses the Data Loader component of the DTF in order to take advantage of its sophisticated multiple table load/extract and foreign key resolution capabilities. The DIF Data Loader emulates the functionality of the DTF Data Extractor. Instead of extracting table data from a database, the DIF Data Loader extracts table data, based on an interface format, from a source file and passes the data onto the DTF Data Loader to be processed.
The following will review the necessary steps required to use LMTD to load data using a source file based on an interface format.
;Step 1 - Define Interface Format in IDIF
The source file layout needs to be defined as an interface format. The interface format is created and modified in the IDIF screen. Each line of the source file is read and divided into field values based on the field information table in this screen.
For this method of LMTD, only the 'Rec#', 'Field#', 'Name', 'Beg Posn' and 'End Posn' columns are used. The 'Name' column of the table specifies the table and column (or foreign key column) while the 'Beg Posn' and 'End Posn' columns determine which characters of the line in the file represent the field value.
[LMTD INTERFACE FORMAT]
;Step 2 - Define Site Preferences in [IMST]
;:Within the IMST screen, the administrator must define where the Extract and Load files are located and where the load functions are to write the exception files to. These locations are used by the functions to create and retrieve information. The customer can modify these values to represent any directory on their system, which is convenient.
;:The extract directory is defined using the [EXTRCT DATA FLDR] preference. The load directory is defined using the [LOAD DATA FLDR] preference. The [ATTACHMENTS FLDR] must be specified so the load functions know where to write the report exceptions log to.
;:If the full path is not specified here, then the path used by the HL Application Directory will apply. The user who is running the LMTD must have Read/Write/Create access to this directory on the server.
;Step 3 - Run LMTD to load the file
;Step 4 - Review Report and Exception File for Results
;:During the load process, all non-fatal exceptions as well as the number of insertions and updates (by table) are logged in the execution run logs. After the load process completes, a report is written to a text file indicating the number of records read, number of records bypassed, number of records processed, number of records that were bypassed because an identical one already existed in the table, and number of records updated/inserted by table. The report will display all non-fatal business exceptions encountered during the load with a record number indicating for which record the exception occurred. The report will also display the amount of time that was taken for the load process to complete.
;:There are 2 tiers of exceptions that can be generated during the load process. The first tier involves extracting and building data from the source file and the second involves processing the extracted data. However, there will only be ONE exception for each failed load. Basically, if the record fails to build data correctly, the program will not bother loading it so you will never see 2 exceptions for a single failed record load.
;:Exceptions generated while extracting and building data from the source file will indicate that the exception occurred while "reading data from file". Exceptions generated while processing the extracted data will indicate that the exception occurred while "loading data into the DB".
;:When a Null Pointer Exception (NPE) occurs during the load process, the entire stack trace to the NPE is captured and displayed as part of the exception message in the load report.
;:The report text file will be located in the directory which was specified in the [IMST] preference [LOAD DATA FLDR].
;Step 5 - Verify Data has been loaded
;:If there are no exceptions in the report, you should navigate to the table(s) where the data was loaded and verify that the load was done correctly.
This page (revision-31) was last changed on 26-Nov-2021 10:22 by Kevin HiggsTop