2. Converting COBOL Data to SQL Data: GDT-ETL Part 1

3. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California GDT ETL  What is ETL? n Extract Transform Load l Extract data from your legacy system. l Transform (Convert or cleanse). l Load the data into the relational database, data warehouse, data mart or Operational Data Store.  Why GDT-ETL? n Built for Legacy Systems. n Native access to ISAM data. n Transformation from COBOL data types to more general types. n Allows you to focus on where the data is going not how you get access to it.

4. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California GDT ETL  Data flow for a system running GDT n MF file system to a relational database MF Files GDT RDMS GDT-ETL RDMS

5. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California  Data flow for Legacy Systems MF Files GDT GDT-ETL GCOS Fileconv2 UFA S RDMS GDT ETL Fileconv2 GDT-ETL RDMS

6. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California GDT ETL  ETL Features n Command line tool and now with a Graphical User Interface (to be shown) n Object Oriented Design. n Will work on Windows or UNIX platforms. n Comprehensive error logging and process monitoring. n Will work with existing enterprise scheduling facilities.  ETL Requirements n Windows l GDT V4.1 l.NET Framework 2.0 n Solaris l GDT V4.1

7. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California What can I do with GDT-ETL?  There are two main features of this tool. n Help build database schemas from your ISAM data. l Creating Data marts l Data warehouse l Data Stores n Loading data into a database system.

8. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California GDT ETL A Working Example! ICOM system

9. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California SQL1GEN process  GDT process that generates a File Access Component (GDTFAC.DLL) n To be used by GDT ETL to generically access all Micro Focus ISAM files n Configuration l SQLCONV.PAR l FILE DEFINITION(S) fd’s l FILE SELECT STATEMENTS l MULTIPLE 01 RECORD DEFINITION PARAMETER FILES n Requires GDT V4.1  n GDTFAC.DLL contains all necessary file info. l Key information, data types, record types, fields etc..

10. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California A Typical Production Environment ISAM Order Master File Report Generation Customer Data Batch process Report Data

11. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California PLUS FILE DEFINITION FILE COBOL Select statement GDTSQL Parameter File ------------------------------------------------------------------------------------------- SQL1GENSQL1GEN Creates GDTFAC.DLL Using the ORDERS file

12. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California The ETL process GDTETLGDTETL GDTFAC DLL Order Info. PLUS ------------------------------------------------------------------------------------------- Create ORDER Master Database Table ETL PARAMETER FILE File Access Component Load ORDER Master Database Table Delete ORDER Master Database Table

13. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California Data is in the database via the ETL Load Results of the ETL process

14. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California ETL in a nutshell  Step 1: File Analysis n Using the File Access Component built by GDT GDTFAC.DLL, the ETL tool first analyzes the file layout for each file.

15. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California ETL in a nutshell  Step 2 - Build the tables and the relationships in SQL n Each record denotes a new table. n Each sub record denotes a new table with a 1 to many relationship with the parent record. l Nested OCCURS within a file produces more relationships. n Each field produces a field in a SQL table and ETL maps COBOL data types to the SQL Servers data types.

16. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California ETL in a nutshell  Step 3: n Optional l Modify the table schema and stored procedures to suit your needs.  Build transformations as needed.  Step 4: n Load the Data l Read each file, extract the data, transform from COBOL data types to database server data types and load the data into the database.

17. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California GDT ETL IN ACTION  How does ETL build database schemas? n When Generating a table schema GDT-ETL makes the following assumptions. l Each file consists of 1 or more record types. l Each record in a file consists of 1 or more fields and it may contain hierarchy of sub records as denoted by the OCCURS clause. l Each field has a location in a record as well as an associated COBOL data type. l Group name and redefines are ignored.

18. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California GDT ETL IN ACTION  Create the database schema and stored procedures. n Build the tables and relationships n Three basic data type transformations: l PIC X(NN) to char(NN) l PIC 9(NN) to INT l PIC S9(NN)V9(NN) to double n Setup default stored procedures for each table: l For each table there are 3 separate stored procedures  Insert  Update (ODS Only)  Delete (ODS Only) l The tables can be modified but the stored procedures must reflect any schema modifications. l The store procedures can be modified but the parameters must stay the same.

19. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California GDT ETL IN ACTION  Build the default database objects n Setup the ETL Configuration. l Set database properties. l Reference the location of the file access component. n Build the database objects.

20. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California Building Transformations  What value do transformations add? n Allows for cleansing of data before it enters the database. l Determine if it is accurate. l Avoid data corruption. n Create more interesting data transformation l Example: Date time

21. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California Building Transformations  Use Data Integration tools n BusinessObjects Data Integrator.  Develop the transformations by hand. l Modify the table layout if desired. l Modify the default stored procedures to setup desired transformations.

22. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California ETL in a nutshell  ETL Configuration Utility - GUI l Set the database properties. l Modify which files you want to load. l AND MORE TO COME…..

23. Converting COBOL Data to SQL Data: GDT-ETL Part 1 Doug Evans GDT 2006 International User Conference: Evolving the Legacy – Revolutions June 25 - 28  Palm Springs, California GDT-ETL  THE DEMONSTRATION!