Creating Data Marts from COBOL Files (ISAM to RDBMS)

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada GDT ETL  What is ETL? n Extract Transform Load l Extract data from your legacy system. l Transform (Convert or cleanse). l Load the data into the relational database, data warehouse, data mart or Operational Data Store.  Why GDT-ETL? n Built for Legacy Systems. n Native access to ISAM data. n Transformation from COBOL data types to more general types. n Allows you to focus on where the data is going not how you get access to it.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada GDT ETL  Data flow for a system running GDT n MF file system to a relational database MF Files GDT RDMS GDT-ETL RDMS

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada GDT ETL  ETL Features n Command line tool and Graphical User Interface (ETLPortal) n Comprehensive error logging and process monitoring. n Will work with existing enterprise scheduling facilities.  ETL Requirements n Windows l GDT V4.1 or greater l.NET Framework 2.0

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada PLUS FILE DEFINITION FILE COBOL Select statement Multi 01 Record Definitions SQL1GENSQL1GEN Creates GDTSQL01.DLL Using the ORDERS file SQL1GEN process SQLCONV.PAR

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada SQL1GEN process  GDT process that generates a File Access Component (GDTSQL01.DLL) n To be used by GDT ETL to generically access all Micro Focus ISAM files n Configuration l SQLCONV.PAR l FILE DEFINITION(S) fd’s l FILE SELECT STATEMENTS l MULTIPLE 01 RECORD DEFINITION PARAMETER FILES n Requires GDT V4.1  n GDTSQL01.DLL contains all necessary file info. l Key information, data types, record types, fields etc..

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada The ETL process GDT_ETL GDTSQL01 DLL Order Info. File Access Component – Found via the GDTSYS.PAR SQL Setting ETLCONFIG.XML – Configuration for ETL to find / access FAC and DB

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada GDT_ETL Command Line Control  [-v | -version] n Displays the programs build version. No other commands will be executed after this one.  [-c | --createobj] n Will create the necessary database objects. is the flat file that will contain all of the SQL that would have been used to create the objects. Default is no value which will load everything into the database  [-l | --loaddata] n Will load the data into the database. is an integer representing how often GDT-ETL will report the status of the load. [DEFAULT = 0]

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada GDT_ETL Command Line Control  [-r | --record] n Will only execute the given commands [-lDdcf] when it finds the record that matches the name given in. If used with [-i] it will limit the command for the found in.  [-i | --isamfile] n Will only execute the given commands [-lDdcf] when it finds the ISAM file that matches Can be used to limit the [-r] to a specific record.  [-D | --delobj] n Will delete all objects in the database. Limit using [-ri] commands. Do not use at the same time as [-d].  [-d | --deldata] n Will delete all data in the database NOT the objects. Limit using the [-ri] commands. Do not use at the same time as [-D].

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada GDT_ETL Command Line Control  [-f | --filelayout] n Will display the layout of the file(s) in the file access component. [-t | --dbtype] n Will display the DB type defined in the XML config file.  [-T | --testload] n Will test GDT_ETL to make sure it will process the file access component properly before attempting any action against the database.  [-? | --help] n Short help usage message will be displayed by [-?], verbose use [--help]

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada Data is in the database via the ETL Load Results of the ETL process

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETL in a nutshell Hear from the Chief Engineer  Build the tables and the relationships in SQL n Each record denotes a new table. n Each sub record denotes a new table with a 1 to many relationship with the parent record. l Nested OCCURS within a file produces more relationships. n Each field produces a field in a SQL table and ETL maps COBOL data types to the SQL Servers data types.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETL in a nutshell Hear from the Chief Engineer  Optional l Modify the table schema and stored procedures to suit your needs.  Build transformations as needed.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETL in a nutshell Hear from the Chief Engineer  How does ETL build database schemas? n When Generating a table schema GDT-ETL makes the following assumptions. l Each file consists of 1 or more record types. l Each record in a file consists of 1 or more fields and it may contain hierarchy of sub records as denoted by the OCCURS clause. l Each field has a location in a record as well as an associated COBOL data type. l Group name and redefines are ignored.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETL in a nutshell Hear from the Chief Engineer  Create the database schema and stored procedures. n Build the tables and relationships n Three basic data type transformations: l PIC X(NN) to char(NN) l PIC 9(NN) to INT l PIC S9(NN)V9(NN) to double n Setup default stored procedures for each table: l For each table there are 3 separate stored procedures  Insert  Update  Delete l The tables can be modified but the stored procedures must reflect any schema modifications. l The store procedures can be modified but the parameters must stay the same.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETL in a nutshell Hear from the Chief Engineer  Build the default database objects n Setup the ETL Configuration. l Set database properties. l Reference the location of the file access component. n Build the database objects.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETL in a nutshell Hear from the Chief Engineer  What value do transformations add? n Allows for cleansing of data before it enters the database. l Determine if it is accurate. l Avoid data corruption. n Create more interesting data transformation l Example: Date time

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETL in a nutshell Hear from the Chief Engineer  Use Data Integration tools n BusinessObjects Data Integrator.  Develop the transformations by hand. l Modify the table layout if desired. l Modify the default stored procedures to setup desired transformations.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETL Reference Guides  Installation, Configuration and Operations Guide  ETL Portal User Guide  Command Line HELP  ETL Portal HELP

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal Setup for SQL1GEN process. All of the pieces that we talked about when processing via the Command Interface.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal Selection of Files via the SQLCONV.PAR file. By selecting the Disabled box prevents that file from being processed into the GDTSQL01.DLL. (like commenting it out in the SQLCONV.PAR file).

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal Running the SQL1GEN Process

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal ETL Command Selection.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal Create Objects is selected. This will create the tables and stored procedures. Filtering by File and / or Record is possible. Force ILLDEC recovery is checked to allow ETL to recover from Illegal Decimal.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal ETL create is successful!

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal Viewing the TABLES produced under SQL Server Manager.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal Viewing the Stored Procedures produced under SQL Server Manager.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal Viewing the Stored Procedures produced under SQL Server Manager.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal Load Data is selected. This will load the tables by processing the stored procedures.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal ETL Load is successful!

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal ETL output window shows issues with the data.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal Viewing the Table contents under SQL Server Manager

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal A look at the Insert stored procedure for the PRODUCT_REC table. As is from the creation by ETL.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal A look at the Update stored procedure for the PRODUCT_REC table. As is from the creation by ETL.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada ETLPortal A look at the Delete stored procedure for the PRODUCT_REC table. As is from the creation by ETL.

Creating Data Marts from COBOL Files Doug Evans / Chris Martin GDT 2008 International User Conference August 10 – 13  Lake Las Vegas, Nevada GDT-ETL  THE DEMONSTRATION! and Questions?