Java/XML ETL Engine By Bob Timlin

Outline Data Extraction, Transformation, and Loading (ETL). Java & XML Meta-Data Mapping Data from Source to Target

Outline Proposed XML Usage. XML for Meta-Data Challenges/Issues Sample XML Data File Sample XML Meta-Data File

Extract/Transform/Load (ETL): The process of getting data from the source system(s) into the data- warehouse is easily 80% of the effort of the entire data-warehouse. This is because of the complexity of the source systems, the cleansing or transformation process, and all of the prep work to get the detail operational data into summary data-warehouse data. The more the source systems you have the harder this process is and this increases exponentially. Cleaning/Transforming the data is probably the most complicated part of this process. Transformations can either be done on the source system or the target system.

Java & XML Currently ETL processes are mostly written in Cobol and C with Embedded SQL. There are many GUI tools out there to streamline this process. These tools mostly generate proprietary code that is then executed by an scheduling program. All of the big vendors in this field are pushing XML as a language to store transformation meta-data and all of the big plays, sans Microsoft, are backing Java as the language to implement transformations. For some weird reason Microsoft doesn’t seem to like Java. The Major Vendors include: IBM, Oracle, and Microsoft.

Meta-Data Data about data. In terms of data warehouse it stores information about the structures of both source and destination data and how to extract, transform, and load data. It may also maintain network configuration information like ip-addresses and ports. The meta-data coalition recently merged with Object Management Group (OMG) They are backed by many heavy-hitters including Oracle, IBM, and Microsoft. The industry seems to be moving towards using XML for storing meta-data. This makes the meta-data very standardized and portable.

Mapping Data from Source to Target: Target: Name: The name of the logical table in the data-warehouse. Source: table name in the xml data file. Driver: JDBC driver name Url: Path to the data-warehouse. Username: username to connect to the data-warehouse Password: password to connect to the data-warehouse

Mapping (continued) Column: Name: The name of the logical column in the dw. Type: The data type of the logical column in the data warehouse. Key: Is this a primary key, if so the engine will use it in the where clause. Source: The name of the column in the xml data file

Proposed XML Usage For meta-data about the ETL processing. This will contain all information about mapping source to target, including transformation rules. As a data-file to store data from database’s.

XML for Meta-Data The specification is designed to be flexible enough to support many protocols, however for our project we will only implement two protocols. 1. XML Data File, 2. JDBC The Protocol will be part of the url attribute of the target or source node. Every transformation will have a source and target. … <target driver="oracle.jdbc.driver.OracleDriver" username=“scott" password=“tiger" name="srctest">

The basic construct of a XML meta-data file is: [ ] [ [ ] ] [ ] [ [ ] ]

Challenges/Issues Mapping multiple sources to multiple targets. Transformations can involve very complex coding. Especially eliminating duplicates, merging, and purging of data. These transformations usually involve “fuzzy” logic.

<target driver="oracle.jdbc.driver.OracleDriver" username=“scott" password=“tiger" name="srctest"> <! As the target, connect to the database using JDBC and Insert the data from the source XML file and rules that follow> source.replace("'", "") INITCAP(SUBSTR(source, 1, INSTR(source, ',') -1))

source.replace("'", "") INITCAP(SUBSTR(source, INSTR(source, ',') +1)) TO_DATE(source, 'DD/MM/YYYY')

<source driver="oracle.jdbc.driver.OracleDriver" username=“scott" password=“tiger" name=“targetTest"> source.replace("'", "") INITCAP(SUBSTR(source, 1, INSTR(source, ',') -1))

source.replace("'", "") INITCAP(SUBSTR(source, INSTR(source, ',') +1)) TO_DATE(source, 'DD/MM/YYYY')

data for column 1 data for column 2 data for column 1 data for column 2 data for column 1 data for column 2 Sample XML Data File

FRFSFEM1E

1 S 17 EXCEPT FOR TRAUMA]]>

Sample XML Meta-Data <target name="admits" source="patient" driver="org.gjt.mm.mysql.Driver" url="jdbc:mysql://localhost:3306/test" username="test" password="">

Thank You