SQData Enterprise Information Management Product Version 1 Release 3 Executive Briefing

Topics  Objectives  SQData Overview  SQData Components  SQData in Action  Q&A

Briefing Objectives  Provide a high level overview of SQData V1R3  Describe the SQData components  Illustrate some SQData deployment scenarios  Address Questions

SQData – Company Background  Founded in 1999  Mission: To provide our customers with outstanding technology and services in an efficient and cost effective manner  Focus: Integrating legacy data with newer technologies Growth through channel partnerships Outperform the competition (functionality and CPU cycles)  Re-Marketing Agreement with IBM Middleware offering (Santa Teresa Lab) IMS near-real-time changed data capture Integration with MQSeries Multi-Platform full data refresh

What is SQData?  Multi-Purpose Data Management Utility High Performance MQSeries Enabled Relational & Non-Relational Databases/Files Cross Platform Operation  What SQData Has/Can Been Used For: Bulk Data Extraction and Transformation Near-Real-Time Changed Data Propagation – IMS and DB2 VSAM Changed Data Propagation (asynchronous) Integration with 3 rd Party COTS Packages Lookups to External Data Sources XML Message Formatting Workload Generation for Testing Data Cleansing & Filtering

SQData Evolution  June 2000 – Development Started  Version 1, Release 1 (V1R1) Available: 4Q 2000 Focus: Data Extraction and Transformation Support for Multiple Database/File Formats Data Type Translation Cross Platform Operation  Version 1, Release 2 (V1R2) Available: 3Q 2001 Focus: Near-Real-Time IMS Data Propagation Integration with MQSeries Enhanced XML Message Formatting Complex Data Structures Additional Commands & Functions

SQData Evolution…  Version 1, Release 3 (V1R3) Available: 1Q 2003 Focus: Script Encapsulation through Mapping Procedures (PROCS) Process Monitor for Multiplatforms Product Normalization for Repeating Groups Smart Apply’ for Propagating Data to Target Relational Databases IMS Change Data Capture Enhancements (auto-commit, exclude pgm) Additional Commands, Functions and Performance Enhancements  Version 2, Release 1 (V2R1) Planned Availability: 4Q 2003 Focus: Integrated Graphical User Interface (GUI) Subsystem Framework (MQ Broker) Support for Additional Non-Relational Databases Tighter Integration with COTS Packages (Peoplesoft, SAP, etc.)

Capabilities: Supported Platforms/Databases  SQData for OS/390 and z/OS Mainframe IMS V6 and Higher DB2 V6 and Higher MQSeries V5.2 and Higher VSAM Flat Files  SQData for Multiplatforms UNIX, Linux, Windows 2000/XP Oracle 8.1 and Higher (native support) DB2 UDB V7 and Higher (native support) SQL Server, MS Access and other Relational Databases via ODBC AS/400 and IBM iSeries Servers

Capabilities: Data Formatting and Transport  Supported Data Formats XML Delimited Binary (COBOL copybooks, C headers) Relational Tables IMS Segments DB2 Load Utility Format  Supported Transport Mechanisms Websphere MQ Messaging (MQSeries) TCP/IP

Topics  Objectives  SQData Overview  SQData Components  SQData in Action  Q&A

Terminology  Datastore – Any Object that Contains Data Databases Files MQ Queues  Structure/Description – Defines the Format of Data  Script – Contains SQData Commands and Functions Data Structure Definitions Data Store Descriptions Mapping Scripts  Commands – Direct SQData Operation  Functions – Augment SQData Commands Data Transformation Conditional Logic Table/Segment Lookups

Terminology…  Queue – An MQSeries Queue  CDC – Change data capture (IMS or DB2)  Source Datastore – an input database, file or queue  Target Datastore – an output database, file or queue  Field/column – used interchangeably  Parser – the SQData Parser component  Engine – the SQData Integration Engine component  Distributed – Server based platforms (UNIX, Windows)

SQData Product Architecture Parsed Command Script Source MQ Queues Source Databases & Files Target MQ Queues Lookup Databases & Files Target Databases & Files SQData Parser SQData Integration Engine SQData Command Script

SQData Components  Command Script (Script) Contains the Definitions, Commands and Functions that Control the Operation of the SQData Integration Engine May Contain Imbedded Scripts (Datastores, Propagation, Lookups, etc.)  Parser SQDPARSE (program name) Prepares/Compiles the Command Script for Execution by the SQData Integration Engine  Integration Engine SQDATA (program name) Execution Component Controlled by the Contents of a Parsed Command Script

SQData Utilities  SQDCOPYQ – Copy to and from MQ Queue Copies the Contents of an MQ Queue to a File Copies the Contents of a File to an MQ Queue  SQDEMPTQ – Empty MQ Queue Clears the Contents of an MQSeries Queue Terminates when Queue is Empty  SQDQTERM – Shuts down SQData Integration Engine Used if the input datastore is an MQ queue Commonly used for the SQData propagation engine started task Force or quiece option

Topics  Objectives  SQData Overview  SQData Components  SQData in Action  Q&A

Data Extraction  Highly flexible  Configuration options To/from any SQData supported operating system platform To/from any SQData supported database/file Target apply via DBMS load or through propagation framework  Transformation options Filter out unwanted data Translate data to/from any supported SQData format Include data from other sources via LOOKUP and JOIN

Data Extraction Scenarios OS/390 and z/OS UNIX SQData Integration Engine IMS Database  IMS Segment Extract – Load to Distributed DBMS Distributed DBMS FTP Load Utility Formatted Data DBMS Load Utility or IMS Unload File

Data Extraction Scenarios… OS/390 and z/OS UNIX SQData Integration Engine DB2 Table  DB2 Table Extract – Load to Distributed DBMS Distributed DBMS FTP Load Utility Formatted Data DBMS Load Utility

Data Extraction Scenarios… OS/390 and z/OS UNIX SQData Integration Engine IMS Database  Low Volume Extract – IMS or DB2 to Distributed Distributed DBMS or DB2 SQData Integration Engine Apply MQ

Data Extraction Scenarios… OS/390 and z/OS UNIX SQData Integration Engine  Flat File to XML Formatted File and Distributed DBMS Distributed DBMS Flat File SQData Integration Engine Apply MQ XML Formatted Data

SQData ‘Power Loader’  Fast Data Load of UNIX Distributed Databases Speeds the database load process by 30 to 40 percent Loads extracted data directly into a distributed database without landing the data on the UNIX instance Eliminates the need for DASD required for traditional loads  Used in Conjunction with SQData for Multiplatforms  More Enhancements in the Works with SQData V2R1 FTP as a supported datastore Provides for true straight through processing Legacy data extracts directly into the target distributed database(s) Eliminates the need for auxiliary DASD Much higher performance expected

IMS Near-Real-Time Data Propagation  Uses an IMS Segment Capture Exit Receives Changed Data from IMS Creates an SQData IMS CDC Message with a before and after image ‘Puts’ the SQData IMS CDC Message to an MQSeries Queue Maintains Commit Scope Integrity with Application IMS Online and BMPs – Two-Phase Commit DLI Batch – Single Phase Commit  SQData Exit Features Assign Certain Segments to Specific Queues Include/Exclude Specific Programs for/from Data Capture Include/Exclude Different IMS Region Types for/from Data Capture Auto-Commit for DLI Batch Programs

IMS Near-Real-Time Data Propagation…  Propagation Features/Functions SQData Engine Sends Only Changes to Mapped Fields If at least one of the mapped fields does not change, the changed data record is discarded. Obtain Certain IMS Information for Changed Segments Database Name PSB Name IMS Region Type Time of Update (STCK and Timestamp) IMS Change Operation (I – ISRT, R- REPL, D-DLET) Userid of Transaction Making Data Change Determine Whether or Not a Source Field Has Changed FLDCHG Function

IMS Near-Real-Time Data Propagation Scenario MQ IMS Databases XML Formatted Messages SQData IMS Change Data Capture Component IMS Applications SQData Integration Engine MQ Distributed Databases & Files Legacy Databases & Files Integration Broker MQ Applications

IMS Near-Real-Time Data Propagation Scenario… MQ SQData IMS Change Data Capture Component SQData Integration Engine IMS Databases IMS Applications Distributed Tables  Multi-platform IMS near-real-time data propagation OS/390 and z/OS UNIX MQ Integration Broker

IMS Near-Real-Time Data Propagation Scenario… SQData IMS Change Data Capture Component IMS Databases IMS Applications Initiates IMS Trx  Triggering a process based on the occurrence of an event MQ IMS Trigger Monitor MQ Message Triggers A Process

SQData – Sample Customer Implementation WR ITEM WR ORD DOPAD VTS Oracle DBs SQData IMS Data Capture IMS Databases IMS Applications DB2 Databases DB2 Applications SQData Runtime Engine(s) XML MQI Solaris Z/OS WFADODOPAD DB2 Trigger with DB2/MQSend SQData MQ Client DOPAD ORD WR ITEM DOPAD ORD XML SQData Runtime Engine

DB2 Near-Real-Time Propagation  Integration with IBM’s DataPropagator Relational (DPropR) Works in conjunction with the DPropR Capture component Provides high performance near-real-time DB2 changed data capture Engineered to work seamlessly with future releases of DPropR  Extends the Database Apply Options Non-relational databases/files (IMS, VSAM, etc.) Relational databases not supported by DPropR  Utilizes MQSeries (WMQI) as the Transport Method Allows for integration into an enterprise application framework Common format for all changed data  Enabled Through an SQData ‘Active Listener’ Component Watches for changes to the DPropR staging tables Immediately propagates the changes to any SQData supported database

DB2 Near-Real-Time Data Propagation…  Propagation Features/Functions Allows for complex data transformations prior to apply Provides the ability to combine (join) data from other sources SQData Engine sends only changes to mapped fields If at least one of the mapped fields does not change, the changed data record is discarded. Obtain Certain DB2 information for changed table segments Database name Table name Time of Update (STCK and Timestamp) DB2 Change Operation (I – ISRT, R- REPL, D-DLET) Userid of Transaction Making Data Change Determine Whether or Not a Source Field Has Changed FLDCHG Function

DB2 Near-Real-Time Data Propagation Scenario XML Formatted Messages SQData Integration Engine MQ DB2 Tables DB2 Applications  DB2 Propagation with IBM’s Data Propagator Relational Product Distributed Databases & Files Legacy Databases & Files Integration Broker MQ Applications DataPropagator Relational DPropR Staging Tables SQData Active Listener

‘Smart Apply’ Feature of SQData  Used for Relational Database target datastores  Ensures that only latest changes are propagated Presence of a control column (SQDSTCK) No script changes required  Eases timing restrictions on full data refresh process Continue propagating while conducting full data refresh Make sure to shut down SQData Engine (apply) during refresh

Topics  Objectives  SQData Overview  SQData Components  SQData in Action  Q&A

SQData Enterprise Information Management Product Version 1 Release 3 Executive Briefing