ioNode A Framework for Interoperability Shell Project - ETL Activities October 15 th 2003 John Bigerstaff Wynne Rees

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ETL Solutions Ltd Overview MBO of Prismtech April 2002 Data Transformation Team and Product Based in Blaenau Ffestiniog Wales, UK 10 member team (8 technical) Sales and Marketing Office in North East Standards based solutions POSC, PPDM, W3C standards (XML, XSLT) Product and services company >$4m investment in development of Extract Transform Load (ETL) tools Principal market is large enterprises in Finance, Oil and Gas, Education and Health...

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ETL Solutions Strengths ETL product, Transformation Manager (TM) System Integrators of Messaging and ETL solutions Highly qualified and experienced 10 member company with IT and Transformation Skills Requirements Capture and Analysis Data Modelling using EXPRESS, UML, Many years of Java and C++ Programming RDBMS design, development and tuning Extensive knowledge of W3C standards XML, XSD, XSLT Project Management Transformation Manager ETL Tool Meta Data Code Generator Java components for easy deployment in client architectures

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioNode A framework for interoperability to meet the interoperability challenges of today Flexible Hardware and Software configuration Developed in Java to run anywhere Software based on Open Source middleware software Low cost transformation component Standards based SOAP [Simple Object Access Protocol] XML [eXtensible Mark up Language] ebXML [electronic business XML] JMS [Java Messaging Service]

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioNode Software

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ETL Role in ioNode Responsible for the design and implementation of ioNode software components: ioAgent – messaging component ioHub – message routing ioTransform – data transformation ioDB – XML persistent data store Utilise Transformation Manager for design of transforms For SHELL project CSV to IMS-LIP LSC to HESA code lookup and vice versa

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Shell Hub and ioAgents

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Scalability of ioNode and SHELL Architecture

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. SHELL Data Flow and Transforms

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Messaging Requirements Once and once only message delivery ebXML uniquely identified messages are acknowledged upon delivery checks for duplication are carried out Send and re-send ebXML messages not successfully acknowledged are re-sent a fixed number of times Persistent storage Provides a record of the message exchanges and allows messages to be re-sent if required

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioAgent at College Messages posted using one of the following: Direct to outbound message queue A file drop directory A HTTP Post interface Send Message Respond to messages posted in outbound message queue (ionodepost). Initiate transforms that have been configured e.g. CSV to IMS-LIP and LSC to HESA code conversions Retry messages preset number of times if synchronous reply not received Receive Message Respond to messages in ionodereceive Initiate transforms that have been configured e.g. IMS-LIP to CSV and HESA to LSC to code conversions Place message in inbound message queue Persistence provided by PostgreSQL

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioAgent

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Synchronous Message Response

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioAgent – ioNodePost indexField nametypeDescription 1 indexvarchar(80) Unique identifier for entry (Primary Key) 2 rowcountbigint, Index of number of rows 3 messageIdvarchar(80) Unique message ID, set by ioNode, used for message identification 4 timestampbigint, Timestamp set when table is read by ioNode 5 status_clientvarchar(20), Flag for success of failure of message use by application 6 status_messagevarchar(20), Flag for success or failure of message delivery 7 sourcedatatypevarchar(20), Type of source data in data field eg. CSV 8 sourcedatatypeversionvarchar(20), Version of source data in data field eg. v1 9 targetdatatypevarchar(20), Type required of target data eg. LIP 10 targetdatatypeversionvarchar(20), Version required of target data eg v1 11 senders_authorisationvarchar(80), Authorization field for sender 12 function_typevarchar(80), Action type of post eg. Add 13 targetvarchar(80), Intended recipient of message eg. UoP. 14 domainrefvarchar(80), The identity of the sender eg. Cornwall 15 localidentifiervarchar(80), Security identity field for the sender 16 altdomainrefvarchar(80), Alternative identity for the sender 17 altlocalidentifervarchar(80), Alternative security field for the sender 18 databytea The data to be sent as payload eg. CSV

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. IoAgent - ioNodeReceive indexField nametypeDescription 1 rowcountbigint, Index of number of rows 2 messageIdvarchar(80) Unique message ID, set by ioNode, used for message identification 3 timestampbigint, Timestamp set when table is read by ioNode 4 message_typevarchar(80) Type of data eg. CSV 5 versionvarchar(20) Version of data eg v1 6 senders_authorisation varchar(80), Authorization field for sender 7 function_typevarchar(80), Action type of post eg. Add 8 fromTargetvarchar(80), URL of sender of message 9 domainrefvarchar(80), The identity of the sender eg. Cornwall 10 localidentifiervarchar(80), Security identity field for the sender 11 altdomainrefvarchar(80), Alternative identity for the sender 12 altlocalidentifervarchar(80), Alternative security field for the sender 13 databytea The data received

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioHub at University Implements IoAgent send and receive of messages Sends synchronous and asynchronous messages replies May act as message router Incoming messages received in Message Receive Queue (ionodereceive) Acts as router to other ioAgents if required In this case makes synchronous reply only when successfully forwarded May transform incoming messages before placing in inbound queue Outgoing messages Responds to asynchronous messages placed in outbound message queue May transform outgoing messages before placing in message delivery queue Acts as message router for asynchronous replies

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Hub as Router

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Hub as a Router – Seamless Connection

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Messaging Implementation ioAgent to ioAgent

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioTransform Ability to deploy java transforms as an integral part of messaging Transforms automatically generated using ETL Integrated Development Environment, Transform Manager Open Interfaces allow transforms to and from virtually any format

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Transformation Manager Rapid Development Process Transformation Manager supports the whole development process Design Testing Deployment The heart of the system is Transformation Builder, a state-of-the-art interactive transform designer Close integration with Transformation Tester to quickly verify transforms Flexible Deployment via mixed Java and XSLT run time environments

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Step By Step Approach Load data models into repository In-built support for RDBMS, XML, and Java Open Interface to access virtually any kind of data Create Transforms Define transforms by drag and drop Add more complex mapping and complete business logic Viewing instance data to provide assistance Test transforms Viewing instance data Rework transforms Deploy Simple Java deployment Update and maintain Transforms generate HTML documentation Transforms are easily updated as models change

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Transformation Manager Design Environment

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Open Interfaces – Extends TMs Ability to Any Scenario Access to User Specified Models Supports Associated Instance Data Fits Client Architectures Supports Push and Pull modes Pull Model supports Propagation of Selection Example Uses Express\Epicentre Part21 ASCII LIS, LAS, Binary RP66 Edifact Messages

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Truly Open Interfaces Access to all forms of models and instance data Allows user models to be loaded with full constraints, relationship cardinality, etc. Easily create user-defined read and write adapters for accessing instance data Transforms called by client architecture with parameter passing and callbacks as required Java interface description published for users to implement Models and instance data can be accessed and viewed as any other data in TM Numerous uses and examples available Edifact, CSV files, EXPRESS Models and Part 21

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioTransform Interface ioNode is designed to support a very simple and generic transformation interface public ArrayList transform(String configFile,ArrayList dataIn)throws Exception; Carry out a transform on a list of source data, dataIn, and return a list of target data entities. The configuration of the transform is contained in the configuration file supplied by the caller. The configuration file is read using methods contained by the transform provider so can be of any schema and format

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioDB Requirements Storage of student learner records as native XML Generation of a consortium and LRDB unique identifier Update student learner records Roll back log of present number of transactions or messages Generation of Asynchronous messages and responses to business transactions such as new record. The ability to switch and configure the number of transactions stored per consortium and LRDB unique identifier. The ability to switch the facility for extending and building individual master records using xpath updates and a unique identifier. An admin interface for record retrieval and reporting is provided. A standard API for third party interfaces such as the Learner record portal is to be made available and documented.

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Learner Record Database

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioDB

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioDB Interface MethodDescription setDatabaseConnection Connect to the database that hosts the learner records repository createCollection Create a new collection in the learner records setLearnerRecordCollection Set the name of the collection that holds the learner records setArchiveLearnerRecordCollection Set the name of the collection that holds the archived learner records getCollectionList Get a list of collection names in the database removeCollection Remove a collection from the learner records database addRecord Add a new student to the learner record collection updateRecord Update an existing student in the learner record collection getRecord Get a particular learner record in the database queryCollection Query the learner record database using the supplied XPath deleteRecord Delete a student in the learner record collection resetUniqueIDSequence Reset the unique ioDB sequence number setNumberofArchivedRecords Set maximum number of archived entries for each learner record rollbackRecord Reinstate archived learner record as the current learner record

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioDB Interface Sequence Diagram

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Current Activities Example CSV files provided by Phosphorix Create, Update, and Withdraw student records now implemented Transform CSV files to IMS-LIP LSC to HESA Lookup Secure messaging between ioAgents and ioHub Stored in ioDB with unique Global ID Asynchronous replies generated Transform IMS-LIP files to CSV files on round-trip completion HESA to LSC Lookup

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Next Steps - Some Issues Web Browser access to all software interfaces Provide easy to use access to data via existing interfaces Resolve a few incompatibilities between CSV and IMS-LIP formats Agree format for asynchronous replies Implement Publish / Subscribe messaging Implement rollback an archive logic in ioDB Implement full query logic in ioDB Testing of routing logic Discuss logic for multi-hop messages Complete full JMS messaging (currently used only as a persistent messaging mechanism) Obtain full set of messages and transform descriptions Create and test full set use TM HTML documentation for review Maintain as regression test suite Determine timescales for rollout

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Q&A ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Further Information John Bigerstaff +44 (0) (0) or