Streams – DataStage Integration InfoSphere Streams Version 3.0 Mike Koranda Release Architect

Agenda What is InfoSphere Information Server and DataStage? Integration use cases Architecture of the integration solution Tooling

Information Integration Vision Transform Enterprise Business Processes & Applications with Trusted Information Deliver Trusted Information for Data Warehousing and Business Analytics Address information integration in context of broad and changing environment Simplify & accelerate: Design once and leverage anywhere Secure Enterprise Data & Ensure Compliance Build and Manage a Single View Key Speaking Points InfoSphere is the first true IIG platform in the market Comprehensive It has the most mature and comprehensive set of capabilities in the market. Each component is a recognized market leader in its respective technology Integrated The components share a common foundation of meta data, data discovery, and a business glossary. The components are integrated to work with one another to address the enterprise use cases. Intelligent The objective of InfoSphere is to offer the functionality required to actually integrate and govern data – its objective is to be far more than a tool, or a collection of tools. It includes automation of repeatable tasks, and business logic to manage proactive alerts and notifications Consolidate and Retire Applications Integrate & Govern Big Data Make Enterprise Applications more Efficient 3

Structured Repeatable Linear Unstructured Exploratory Iterative IBM Comprehensive Vision Traditional Approach Structured, analytical, logical New Approach Creative, holistic thought, intuition Data Warehouse Hadoop Streams Data Warehouse Hadoop Streams Transaction Data Web Logs Internal App Data Social Data Structured Repeatable Linear Unstructured Exploratory Iterative Information Integration & Governance Mainframe Data Text & Images Key Points Traditional technologies are very well suited to structured, repeatable tasks – when you do something many times it makes sense to structure it Also have controls in place for the accuracy and quality of the data Historical data – trend analysis New technologies are complementary – they address speed and flexibility Very good an one-time or ad-hoc analysis Also good at exploration – determining new questions to ask The point is organizations need both sides – and data growth (or big data) is a challenge for both sides. A big data platform has to address both sides to truly address enterprise needs. OLTP System Data Sensor Data Traditional Sources New Sources ERP data Traditional Sources New Sources RFID

1 4 2 5 3 6 IBM InfoSphere DataStage Industry Leading Data Integration for the Enterprise Simple to design - Powerful to deploy Rich capabilities spanning six critical dimensions Developer Productivity Rich user interface features that simplify the design process and metadata management requirements Runtime Scalability & Flexibility Performant engine providing unlimited scalability through all objects tasks in both batch and real-time 1 4 Transformation Components Extensive set of pre-built objects that act on data to satisfy both simple & complex data integration tasks Operational Management Simple management of the operational environment lending analytics for understanding and investigation. 2 5 Connectivity Objects Native access to common industry databases and applications exploiting key features of each Enterprise Class Administration Intuitive and robust features for installation, maintenance, and configuration 3 6

Use Cases - Parallel real-time analytics An enterprise may wish to send data from DataStage to Streams to perform near real-time analytic processing (RTAP) on the data stream. By sending data to Streams from the DataStage flow, Streams can perform RTAP in parallel to data being loaded into a warehouse by DataStage.

Use Cases - Streams feeding DataStage Alternatively, an enterprise may wish to send data from Streams to DataStage. A typical use-case might be processing telco call details: the Streams job performs RTAP processing, and then forwards the data to Data Stage to enrich, transform and store the call details for archival and lineage purposes. A Streams application may require a data source that is not provided by Streams but has a first-class connector in DataStage (e.g. SAP)

Use Cases – Data Enrichment In enrichment deeper analytics are offloaded from the main DataStage flow performed in streams either because of scalability or stream specific analytic capabilities like text analytics, video analytics time series analysis, SPSS model scoring, etc. Care should be taken in this scenario as DataStage can provide a transactional flow while the interaction to steams can lose tuples unless the applications is properly architected.

Runtime Integration High Level View DataStage Streams Job Job DSSource / DSSink Operator Streams Connector TCP/IP Composite operators that wrap existing TCPSource/TCPSink operators

Streams Application (SPL) use com.ibm.streams.etl.datastage.adapters::*; composite SendStrings { type RecordSchema = rstring a, ustring b; graph stream<RecordSchema> Data = Beacon() { param iterations : 100u; initDelay:1.0; output Data : a="This is single byte chars"r, b="This is unicode"u; } () as Sink = DSSink(Data) { param name : "SendStrings"; config applicationScope : "MyDataStage"; When the job starts, the DSSink/DSStage stage registers its name with the SWS nameserver

DataStage Job User adds a Streams Connector and configures properties and columns

DataStage Streams Runtime Connector Uses nameserver lookup to establish connection (“name” + “application scope”) via HTTPS/REST Uses TCPSource/TCPSink binary format Has initial handshaking to verify the metadata Supports runtime column propagation Connection retry (both initial & in process) Supports all Streams types Collection types (List, Set, Map) are represented as a single XML column Nested tuples are flattened Schema reconciliation options (unmatched columns, RCP, etc) Wave to punctuation mapping on input and output Null value mapping

Tooling Scenarios User creates both DataStage job and Streams application from scratch Create DataStage job in IBM Infosphere DataStage and QualityStage Designer Create Streams Application in Streams Studio User wishes to add Streams analysis to existing DataStage jobs From Streams Studio create Streams application from DataStage Metadata User wishes to add DataStage processing to existing Streams application From Streams Studio create Endpoint Definition File and import into DataStage

Streams to DataStage Import On Streams side, user runs ‘generate-ds-endpoint-defs’ command to generate an ‘Endpoint Definition File’ (EDF) from one or more ADL files User transfers file to DataStage domain or client machine User runs new Streams importer in IMAM to import EDF to StreamsEndPoint model Job Designer selects end point metadata from stage. The connection name and columns are populated accordingly. IMAM Streams command line or Studio menu ADL EDF EDF Xmeta ADL FTP

Stage Editor

Stage Editor

DataStage to Streams Import On Streams side, user runs ‘generate-ds-spl-code’ command to generate a template application that from a DataStage job definition The command uses a Java API that uses REST to query DataStage jobs in the repository The tool provides commands to identify jobs that use the Streams Connector, and to extract the connection name and column information The template job includes a DSSink or DSSource stage with tuples defined according to the DataStage link definition Streams command line or Studio menu Java API Xmeta SPL REST API HTTP

DataStage to Streams Import

Availability Streams Connector available in InfoSphere Information Server 9.1 The Streams components available in InfoSphere Streams Version 3.0 in the IBM InfoSphere DataStage Integration Toolkit