1. Oracle Data Integration Strategy and Roadmap Oracle Fusion Middleware Product Management

2. Agenda Introduction to Oracle Data Integration
Business Drivers for Data Integration
Benefits from a Modern Data Integration Platform
Key Oracle Data Integration Products
Oracle Data Integration Solution
Oracle GoldenGate Overview
Data Integrator Overview
ODI & GG Together
Best of Breed Integration for Batch and Realtime Data Integration
Support any Type of Data Integration Use Case
Implementing Best-Practice Technical Pattern for Data Warehousing
Technical Details – How it Works
Demonstration and Q&A (if available)

3. Data Integration Portfolio I N T R O D U C T I O N

4. Business Drivers for Data Integration Essential Ingredient for Information Agility
Strategic Value of Data Integration
Consistency for major enterprise initiatives like BI, DW, & MDM
Common technical foundation platform across data silos
Central point for data governance, availability and controls
Today’s organizations are looking at multiple paradigms for agility either separately or together.
In each case, there is a common element among them. Data Integration. Data Integration provides the important aspects for moving data and turning it into re-usable information – it acts as a key enabler for each architecture principle (SOA, BI, or Data Warehousing) and can also act as a key bridge between them.
SOA and DI Focus example: For example, lets say I want to implement a process in my Service Oriented Architecture that accesses large amounts of data. Unfortunately that data is part of a data warehouse, to ensure that I only get the updated information that is constantly being synchronized, I’ll need to leverage capabilities in Data Integration and make a call to it from the service in my SOA. If it isn’t connected easily the architects don’t know how to get to it. It also needs to be optimized for performance, the same way that it is from it being managed by my data warehouse.
Key Data Integration Use Cases
BI, DW, and OLTP Data Integration & Replication
SOA, Enterprise Integration & Modernization
Migrations and Master Data Management

5. Benefits from a Modern DI Platform Data Integration is Infrastructure that enables Business Value
Do More with Less
Design metadata-driven integration
Leverage skills & dictate patterns
Compete Globally 24X7
Ensure continuous uptime
Access data in real time
Use Data for Competitive Advantage
Ensure the quality of your data
Actively govern most valuable asset
Automate and Adapt Business Processes
Expose data services for reuse
Orchestrate processes using SOA

6. Key Data Integration Products
Real Time Data Replication
Changed Data Capture
DBMS High Availability
Disaster Tolerance
Heterogeneous E-LT & ETL
High-speed Transformations
OLAP Data Loading
Data Warehouse Loading
Comprehensive Integration
ELT/ETL for Bulk Data
Service Bus
Process Orchestration
Human Workflow
Data Grid
Data Service Modeling
XQuery Data Federation
Data Security/Redaction
XA Compliance
The Empirix family of products – now called the Application Test Suite (or ATS) comprises three main solutions: the first two tools to automate the actual testing:
One for Functional testing – which is exercising the various features of an application as a single user to ensure that everything works
The other is for Load testing (also know as performance testing) – which allows customers to drive large amounts of traffic against their applications to ensure they can scale to expected levels.
The third solution is Test Process Management – which as the name suggests, allows customers to manage their entire QA process, includes things like:
Definition of requirements
Developing test cases
Tracking defects, etc.
These solutions can be sold individually or as a suite – depending on the customer’s need. But they now all fall under the Enterprise Manager Brand – within the SAM bundle, and I’ll describe what that is in a second.
One important point to note, though, is that ATS is currently focused on WEB Applications and Web derivatives (such as web services, SOA, AJAX, etc).
So from a selling standpoint the immediate sweet spot for you are Oracle customers running or deploying web-based applications (such as Siebel or Peoplesoft) or customer with websites powered by Oracle middleware or databases. For Web-based applications ATS is the best solution on the market – in terms of ease-of-use and feature set and the accuracy of the test conducted. Feel free to contact me offline to learn more about these differentiated capabilities. There is a bunch of stuff on the Enterprise Manager website as well.
The roadmap calls for supporting the non-web applications such as EBS - product management will be providing these details and timelines shortly.
Business Data / Metadata
Statistical Analysis
Time Series Reporting
Integrated Data Quality
Cleansing & Parsing
De-duplication
High Performance
Integrated w/ODI 6

7. Business Intelligence
Oracle Data Integration Solution Best-in-class Heterogeneous Platform for Data Integration
MDM Applications
SOA Platforms
Oracle Applications
Business Intelligence
Activity Monitoring
Custom Applications
Comprehensive Data Integration Solution
SOA Abstraction Layer
Process Manager
Service Bus
Data Services
Data Federation
Oracle Data Integrator
Oracle GoldenGate
Oracle Data Quality
ELT/ETL
Real-time Data
Data Profiling
Data Transformation
Log-based CDC
Data Parsing
Bulk Data Movement
Bi-directional Replication
Data Cleansing
Data Lineage
Data Verification
Match and Merge
Oracle already had a comprehensive offering for Data Integration which helped move data in bulk across heterogeneous sources and targets. With the best-in-class ELT/ETL technology of ODI-EE and the best-in-class data cleansing solution: Oracle Data Profiling and Data Quality, these solutions could be integrated together with Service Oriented Architecture approaches to build re-usable data services.
Oracle GoldenGate now completes the picture with the addition of real-time change data capture and replication for high availability.
Storage
Data Warehouse/ Data Mart
OLTP System
OLAP Cube
Flat Files
Web 2.0
Web and Event Services, SOA

8. Data Integration Portfolio G O L D E N G A T E

9. Oracle GoldenGate Overview Enterprise-wide Solution for Real Time Data Needs
Standardize on Single Technology for Multiple Needs
Deploy for Continuous Availability and Real-time Data Access for Reporting / BI
Highly Flexible
Fast Deployments
Lower TCO & Improved ROI
Disaster Recovery, Data Protection
Standby
(Open & Active)
Zero Downtime Migration and Upgrades
Log Based, Real-Time Change Data Capture
Operational Reporting
Reporting
Database
OGG
ETL
ODS
EDW
ETL
Heterogeneous Source Systems
Real-time BI
EDW
Query Offloading
Data Distribution 9

10. How Oracle GoldenGate Works
Modular De-Coupled Architecture
Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs.
Trail: stages and queues data for routing.
Pump: distributes data for routing to target(s).
Route: data is compressed, encrypted for routing to target(s).
Delivery: applies data with transaction integrity, transforming the data as required.
Trail
Trail
Capture
Pump
Delivery
LAN/WAN Internet
TCP/IP
Source Database(s)
Target Database(s)
Bi-directional 10

11. Data Integration Portfolio D A T A I N T E G R A T O R

12. Oracle Data Integrator Enterprise Edition Optimized E-LT for High Performance, Productivity and Low TCO
Legacy Sources
E-LT Transformation vs. E-T-L
Any Data Warehouse
Application Sources
Declarative Set-based design
Change Data Capture
Any Planning System
Main Points: ODI-EE is a the strategic ETL tool from Oracle; it’s ELT based which is unique in the industry, it combines the offering from both ODI and OWB.
Oracle Data Integrator Enterprise Edition delivers unique next-generation Extract Load and Transform (E-LT) technology that improves performance, reduces data integration costs, even across heterogeneous systems. Oracle Data Integrator offers the productivity of a declarative design approach, as well as the benefits of an active integration platform for seamless batch and real-time integration. In addition, Knowledge Modules provide modularity, flexibility, and extensibility.
Oracle Data Integrator Enterprise Edition is optimized for Oracle Database to provide real-time data warehousing with advanced ETL/ELT and data modeling. Oracle Data Integration Enterprise Edition, a foundational component to the Oracle Data Integration Suite is Oracle Fusion Middleware's strategic data integration offering; which has emerged from combining the best of both Oracle Data Integrator (ODI) and Oracle Warehouse Builder (OWB) into a single unified product offering
Hot-pluggable Architecture
OLTP DB Sources
Pluggable Knowledge Modules 12

13. Business Intelligence & Data Warehouse
How ODI Works: E-LT Architecture High Performance, Flexible, Lightweight Architecture
Key Architecture Benefits: 100% Java, Open APIs, fast E-LT
Packaged Application
Business Intelligence & Data Warehouse
ODI Agent
ODI Agent may be deployed in any part of the architecture
LKM
IKM
JKM A D B
C$_0
Embeddable Java Agent consumes very little CPU or RAM
E-LT Architecture uses DBMS CPUs, not the Application CPUs
Open APIs, Open Metadata, Open XML Knowledge Modules
E-LT architecture enables that the agent can be deployed on any part of the source, target system I$
E$ (Errors)
LKM
C$_1
File C
IKM
CKM
RKM
Extract-Load
Transform
Check-Load

14. Data Integration Portfolio O D I & G G T O G E T H E R

15. Oracle Data Integration Solution Best-in-class Heterogeneous Platform for Data Integration
Oracle Applications
Custom Applications
MDM Applications
Business Intelligence
Activity Monitoring
SOA Platforms
Comprehensive Data Integration Solution
SOA Abstraction Layer
Process Manager
Service Bus
Data Services
Data Federation
Oracle Data Integrator
Oracle GoldenGate
Oracle Data Quality
ELT/ETL
Real-time Data
Data Profiling
Data Transformation
Log-based CDC
Data Parsing
Bulk Data Movement
Bi-directional Replication
Data Cleansing
Data Lineage
Data Verification
Match and Merge
Oracle already had a comprehensive offering for Data Integration which helped move data in bulk across heterogeneous sources and targets. With the best-in-class ELT/ETL technology of ODI-EE and the best-in-class data cleansing solution: Oracle Data Profiling and Data Quality, these solutions could be integrated together with Service Oriented Architecture approaches to build re-usable data services.
Oracle GoldenGate now completes the picture with the addition of real-time change data capture and replication for high availability.
Storage
Data Warehouse/ Data Mart
OLTP System
OLAP Cube
Flat Files
Web 2.0
Web and Event Services, SOA

16. Oracle E-LT + Real-time
Best-of-Breed Data Integration Heterogeneous, Real-time, Non-Invasive, High Performance E-LT, and Low Hardware Costs
Traditional ETL + CDC
Oracle E-LT + Real-time
Invasive Capture on OLTP systems using complex Adapters
Transformations in ETL engine on expensive middle tier servers
Bulk load to the data warehouse with large nightly/daily batch
Continuous feeds from operational systems
Non-invasive data capture
Thin middle tier with transformations on the database platform (target)
Mini-batches throughout the day or bulk processing nightly
Extract
Trickle
Xform
Xform
GG+ ODI
Bulk
GG+ ODI
Lookup Data
Lookup Data
Staging
Load
Heterogeneous 16

17. Support Any Type of Data Integration Best of Breed means using the Right Tools for the Job!
Analytical
OLTP
Query / Report
ODS
EDW
ODS
OLTP
OLTP
Operational
57% of Survey Respondents say that ETL is preferred technology for Operational DI Projects
Key Business Initiatives for Op DI projects Reported:
- 38% for improving business operations
34% for data as enterprise asset
35% for governance and compliance programs
40% to IT platform standardization, modernization / centralization
“Half of organizations experience continuous OpDI work …Solutions commonly include multiple tool types and users report using combinations of ETL and Replication…
Phil Russom, TDWI Operational Data Integration Q2, 09 Report
OLTP
Old
OLTP
New
OLTP
OLTP
OLTP
OLTP
OLTP
Heterogeneous
Heterogeneous
Heterogeneous 17

18. Replicated Source Tables
ODI-EE Integration with GoldenGate Non-invasive Data Capture combined with ODI ELT strengths
Key Benefits:
Eliminate Overhead  no need for DB API overhead on the Source, or the invasiveness of the ODI J$ objects on the Source system,
Automate GoldenGate  automation of GG deployment directly from ODI GUI
Provide Common DW Pattern  supplies a common pattern for mini-batch style (non-real-time) DW aggregate loads
Source Systems
ODI Staging & Target
Source DB’s
ODI J$ Tables
Target EDW
Replicated Source Tables
Transactional
RDBMS
Replicated Source Tables
ODI CDC Framework
Target Tables
Source Tables J$
Generate all ODI CDC infrastructure
Execute end-to-end CDC
ODI
Generate all GG deployment files 18

19. Oracle’s Data Integration Joint Solution Best-of-Breed and Proven
Oracle GoldenGate
Oracle Data Integrator Enterprise Edition
Technology Differentiators:
E-LT architecture for best performance of high data volume transformations
Lowest latency and highest throughput; non-invasive, low overhead
Performance
De-coupled architecture; multiple deployment styles; open and extensible
Knowledge Module architecture for extensibility and flexible connectivity
Extensible &
Flexible
Embedded Strategy Reinforces Standalone Strategy
Leverage Oracle’s Unique Position – Whole Stack, Single Vendor
Offer Customers Lower TCO than Competition Can
Improve the Product through Challenging Use Cases
Establish Market Share & References to Drive Standalone Business
Standalone Strategy Reinforces Embedded Strategy
Customers Reluctant to Accept Non-Proven Embedded Technology
Internal Teams Reluctant to Accept Non-Proven Embedded Technology
Upgrading to Full Use Is Attractive for Strong Standalone Product
Allows Customers Opportunity to Further Leverage Investment
You need both, and only Oracle can do that.
Maintain transactional integrity; resilient against interruptions and failures
SOA-native, integrated with Fusion MW to fit future enterprise architectures
Enterprise 19

20. ODI & GG Together T E C H N I C A L D E T A I L S

21. Overview of the ODI KM Framework
Pluggable Knowledge Module Architecture
Reverse
Engineer Metadata
Journalize
Read from CDC Source
Load
From Sources to Staging
Check
Constraints before Load
Integrate
Transform and Move to Targets
Service
Expose Data and Transformation Services
Reverse
Integration w/GoldenGate is here! WS WS WS
Staging Tables
Load
Integrate
Services
CDC
Journalize
Check
Target Tables
Sources
Error Tables
Benefits
Leverage Database Optimizations: Native SQL; Native Functions; Native Loads; Native Journaling / CDC
Tailor to an organization’s existing best practices
Ease administration work
Reduce cost of ownership
Main Point: Knowledge Modules are a core architectural element of ODI that help to reduce TCO and improve flexibility and extensibility.
Knowledge Modules (KMs) are the core architectural element of ODI. They drive its modularity, flexibility and extensibility. KMs define the technical implementation of integration processes defined using declarative design. They implement the actual data flows and define the templates for code generation across the multiple systems involved in each process.
KMs are at the same time generic (they allow the generation of data flows regardless of the transformation rules) and highly specific (the code they generate and the integration strategy they implement are finely tuned for a given technology).
A comprehensive library of KMs (100+) is provided with ODI out of the box, but they can be tailored to implement existing best practices (e.g. for highest performance, for adhering to corporate standards, for specific vertical know-how, etc.).
ODI’s KM framework helps companies capture and reuse technical expertise and best practices, thus reducing cost of ownership. 21 21

22. ODI CDC in a Nutshell A General Framework for Change Capture on Source DBs
Automatic w/JKMs
Journal Tables
Capture Services
Create Capture Process
Start/Stop Capture Process
Subscription Services
Manage Consistency Sets
Register/Un-register Subscriber
Consumption Services
Consumption Views
Consumption Operations
Extend Window
Lock/Unlock Subscriber
Purge Operations
Consumers
Consumption Services
ExtendWindow()
PurgeJournals()
LockSubscriber()
UnLockSubscriber()
View: CONT
CONTID
CUSTNAME
CUSTID
I003
Thomas S.
C003
View: CUST
CUSTID
CUSTNAME
EMPID
C002
YAHOO
E003
View: EMP
EMPID
ENAME
E001
Joe Celko
Subscription Services
Journal Tables
J$CONT
CONTID
WID
I002 -
I003 20
J$CUST
CUSTID
WID
C002 20
J$EMP
EMPID
WID
E001 - 20 19
RegisterSubscriber()
AddTableToConsistencySet()
Capture Services
Capture Process
CDC_SET_SUBSCRIBER
CDC_SET
SUBSCRIBER
MIN_WID
MAX_WID
CDC000
FUSION_BI 10
PILLAR_HCM 9
Source Data
Table: CONT
CONTID
CUSTNAME
CUSTID
I001
Vijay R.
C003
I002
Raghu M.
C002
I003
Thomas S.
Table: CUST
CUSTID
CUSTNAME
EMPID
C001
AT&T
E003
C002
YAHOO
C003
GOOGLE
E002
Table: EMP
EMPID
ENAME
E001
Joe Celko
E002
Albert Einstein
E003
John Doe

23. Overview of the Integration Using ODI & OGG Together
Transactional RDBMS
Source Tables
Staging DB
Target DB
Target Tables
Replicated
Source Tables
ODI CDC
Framework
ODI
Interfaces
Datapump
Extract
WAN
Replicat
Source trail
files
Staging trail
files

24. Scenario: Analytics & Reporting
Transactional RDBMS
Source Tables
Staging DB1
Replicated
Source Tables
ODI CDC
Framework
Target DB
Target Tables
Datapump
Staging DB2
WAN
Replicat
Extract
Staging trail
files
Historic Analytics/
Reporting
Datapump
Replicat
Source trail
files
WAN
Replicated
Source Tables
Staging trail
files
Realtime
Reporting

25. Step-by-step How It Works
Transactional RDBMS
Source Tables
Staging DB
Target DB
Target Tables

26. 1. Replicated tables created with Common Format Designer (using ODI)
Transactional RDBMS
Source Tables
Staging DB
Target DB
Target Tables
Replicated
Source Tables

27. 2. Start Capturing Changed Data in Source (OGG Extract process)
Transactional RDBMS
Source Tables
Staging DB
Target DB
Target Tables
Replicated
Source Tables
ODI CDC
Framework
Extract
Source trail
files

28. 3. Initialize Staging and Target Data (with ODI or optionally, OGG)
Transactional RDBMS
Source Tables
Staging DB
Target DB
Target Tables
Replicated
Source Tables
ODI CDC
Framework
ODI
Interfaces
ODI
Interfaces
Extract
Source trail
files

29. 4. Start Replication / Propagate Changes to Target DB (OGG and ODI)
Transactional RDBMS
Source Tables
Staging DB
Target DB
Target Tables
Replicated
Source Tables
ODI CDC
Framework
ODI
Interfaces
Datapump
Extract
WAN
Replicat
Source trail
files
Staging trail
files

30. Simplified & Productive Development
D E M O N S T R A T I O N
Show the DT portion of the end-to-end demonstration here.
You can either use the viewlet, available here:
Or do it live using JDev 11g and the project stored at the above location.