Data Replication with Advanced Replication & Oracle Streams John Abrahams Technology Sales Consultant Oracle Nederland

What is Replication  Multiple copies of data at different sites  Increased availability  Manual data replication implementations – Export/Import – CREATE TABLE AS SELECT FROM REMOTETABLE – COPY

Oracle9i Features for Information Sharing  Features Introduced In Prior Releases – Data Guard – Physical Standby Database – Advanced Queuing -- Message Queuing – Advanced Replication -- Replication of Data – Change Data Capture -- DW Loading  Features available in Oracle9iR2 – Oracle Streams -- a comprehensive information sharing solution – Data Guard -- Logical Standby Database based on Oracle Streams

Oracle Advanced Replication

Oracle9i Replication Technology  Oracle9i provides built-in technology to create and manage replicated environments – Integrated, no add-ons, no special commands – Managed with Oracle Enterprise Manager  Advanced data replication technology – Bi-directional, all copies potentially updatable – Automatic conflict detection and resolution – Tables and supporting objects – Full copies or subsets – DDL (schema changes) as well as DML (transactions) – Continuous or on demand replication

Replication Usage Examples Information Dissemination Move data (such as price lists) locally for improved response times Offload queries from master site Only subsets of data need to be replicated

Replication Usage Examples Call Centers Same data available at all sites Very useful for balancing usage Viable failover strategy; if one sites fails, others remain available Updates can be done anywhere

Replication Usage Examples Branch Office Automation Data is located close to users at local sites Data is consolidated at central site for processing and rollups Only subsets of data need to be replicated at each site Updates can be done anywhere

Replication Usage Examples Mass Deployment Mobile users must be able to operate even when disconnected from central sites Mobile users can replicate only data they need to their laptop Synchronization can be done when re-connected

Architectural Overview Single master replication Single, updatable master Multiple updatable or read-only materialized views (snapshots) – Full copies or subsets of master All conflicts resolved at parent site of materialized view Oracle9i standard edition master materialized view master materialized view

Support for multiple, n-way connected, updatable masters Improves scalability and availability Oracle9i Enterprise Edition master M View Master M View M View Architectural Overview Multiple master replication

Master vs. Materialized View Replication MultimasterMaterialized View Server-to-Server Mass Deployment Large number of small remote sites HQ and regional offices Continuous, near real- time data propagation Periodic Bulk Transfer Updates per transaction Final Values of Changed Rows Only Full Copies Subsets or Full Copies

Advanced Replication Key Features  Near real-time replication (multimaster) – Parallel data propagation – Multiple, pre-defined conflict resolution methods  Mass deployment (materialized views) – subquery materialized views – deployment templates – multitier materialized views  Oracle Enterprise Manager for configuration and administration  Specialized options: procedural and synchronous replication

Near Real-time Replication  Benefits: – availability, scalability, failover  Uses: – Telesales, support  Requirements: – Efficient data capture and storage – Efficient data propagation – Continuous data propagation – Automatic resolution of conflicting updates

Architectural Overview Multimaster Groups of related schema objects kept in synch at multiple locations Sites communicate by broadcasting changes to all other sites master

Replication Objects  Database object replication to multiple servers  The following database objects can be replicated – Tables, Indexes, Views, Synonyms, Triggers – Packages, Procedures, Functions – Advanced Data Types  User-Defined Types, Indextypes  Tables with column objects, object tables  Nested Tables, Varrays

Efficient Data Capture and Storage  Committed changes are added to queue for later propagation to remote sites – Enqueued using advanced queueing mechanism – Captured and applied using internal C code – Minimum data needed to apply change is captured Updates Internal Trigger Advanced Queue Source tables

Efficient Data Propagation  Queued changes are pushed to remote sites in parallel for improved performance – Single parallel stream – Maintains transactional consistency – Automatically detects transaction dependencies If update … If delete … If insert... If update … If delete … If insert... internal procedure background process

Parallel Data Propagation  Dependencies – Transaction “B” is dependent on transaction “A” if “B” accesses data “A” has updated – Dependency detection is dynamic and light- weight  Ordering – Dependent transactions are propagated in dependency order – All other transactions are propagated in parallel

Continuous Propagation  Changes can be continuously propagated or at a fixed interval, fixed time, or on demand  Different intervals can be used for each location  Different intervals can be used for each group  Dynamic Views to monitor propagation activity and throughput

Automatic Conflict Resolution  Automatic conflict detection with user- selectable conflict resolution routines – latest timestamp, earliest timestamp, maximum or minimum value, overwrite, priority group, discard, site priority, average, or additive  User-definable resolution routines  Detection and resolution based on column groups

Mass Deployment Replication  Benefits: – disconnected, updatable  U ses: – field sales, field service  Requirements: – Easily define unique subsets – Easily deploy to 100’s of sites – Support mobile users  refresh on demand

– Full transactional consistency – Efficient, batch-oriented refresh l Scheduled or on demand l Refresh groups preserve master-detail relationships – Updatable materialized views use deferred transactions to push changes to master Architectural Overview Materialized Views SELECT … FROM... network Master table log Materialized View Updatable or read-only copy of a table, or portion of a table, at a point in time

Managing Advanced Replication

Specialized Options  Synchronous Replication – always up to date, no conflicts – slower response, network dependent  Procedural Replication – faster for batch processing – must be serialized, best done in off hours – useful for purging

Replication Summary  Full and subset  Near realtime or on demand  Graphical administration tool  Sophisticated Functionality – Efficient data capture and storage – Parallel data propagation – Automatic conflict detection and resolution – Subquery subsetting – Deployment templates – Multitier materialized views – Specialized options

Oracle Streams

Oracle Streams – Unified Messaging and Data Movement Oracle Advanced Queuing (AQ) Oracle Advanced Replication Update Standby Feed Data Warehouse Messaging Replication Standby Data Warehousing Oracle Streams Intelligent, Unified, Time-Ordered Information Stream

Oracle Streams  A new solution for information sharing  Provides a unified architecture for all information sharing solutions – uniquely flexible replication – message queuing – data warehouse loading – event management and notification  The foundation of Data Guard Logical Standby Database

Streams Basic Elements  Three basic elements in each database – Capture – Staging – Consumption (apply) Consumption StagingCapture

Multi-Database Streams  A stream can contain multiple elements from multiple databases  Events flow between staging areas Consumption Staging Capture Consumption Staging Capture

Capture  Streams captures events – Implicitly: log-based capture of DML and DDL – Explicitly: Direct enqueue of user messages  Captured events are published in the staging area Capture

Log-Based Change Capture  Low overhead, low latency change capture – Changes to the database are written to the online redo log – Oracle Streams can extract changes from the log as it is written (mining the active log) – Changes are formatted as a Logical Change Record (LCR), a SQL like representation of the change Capture

Direct Enqueue  User applications can explicitly enqueue user messages into the staging area – Multiple open interfaces supported: JMS, C, PLSQL, SOAP (XML/HTTP), XML/SMTP – Allows applications to communicate at a higher level – Allows users to introduce events into the stream from non-Oracle systems Capture

Staging  Streams publishes captured events into a staging area – Implemented as a queue – Supports for new self-describing type “any” datatype allows a single staging area to hold any type of data – All events, LCRs and user-messages, can be staged in the same queue – Messages remain in staging area until consumed by all subscribers Staging

Staging Area Propagation  Other staging areas can subscribe to events – in same database – in a remote database  Events can be routed through a series of staging areas Propagation Staging Staging

Transformations  Transformations can be performed – as events enter the staging area – as events leave the staging area – as events propagate between staging areas  Transformation examples – change format, data type, column name, table name Staging

Consumption  Staged events are consumed by subscribers – Implicitly: Apply Process  Default Apply  User-Defined Apply – Explictly: Application dequeue via open interfaces  JMS, C, PLSQL, SOAP (XML/HTTP), XML/SMTP Consumption

Default Apply  The default apply engine will directly apply the DML or DDL represented in the LCR – apply to local Oracle table – apply via DB Link to non-Oracle table  Automatic conflict detection with optional resolution – unresolved conflicts placed in exception queue  Parallel apply maximizes concurrency Consumption

User-defined Apply  User-written custom apply functions  Written in PL/SQL, Java, C, C++  Uses: – custom transformations – column subsetting – normalizing or denormalizing data – populating related fields or tables Consumption

Rule-based Subscription  Consumers subscribe to published events  Content-based subscriptions limit delivered events to those meeting the subscription criteria  Rules govern capture, staging, and consumption Staging Area Rules Engine Publish Subscribe UPDATE EMP... WHERE OBJECT = ‘EMP’

NY(master) London(subset) Milan(subset)Paris(subset) Directed Networks  Propagation independent of Apply  Rules-based subscription determine if event is locally applied – London applies UK only  WAN Friendly – Send once, fan out – NY-->London, London-->Milan, London-->Paris INSERT … VALUES (‘EUROPE’,’ ITALY’) ITALY EUROPE FRANCE

Heterogeneous Support  Oracle to non-Oracle apply via gateway – Apply process on Oracle node applies change  Non-Oracle to Oracle change capture supported via explicit enqueue of LCRs  Message Gateways – MQ Series – Tibco LCR or user message Gateway MessageGateway MQ Series Sybase

Streams Deployments  Streams can be deployed to meet a variety of information sharing requirements – Replication – Data Warehouse Loading – Event Notification – Message Queuing – Data Guard Logical Standby Database

Replication  Streams asynchronously maintains multiple copies of objects via automatic apply – Identical objects – Related via a transformation or function  Streams automatically captures, propagates, and applies DML and DDL changes – Detects and optionally resolves conflicts  Supports flexible data movement and subsetting  Gateways and APIs for heterogeneous support  Compatible with Materialized Views

Replication  Benefits: – No quiesce for DDL – Lower overhead on production system – Reduced network traffic – Flexible configurations Log-based Capture Capture Stage DefaultApply Log-based Stage DefaultApply Propagation

Data Warehouse Loading  Streams can load data warehouse staging areas and Operational Data Stores – Updates captured from a production system – Messages and business events from a process flow  Supports continuous or batch loading  Automatically transforms data to appropriate format and schema during Operational Data Store load

Data Warehouse Loading  Benefits: – low overhead – automatic transformation – near real-time loading of operation data stores Stage User-definedApply Propagation Stage Log-based Capture Capture Production Database Staging Table or ODS

Event Notification  Streams can notify subscribers that events of interest have occurred – Pager notification of flight delays (Orbitz) – Notification of price drops (CNET Shopper) – Notification to sales manager of Gold Customer purchase (CRM App)  Streams can evaluate DML events and send notifications to applications that send s, page users, etc – Users get information they want

Event Notification  Benefits: – scalable – reduced custom development Log-based Capture Capture Stage ExplicitDequeue

Message Queuing  Streams can be deployed as an enhanced database integrated message queuing solution – Point-to-point messaging, publish and subscribe – Single data, security and transactional model for database and message queuing operations – Centrally managed and multi-consumer queues to simplify configuration – Content-based subscriptions, internet access – Automatic dequeue to server-run user function – Automatic transform DML/DDL into messages

Message Queuing  Benefits: – Reduced development costs – Easy database integration – Single development, operational, security model – Reliability and integrity of database Stage ExplicitDequeue Propagation Stage ExplicitEnqueue Source Database Destination Database

Data Guard Logical Standby Database  Multiple copies of data protects from human and data errors, and disasters  Special case of replication – Entire database (by default) – One direction only  Streams supports reporting from standby as updates are applied  Data Guard adds higher level interface, tailored GUIs and broker

Data Guard Logical Standby Database  Benefits: – Open while protecting data – Support near real-time reporting from standby – Protects from physical corruptions – Additional indexes and materialized views DefaultApply Remote Logging Log-based Capture Capture Production Database Logical Standby Database

Other Oracle9i Information Sharing Features  Advanced Replication – provides compatible replication with Oracle 8, 8i, and 9i databases – Migration path to Oracle Streams in future release  Advanced Queuing – Compatible with Oracle Streams – Most functionality offered in Streams – API’s retained for compatibility – Migration path to Oracle Streams in future release

Other Oracle9i Information Sharing Features  Data Guard Physical Standby Database – Uses media recovery mechanism to apply changes to database – Creates an exact copy of the production database  block-for-block copy  Same version of Oracle, same hardware/software architecture – Supports very high transaction workloads – Will coexist with Data Guard Logical Standby Database

Summary  Oracle Streams unifies all enterprise information into a single Stream – Unifies database, messaging, replication, publish/subscribe APIs and capabilities  Allows deployment of a variety of solutions  Provides a single, unified solution to the problem of Information Sharing Consumption Staging Capture