DAT201: Introduction To SQL Server ™ Replication Matt Hollingsworth Program Manager Microsoft Corporation

Agenda SQL Server Replication Review SQL Server Replication Review – Concepts, terminology Snapshot Replication Snapshot Replication Transactional Replication Transactional Replication – Change tracking and forwarding – Deployment examples Merge Replication Merge Replication – Change tracking and forwarding – Deployment examples Summary Summary

What Is SQL Server Replication? Replicates database objects Replicates database objects Keeps the data synchronized Keeps the data synchronized Maintains 1 to 1 data row relationship Maintains 1 to 1 data row relationship Optionally subsets or transforms data Optionally subsets or transforms data Can push or pull data to destination Can push or pull data to destination Utilizes a Publish and Subscribe paradigm Utilizes a Publish and Subscribe paradigm

Publish/Subscribe Paradigm Publication: Publication: – Defines a group of source database objects to replicate – The individual database objects are identified as publication “articles” Subscription: Subscription: – Defines subscriber as a destination for data flowing from the specified source publication Filters – row and column Filters – row and column – Static filters define data partition during publication creation and apply to all future subscriptions – Dynamic filters are calculated during synchronization and allow data filtering on a per subscriber basis.

Publisher/Distributor (SQL Server) Servers (SQL Server, Oracle, Other) Publishers And Subscribers Laptops (SQL Server, MSDE) Handhelds (SQL Server CE) odbc/ole-db odbc/ole-db http Subscribers

Common Replication Scenarios Offloading query workload Offloading query workload – Web data caching – Data warehouse staging – Reporting – Warm standby/geographic fail-over Mobile clients Mobile clients Data consolidation and distribution Data consolidation and distribution Heterogeneous system integration Heterogeneous system integration

Types Of Replication Snapshot Replication Snapshot Replication – Point in time set of data created using bulk copy API – Used for replicating non-volatile data sets like price lists – Delivers initial data for transactional and merge replication Transactional Replication Transactional Replication – Targets well connected, server to server, low latency replication of data – Row changes are replayed in original order, preserving transactional consistency Merge Replication Merge Replication – Designed for occasionally connected applications – Delivers net data changes with row by row conflict resolution

Agenda SQL Server Replication Review SQL Server Replication Review – Concepts, terminology Snapshot Replication Snapshot Replication Transactional Replication Transactional Replication – Change tracking and forwarding – Deployment examples Merge Replication Merge Replication – Change tracking and forwarding – Deployment examples Summary Summary

Snapshot Replication Subscriber gets a complete copy of published data Subscriber gets a complete copy of published data Snapshot can be scheduled during off peak hours Snapshot can be scheduled during off peak hours PublishingDatabase SubscribingDatabase SnapshotAgent DistributionAgent Files containing schema and data Published Tables

Snapshot Replication Options Options – Configure a custom pre or post-snapshot script – Specify CAB compression of files – Specify FTP for delivering snapshot files – The snapshot step for transactional and merge publications can be bypassed if the initial data already exists at the subscriber (nosync)

Snapshot Replication demo demo

Agenda SQL Server Replication Review SQL Server Replication Review – Concepts, terminology Snapshot Replication Snapshot Replication Transactional Replication Transactional Replication – Change tracking and forwarding – Deployment examples Merge Replication Merge Replication – Change tracking and forwarding – Deployment examples Summary Summary

Transactional Replication PublishingDatabase SubscribingDatabase DistributionAgent Published Tables Distribution Agent delivers initial snapshot of data prepared by Snapshot Agent Distribution Agent delivers initial snapshot of data prepared by Snapshot Agent Logreader Agent reads changes from database log Logreader Agent reads changes from database log Logreader Agent stores changes in the Distribution database Logreader Agent stores changes in the Distribution database Distribution Agent forwards changes to subscribers Distribution Agent forwards changes to subscribers DBlogDBlog LogreaderAgent DistributionDatabase SnapshotAgent Files containing schema and data

Transactional Replication Change Tracking – Log Reader SQL Server marks transactions in log for replication SQL Server marks transactions in log for replication Log Reader reads last transaction id processed for a publisher from distribution database Log Reader reads last transaction id processed for a publisher from distribution database Retrieves next set of transactions and SQL statements from publisher log Retrieves next set of transactions and SQL statements from publisher log Writes SQL statements to distribution database Writes SQL statements to distribution database Advances replication watermark in the log to coordinate with the log manager of the database engine Advances replication watermark in the log to coordinate with the log manager of the database engine Writes history and error information into the distribution database Writes history and error information into the distribution database

Transactional Replication Forwarding – Distribution Agent The Distribution Agent reads the last transaction received by the subscriber The Distribution Agent reads the last transaction received by the subscriber Retrieves the next set of transactions and SQL statements from the distribution database Retrieves the next set of transactions and SQL statements from the distribution database Applies changes to subscriber Applies changes to subscriber Updates the last transaction received Updates the last transaction received Note: agent can run on the distributor (“push”) or on the subscriber (“pull”) Note: agent can run on the distributor (“push”) or on the subscriber (“pull”)

Transactional Replication Change Tracking - Updating Subscribers Immediate Updating Subscriber 2PC, RPC Publisher Distributor Read-OnlySubscriber Queued Updating Subscriber Queue NETWORK Queue Queue Reader Agent

Transactional Replication Change Tracking And Forwarding - Features Low overhead at publisher Low overhead at publisher Partial updates to BLOBS Partial updates to BLOBS Built in validation option Built in validation option Automatic cleanup of store and forward queues Automatic cleanup of store and forward queues Options for managing constraints at subscriber (NFR) Options for managing constraints at subscriber (NFR) Identity range management Identity range management Indexed view support Indexed view support

Transactional Replication demo demo

Agenda SQL Server Replication Review SQL Server Replication Review – Concepts, terminology Snapshot Replication Snapshot Replication Transactional Replication Transactional Replication – Change tracking and forwarding – Deployment examples Merge Replication Merge Replication – Change tracking and forwarding – Deployment examples Summary Summary

75%+ of typical online DB activity is selecting data! 75%+ of typical online DB activity is selecting data! Will reduce load and contention on OLTP server Will reduce load and contention on OLTP server Improves adhoc query performance and tuning options Improves adhoc query performance and tuning options OLTP Server Publisher Tokyo London Reporting Server Subscriber and Re-Publisher Subscriber Subscriber Redmond OLAP Cube Transactional Replication Reporting Server/DW Staging

Publisher Internet Clients Distributor Updates Replication Load Balancing Read Only Data Tier (Subscribers) App Server Tier Transactional Replication Web Data Caches Updatable Data Tier (Publisher)

Redmond Tokyo London New Delhi Transactional Replication Data Consolidation And Distribution Publisher Subscriber Subscriber Subscriber Publisher Publisher Publisher Subscriber

Transactional Replication Creating A Warm Standby Server Why replication and not log shipping? Why replication and not log shipping? – Continuous read access to the replica – Lower latency – Mirroring a subset of the original DB Otherwise use log shipping Otherwise use log shipping – First choice for maintaining warm standby – All databases changes replicated – Less overhead and easier to fail back

Agenda SQL Server Replication Review SQL Server Replication Review – Concepts, terminology Snapshot Replication Snapshot Replication Transactional Replication Transactional Replication – Change tracking and forwarding – Deployment examples Merge Replication Merge Replication – Change tracking and forwarding – Deployment examples Summary Summary

Merge Replication Change Tracking Net row changes tracked by system triggers in publishing and subscribing databases Net row changes tracked by system triggers in publishing and subscribing databases Merge Agent propagates changes between publishing and subscribing databases Merge Agent propagates changes between publishing and subscribing databases Conflicts are detected and resolved automatically Conflicts are detected and resolved automatically PublishingDatabase SubscribingDatabase MergeProcess Default or Custom Resolver

Merge Replication Change Tracking Metadata Generation – watermark that groups changes Generation – watermark that groups changes Row Lineage: Maintains sequence of changes to a row Row Lineage: Maintains sequence of changes to a row – Represents the identity of all replicas that modified the row, and which version they made – Detect simultaneous changes, subsequent changes and dampen sync loops Column Versions: Who updated the column, and when Column Versions: Who updated the column, and when – For every column, stores identity or replica that made last update, and the version – Attribute level tracking – merge changes to disjoint columns

Merge Replication Key Features Dynamic subscriber data partitions Dynamic subscriber data partitions Customizable conflict resolution Customizable conflict resolution HTTP sync with SQL Server CE HTTP sync with SQL Server CE In-line validation of data In-line validation of data Synchronization to alternate publishers Synchronization to alternate publishers

Merge Replication Dynamic Subscriber Partitions Goals Eliminate or reduce the possibility of conflicts. Eliminate or reduce the possibility of conflicts. Minimize the volume of data maintained at client Minimize the volume of data maintained at client Improve performance over a slow link Improve performance over a slow link Prevent clients from receiving sensitive data Prevent clients from receiving sensitive dataImplementation Vertical partitions - Column filters on tables Vertical partitions - Column filters on tables Horizontal partitions Horizontal partitions – Subset filters on individual tables – Join filters according to relationship between tables – Both static and dynamic partitioning can be supported

Merge Replication Conflict Resolution Source Wins Source Wins – Changes sent to destination Destination wins Destination wins – Discard changes and rely on next phase to propagate winner Generate new row Generate new row – E.g., Additive Resolver – Update destination and propagate changes during next phase to source – Customizable

IISIISIISIIS SQL Server Reconciler App SERVER SQL Server CE Server Agent Client Agent Message Replication Provider SQL Server Replication Provider Device DB SQL DB.IN.OUT Merge Replication HTTP sync with SQL Server CE

Merge Replication demo demo

Agenda SQL Server Replication Review SQL Server Replication Review – Concepts, terminology Snapshot Replication Snapshot Replication Transactional Replication Transactional Replication – Change tracking and forwarding – Deployment examples Merge Replication Merge Replication – Change tracking and forwarding – Deployment examples Summary Summary

Merge Replication Deployment Mobile applications Mobile applications – Sales Force Automation – Distribution Applications – Inventory Tracking Catalog Servers Catalog Servers – Regional servers that make offline local updates with shared data Incident tracking/Call Center applications Incident tracking/Call Center applications – Example: Microsoft RAID bug database

Merge Replication Example Sales Force Automation Topology West Coast East Coast HQ Regional Managers Mobile Users Sales Office Regional Server Sales Office Central Publisher

Merge Replication Server To Server - MSN Search Example Typically data is not partitioned Typically data is not partitioned Will provide better connectivity to client applications in a given region Will provide better connectivity to client applications in a given region Servers are mostly offline and sync regularly over a fast reliable link Servers are mostly offline and sync regularly over a fast reliable link Catalog 2 Catalog 3 Catalog 1

Merge Replication Distribution Apps HTTP synchronization HTTP synchronization Typically data is partitioned for each user Typically data is partitioned for each user Number of devices syncing with the server is large Number of devices syncing with the server is large Publisher IIS Server Firewall http SQL Server Connection Firewall

DW staging server HQ OLTP Server Internet customers Suppliers with heterogeneous databases Putting It All Together… East region server West region server Web data caches Eastern sales force Western sales force Warm Standby server Reporting Server

Related TechEd Sessions DAT412 Transactional Replication Internals DAT412 Transactional Replication Internals – Wednesday 16:45-18:00, Room 2 DAT411 Programming and Deploying Microsoft SQL Server 2000 Replication: Lessons Learned DAT411 Programming and Deploying Microsoft SQL Server 2000 Replication: Lessons Learned – Thursday 15: :15, Room 3

Online Resources Books Online – SQL Server 2000 Books Online – SQL Server 2000http:// Middle Tier Application Data Caching with SQL Server 2000 Middle Tier Application Data Caching with SQL Server middletierdatacaching.asp middletierdatacaching.asp Transactional Replication Performance Tuning and Optimization Transactional Replication Performance Tuning and Optimization Diagnosing and Troubleshooting Slow Partitioned Merge Processes Diagnosing and Troubleshooting Slow Partitioned Merge Processes slowpartitionedmerge.asp slowpartitionedmerge.asp

Don’t forget to complete the on-line Session Feedback form on the Attendee Web site

© 2002 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.