OPS-24: Success with OpenEdge® Replication What you don’t know can hurt you Cyril Gleiman Principal Technical Support Engineer

© 2008 Progress Software Corporation2 OPS-24: Success with OpenEdge Replication Obligatory (gratuitious) Quotation  An ounce of prevention is worth a pound of cure.  grams of prevention is worth.45 kilograms of cure

© 2008 Progress Software Corporation3 OPS-24: Success with OpenEdge Replication Success With Replication  Understand the architecture  Know your requirements  Analyze your system  Plan, configure, deploy,monitor  Succeed

© 2008 Progress Software Corporation4 OPS-24: Success with OpenEdge Replication Agenda  Terminology and architecture – Visual  Replication performance hotspots  Replication availability hotspots  Source and target management

© 2008 Progress Software Corporation5 OPS-24: Success with OpenEdge Replication Replication Terminology  Primary – original production machine  Secondary – original recovery machine  Source – production database  Target – recovery database  Replication server – rpserver (source)  Replication agent – rpagent (target)  DBService queue – stores IPC messages  Network pipe – TCP/IP rpserver -> rpagent

© 2008 Progress Software Corporation6 OPS-24: Success with OpenEdge Replication ! Architecture of Replication – Primary/Source Primary Machine R/Write Clients AI Extents DB Service Q Repl Server AI Extents Source Database Database Brk/Server Processes

© 2008 Progress Software Corporation7 OPS-24: Success with OpenEdge Replication Architecture of Replication – Secondary/Target Secondary Machine Repl Agent Database Brk/Server Processes Target Database R/O Clients

© 2008 Progress Software Corporation8 OPS-24: Success with OpenEdge Replication Architecture of Replication Primary Machine Secondary Machine Repl Agent Database Brk/Server Processes Target Database R/O Clients R/Write Clients AI Extents DB Service Q Repl Server AI Extents Source Database Database Brk/Server Processes

© 2008 Progress Software Corporation9 OPS-24: Success with OpenEdge Replication Agenda  Terminology and architecture - Visual  Replication performance hotspots  Replication availability hotspots  Source and target management

© 2008 Progress Software Corporation10 OPS-24: Success with OpenEdge Replication Performance Hotspots Primary Machine Secondary Machine Repl Agent Database Brk/Server Processes Target Database R/O Clients R/Write Clients AI Extents DB Service Q Repl Server AI Extents Source Database Database Brk/Server Processes

© 2008 Progress Software Corporation11 OPS-24: Success with OpenEdge Replication Source DB Performance Considerations  Speed/power of machine  Replication Plus – offload read only clients to target database  AI files – fixed extents  AI/BI blocksizes should be the same 16 K is the most efficient Need to truncate ai and bi

© 2008 Progress Software Corporation12 OPS-24: Success with OpenEdge Replication Target DB Performance Considerations  System should not be underpowered by comparison to the source system  No need for after-imaging  Read only clients

© 2008 Progress Software Corporation13 OPS-24: Success with OpenEdge Replication Network Performance  WAN vs. LAN  The bigger the pipe the better  Determine size of pipe  Whitepaper available on PDSN 

© 2008 Progress Software Corporation14 OPS-24: Success with OpenEdge Replication Sample Network Bandwidth Calculation  Hourly After Image Blocks from 5 production databases = 713mb  By calculation: Replication size = 1.5 * AI size –713mb*1.5 = mb Add replication overhead (1.1) –1069.5mb*1.1 = mb  Throughput per second: mb/3600=.32679mb/b  Change to kilobytes /s:.32679mb/s * 1000 = kb/b needed to sustain transfer

© 2008 Progress Software Corporation15 OPS-24: Success with OpenEdge Replication Replication Performance – DBService Buffer  DBService Buffer Queue must be sized appropriately – Solution P Failure to do so may result in source slowdown and target falling behind  Analyze AI activity prior to implementing Use PROMON -> R&D -> Status Display (#1)- > DB Service Manager (#16).

© 2008 Progress Software Corporation16 OPS-24: Success with OpenEdge Replication Determining Optimal -Pica  The larger the –pica, the further behind the target may get  Monitor AI writes during busiest period for one hour  -pica = (TAIW/Blockcount) * 1.25  Blockcount depends on version OpenEdge 10 – blockcount is 9.16 Progress ® 9.1x – blockcount is 18.2

© 2008 Progress Software Corporation17 OPS-24: Success with OpenEdge Replication Sample –Pica Calculation  Values up to 8192k for 10.1B01 and later  TAIW over one hour  Formula for OE 10: (34560/9.16) * 1.25 =  Why not just use 8192???

© 2008 Progress Software Corporation18 OPS-24: Success with OpenEdge Replication “Houston, we have a problem” - Promon 03/03/08 Status: Database Service Manager 17:15:02 Communication Area Size : KB Total Message Entries : Free Message Entries : 4 Used Message Entries : Registered Database Service Objects Name Rdy Status Messages Locked by OpenEdge Replication Server Y RUN OpenEdge RDBMS Y REG 0 OpenEdge DB Agent Y RUN 0

© 2008 Progress Software Corporation19 OPS-24: Success with OpenEdge Replication Agenda  Terminology and Architecture - visual  Replication performance hotspots  Replication availability hotspots  Source and target management

© 2008 Progress Software Corporation20 OPS-24: Success with OpenEdge Replication COMP-10: OpenEdge® Management & Replication – Divide et impera! Availability Hotspots Primary Machine Secondary Machine Repl Agent Database Brk/Server Processes Target Database R/O Clients R/Write Clients AI Extents DB Service Q Repl Server AI Extents Source Database Database Brk/Server Processes

© 2008 Progress Software Corporation21 OPS-24: Success with OpenEdge Replication COMP-10: OpenEdge® Management & Replication – Divide et impera! Availability Hotspots Secondary Machine Repl Agent Database Brk/Server Processes Target Database R/O Clients R/Write Clients AI Extents DB Service Q Repl Server AI Extents Source Database Database Brk/Server Processes DOWN! Severed Primary Machine Loss of Database TCP or Process Failure Locked

© 2008 Progress Software Corporation22 OPS-24: Success with OpenEdge Replication Availability Hotspots – AI Management  Replication DOES NOT manage AI files  AI Files must be emptied and backed up 10.1A AI archiver became available.  Database crashes when no empty AI extents.

© 2008 Progress Software Corporation23 OPS-24: Success with OpenEdge Replication Availability Hotspots – Locked AI Files  AI “locked” when AI notes not replicated to the target db Common issue – often caused by simple maintenance routines and failure to monitor status of replication  Configure the rpserver and rpagent to detect and handle outages  All about the *.repl.properties file!!  Configure, test well and sleep easier at night

© 2008 Progress Software Corporation24 OPS-24: Success with OpenEdge Replication Availability – Many Locked AI Files  Use dsrutil monitor against source and target db. Check for replication shared memory  If rpagent is running, restart the replication server on the source  If rpagent is not running, restart target database and restart replication server

© 2008 Progress Software Corporation25 OPS-24: Success with OpenEdge Replication “Houston, we have a problem” – part 2 Extent: 1Extent: 3 Status: BusyStatus: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a1 Path: C:\wrk101c\repl\source.a3 Size: 3192 Size: 6264 Used: 3180 Used: 6200 Start: Wed May 14 14:09: Start: Wed May 14 14:08: Seqno: 5Seqno: 3 Extent: 2Extent: 4Status: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a2 Path: C:\wrk101c\repl\source.a4 Size: Size: Used: Used: Start: Wed May 14 13:45: Start: Wed May 14 14:08: Seqno: 2Seqno: 4

© 2008 Progress Software Corporation26 OPS-24: Success with OpenEdge Replication Availability – All AI Files Locked  Source database activity comes to a halt  Can you afford to bring the db down? If not - disablesitereplication If yes – stop database – add new ai files – prostrct reorder – restart.

© 2008 Progress Software Corporation27 OPS-24: Success with OpenEdge Replication Prostrct add sourcedb addai.st Extent: 1Extent: 4 Status: BusyStatus: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a1 Path: C:\wrk101c\repl\source.a4 Size: 3192 Size: Used: 3180 Used: Start: Wed May 14 14:09: Start: Wed May 14 14:08: Seqno: 5Seqno: 4 Extent: 2Extent: 5 Status: LockedStatus: Empty Type: Variable Length Path: C:\wrk101c\repl\source.a2 Path: C:\wrk101c\repl\source.a5 Size: Size: 120 Used: Used: 0 Start: Wed May 14 13:45: Start: N/A Seqno: 2Seqno: 0 Extent: 3Extent: 6 Status: LockedStatus: Empty Type: Variable Length Path: C:\wrk101c\repl\source.a3 Path: C:\wrk101c\repl\source.a6 Size: 6264 Size: 120 Used: 6200 Used: 0 Start: Wed May 14 14:08: Start: N/A Seqno: 3Seqno: 0

© 2008 Progress Software Corporation28 OPS-24: Success with OpenEdge Replication Prostrct reorder ai sourcedb Extent: 1Extent: 4 Status: BusyStatus: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a1 Path: C:\wrk101c\repl\source.a4 Size: 3192 Size: Used: 3180 Used: Start: Wed May 14 14:09: Start: Wed May 14 13:45: Seqno: 5Seqno: 2 Extent: 2Extent: 5 Status: EmptyStatus: Locked Type: Variable Length Path: C:\wrk101c\repl\source.a2 Path: C:\wrk101c\repl\source.a5 Size: 120 Size: 6264 Used: 0 Used: 6200 Start: N/AStart: Wed May 14 14:08: Seqno: 0Seqno: 3 Extent: 3Extent: 6 Status: EmptyStatus: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a3 Path: C:\wrk101c\repl\source.a6 Size: 1 20 Size: Used: 0 Used: Start: N/A Start: Wed May 14 14:08: Seqno: 0Seqno: 4

© 2008 Progress Software Corporation29 OPS-24: Success with OpenEdge Replication Source and Target DBs Won’t Synchronize  Try restarting a few times.  -Ma, -Mn, -n match?  Changes to.properties files?  If it continues to fail, contact support Source/target db log files AI files pmmgr.properties file –arguments=-logging 2

© 2008 Progress Software Corporation30 OPS-24: Success with OpenEdge Replication Agenda  Terminology and architecture - visual  Replication performance hotspots  Replication availability hotspots  Source and target management

© 2008 Progress Software Corporation31 OPS-24: Success with OpenEdge Replication Managing Replication – DSRUTIL Monitor and Status  Not immediately obvious when a failure occurs  Proactively verify replication performance and status

© 2008 Progress Software Corporation32 OPS-24: Success with OpenEdge Replication DSRUtil Monitor  DSRUTIL source/target –C monitor Attaches to replication shared memory. –Cannot connect to replication shared memory. Status = -1 Checks status of Server and Agent –Server status –Agent status

© 2008 Progress Software Corporation33 OPS-24: Success with OpenEdge Replication DSRUtil source –C monitor OpenEdge Replication Monitor Page 1 Database: C:\wrk101c\repl\source Database is enabled as OpenEdge Replication: Source Server is: In Normal Processing Number of configured agents: 1 Delay Interval (current / min / max): 5 / 5 / 500 Recovery information: State: No recovery being performed Agents needing recovery: 0 Agents connected: 0 Agents in synchronization: 0 Transition information: Type: Manual

© 2008 Progress Software Corporation34 OPS-24: Success with OpenEdge Replication DSRUtil target –C monitor (page 1) Database: C:\wrk101c\repl\target Database is enabled as OpenEdge Replication: Target Agent: Name: agent1 ID: 1 Host name: State: Normal Processing Ready: Yes Critical: No Method: Asynchronous Agent is waiting for: Nothing Maximum bytes in TCP/IP message: 8500 Server/Agent connection time: Wed May 14 13:48: Delay Interval (current / min / max): 5 / 5 / 500 Transition information: Type: Manual The last block received at: Wed May 14 13:51: Activity information: Blocks received: 2084 Blocks processed: 2084

© 2008 Progress Software Corporation35 OPS-24: Success with OpenEdge Replication DSRUtil target –C monitor (page 2) Blocks acknowledged: 0 Notes processed: Transactions started: Transactions ended: Synchronization points: 33 AI Block Information: Source RDBMS Block (Seq / Block): 2 / 2516 Last Processed Block (Seq / Block): 2 / 2494 Latency Information: Repl Server behind Source DB by: 1 second(s) Current Source Database Transaction: Last Transaction Applied to Target: Target Current as of (Target, Source): Wed May 14 13:51: , Wed May 14 13:51: with delta of 000:00:00

© 2008 Progress Software Corporation36 OPS-24: Success with OpenEdge Replication DSRUTIL Status  DSRUTIL source/target –C status  Return code indicates current state of replication server or agent.  Good for automated scripts  Grep for return code  Replication User Guide provides details

© 2008 Progress Software Corporation37 OPS-24: Success with OpenEdge Replication Managing Replication – OE Management  OpenEdge Management 10.1B02/3.1B02+ remotely monitor log files  10.1B02 AdminServer running on the remote system  Attach OE_DB_Replication Log File Ruleset to the log file monitor  Alert/ thrown when message violating rule set appears.

© 2008 Progress Software Corporation38 OPS-24: Success with OpenEdge Replication OE Management – Replication Log File Rule Set

© 2008 Progress Software Corporation39 OPS-24: Success with OpenEdge Replication Summary  Terminology and Architecture  Replication performance hotspots  Replication availability hotspots  Source and target management

© 2008 Progress Software Corporation40 OPS-24: Success with OpenEdge Replication Success With Replication  Understand the architecture  Know your requirements  Analyze your system  Plan, Configure, Monitor  Succeed

© 2008 Progress Software Corporation41 OPS-24: Success with OpenEdge Replication Session ID: Session Title For More Information, go to…  PSDN  Knowledge Centrum ( esupport.progress.com) esupport.progress.com 3.1B01/10.1B01 upgrade: P122926, P123418, P123420, P123424, P123426, P123427, P sizing –pica: P  Documentation OpenEdge Replication 10.1C docs ( OpenEdge Management 3.1C docs (

© 2008 Progress Software Corporation42 OPS-24: Success with OpenEdge Replication Relevant Exchange Sessions  OPS-21: Managing multiple sites part time: Replication & OpenEdge Management Case Study

© 2008 Progress Software Corporation43 OPS-24: Success with OpenEdge Replication Questions ?

© 2008 Progress Software Corporation44 OPS-24: Success with OpenEdge Replication Thank You

© 2008 Progress Software Corporation45 OPS-24: Success with OpenEdge Replication