1. OPS-24: Success with OpenEdge® Replication What you don’t know can hurt you Cyril Gleiman Principal Technical Support Engineer

2. © 2008 Progress Software Corporation2 OPS-24: Success with OpenEdge Replication Obligatory (gratuitious) Quotation  An ounce of prevention is worth a pound of cure.  28.35 grams of prevention is worth.45 kilograms of cure

3. © 2008 Progress Software Corporation3 OPS-24: Success with OpenEdge Replication Success With Replication  Understand the architecture  Know your requirements  Analyze your system  Plan, configure, deploy,monitor  Succeed

4. © 2008 Progress Software Corporation4 OPS-24: Success with OpenEdge Replication Agenda  Terminology and architecture – Visual  Replication performance hotspots  Replication availability hotspots  Source and target management

5. © 2008 Progress Software Corporation5 OPS-24: Success with OpenEdge Replication Replication Terminology  Primary – original production machine  Secondary – original recovery machine  Source – production database  Target – recovery database  Replication server – rpserver (source)  Replication agent – rpagent (target)  DBService queue – stores IPC messages  Network pipe – TCP/IP rpserver -> rpagent

6. © 2008 Progress Software Corporation6 OPS-24: Success with OpenEdge Replication ! Architecture of Replication – Primary/Source Primary Machine R/Write Clients AI Extents DB Service Q Repl Server AI Extents Source Database Database Brk/Server Processes

7. © 2008 Progress Software Corporation7 OPS-24: Success with OpenEdge Replication Architecture of Replication – Secondary/Target Secondary Machine Repl Agent Database Brk/Server Processes Target Database R/O Clients

8. © 2008 Progress Software Corporation8 OPS-24: Success with OpenEdge Replication Architecture of Replication Primary Machine Secondary Machine Repl Agent Database Brk/Server Processes Target Database R/O Clients R/Write Clients AI Extents DB Service Q Repl Server AI Extents Source Database Database Brk/Server Processes

9. © 2008 Progress Software Corporation9 OPS-24: Success with OpenEdge Replication Agenda  Terminology and architecture - Visual  Replication performance hotspots  Replication availability hotspots  Source and target management

10. © 2008 Progress Software Corporation10 OPS-24: Success with OpenEdge Replication Performance Hotspots Primary Machine Secondary Machine Repl Agent Database Brk/Server Processes Target Database R/O Clients R/Write Clients AI Extents DB Service Q Repl Server AI Extents Source Database Database Brk/Server Processes

11. © 2008 Progress Software Corporation11 OPS-24: Success with OpenEdge Replication Source DB Performance Considerations  Speed/power of machine  Replication Plus – offload read only clients to target database  AI files – fixed extents  AI/BI blocksizes should be the same 16 K is the most efficient Need to truncate ai and bi

12. © 2008 Progress Software Corporation12 OPS-24: Success with OpenEdge Replication Target DB Performance Considerations  System should not be underpowered by comparison to the source system  No need for after-imaging  Read only clients

13. © 2008 Progress Software Corporation13 OPS-24: Success with OpenEdge Replication Network Performance  WAN vs. LAN  The bigger the pipe the better  Determine size of pipe  Whitepaper available on PDSN  http://tinyurl.com/6xqp78

14. © 2008 Progress Software Corporation14 OPS-24: Success with OpenEdge Replication Sample Network Bandwidth Calculation  Hourly After Image Blocks from 5 production databases = 713mb  By calculation: Replication size = 1.5 * AI size –713mb*1.5 = 1069.5mb Add replication overhead (1.1) –1069.5mb*1.1 = 1176.45mb  Throughput per second: 1176.45mb/3600=.32679mb/b  Change to kilobytes /s:.32679mb/s * 1000 = 326.79kb/b needed to sustain transfer

15. © 2008 Progress Software Corporation15 OPS-24: Success with OpenEdge Replication Replication Performance – DBService Buffer  DBService Buffer Queue must be sized appropriately – Solution P121969 Failure to do so may result in source slowdown and target falling behind  Analyze AI activity prior to implementing Use PROMON -> R&D -> Status Display (#1)- > DB Service Manager (#16).

16. © 2008 Progress Software Corporation16 OPS-24: Success with OpenEdge Replication Determining Optimal -Pica  The larger the –pica, the further behind the target may get  Monitor AI writes during busiest period for one hour  -pica = (TAIW/Blockcount) * 1.25  Blockcount depends on version OpenEdge 10 – blockcount is 9.16 Progress ® 9.1x – blockcount is 18.2

17. © 2008 Progress Software Corporation17 OPS-24: Success with OpenEdge Replication Sample –Pica Calculation  Values up to 8192k for 10.1B01 and later  34560 TAIW over one hour  Formula for OE 10: (34560/9.16) * 1.25 = 4716.1572052401746724890829694323  Why not just use 8192???

18. © 2008 Progress Software Corporation18 OPS-24: Success with OpenEdge Replication “Houston, we have a problem” - Promon 03/03/08 Status: Database Service Manager 17:15:02 Communication Area Size : 2049.00 KB Total Message Entries : 18733 Free Message Entries : 4 Used Message Entries : 18729 Registered Database Service Objects Name Rdy Status Messages Locked by OpenEdge Replication Server Y RUN 18729 OpenEdge RDBMS Y REG 0 OpenEdge DB Agent Y RUN 0

19. © 2008 Progress Software Corporation19 OPS-24: Success with OpenEdge Replication Agenda  Terminology and Architecture - visual  Replication performance hotspots  Replication availability hotspots  Source and target management

20. © 2008 Progress Software Corporation20 OPS-24: Success with OpenEdge Replication COMP-10: OpenEdge® Management & Replication – Divide et impera! Availability Hotspots Primary Machine Secondary Machine Repl Agent Database Brk/Server Processes Target Database R/O Clients R/Write Clients AI Extents DB Service Q Repl Server AI Extents Source Database Database Brk/Server Processes

21. © 2008 Progress Software Corporation21 OPS-24: Success with OpenEdge Replication COMP-10: OpenEdge® Management & Replication – Divide et impera! Availability Hotspots Secondary Machine Repl Agent Database Brk/Server Processes Target Database R/O Clients R/Write Clients AI Extents DB Service Q Repl Server AI Extents Source Database Database Brk/Server Processes DOWN! Severed Primary Machine Loss of Database TCP or Process Failure Locked

22. © 2008 Progress Software Corporation22 OPS-24: Success with OpenEdge Replication Availability Hotspots – AI Management  Replication DOES NOT manage AI files  AI Files must be emptied and backed up 10.1A AI archiver became available.  Database crashes when no empty AI extents.

23. © 2008 Progress Software Corporation23 OPS-24: Success with OpenEdge Replication Availability Hotspots – Locked AI Files  AI “locked” when AI notes not replicated to the target db Common issue – often caused by simple maintenance routines and failure to monitor status of replication  Configure the rpserver and rpagent to detect and handle outages  All about the *.repl.properties file!!  Configure, test well and sleep easier at night

24. © 2008 Progress Software Corporation24 OPS-24: Success with OpenEdge Replication Availability – Many Locked AI Files  Use dsrutil monitor against source and target db. Check for replication shared memory  If rpagent is running, restart the replication server on the source  If rpagent is not running, restart target database and restart replication server

25. © 2008 Progress Software Corporation25 OPS-24: Success with OpenEdge Replication “Houston, we have a problem” – part 2 Extent: 1Extent: 3 Status: BusyStatus: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a1 Path: C:\wrk101c\repl\source.a3 Size: 3192 Size: 6264 Used: 3180 Used: 6200 Start: Wed May 14 14:09:34 2008Start: Wed May 14 14:08:04 2008 Seqno: 5Seqno: 3 Extent: 2Extent: 4Status: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a2 Path: C:\wrk101c\repl\source.a4 Size: 230008 Size: 25208 Used: 229674 Used: 25063 Start: Wed May 14 13:45:30 2008Start: Wed May 14 14:08:24 2008 Seqno: 2Seqno: 4

26. © 2008 Progress Software Corporation26 OPS-24: Success with OpenEdge Replication Availability – All AI Files Locked  Source database activity comes to a halt  Can you afford to bring the db down? If not - disablesitereplication If yes – stop database – add new ai files – prostrct reorder – restart.

27. © 2008 Progress Software Corporation27 OPS-24: Success with OpenEdge Replication Prostrct add sourcedb addai.st Extent: 1Extent: 4 Status: BusyStatus: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a1 Path: C:\wrk101c\repl\source.a4 Size: 3192 Size: 25208 Used: 3180 Used: 25063 Start: Wed May 14 14:09:34 2008Start: Wed May 14 14:08:24 2008 Seqno: 5Seqno: 4 Extent: 2Extent: 5 Status: LockedStatus: Empty Type: Variable Length Path: C:\wrk101c\repl\source.a2 Path: C:\wrk101c\repl\source.a5 Size: 230008 Size: 120 Used: 229674 Used: 0 Start: Wed May 14 13:45:30 2008 Start: N/A Seqno: 2Seqno: 0 Extent: 3Extent: 6 Status: LockedStatus: Empty Type: Variable Length Path: C:\wrk101c\repl\source.a3 Path: C:\wrk101c\repl\source.a6 Size: 6264 Size: 120 Used: 6200 Used: 0 Start: Wed May 14 14:08:04 2008 Start: N/A Seqno: 3Seqno: 0

28. © 2008 Progress Software Corporation28 OPS-24: Success with OpenEdge Replication Prostrct reorder ai sourcedb Extent: 1Extent: 4 Status: BusyStatus: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a1 Path: C:\wrk101c\repl\source.a4 Size: 3192 Size: 230008 Used: 3180 Used: 229674 Start: Wed May 14 14:09:34 2008Start: Wed May 14 13:45:30 2008 Seqno: 5Seqno: 2 Extent: 2Extent: 5 Status: EmptyStatus: Locked Type: Variable Length Path: C:\wrk101c\repl\source.a2 Path: C:\wrk101c\repl\source.a5 Size: 120 Size: 6264 Used: 0 Used: 6200 Start: N/AStart: Wed May 14 14:08:04 2008 Seqno: 0Seqno: 3 Extent: 3Extent: 6 Status: EmptyStatus: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a3 Path: C:\wrk101c\repl\source.a6 Size: 1 20 Size: 25208 Used: 0 Used: 25063 Start: N/A Start: Wed May 14 14:08:24 2008 Seqno: 0Seqno: 4

29. © 2008 Progress Software Corporation29 OPS-24: Success with OpenEdge Replication Source and Target DBs Won’t Synchronize  Try restarting a few times.  -Ma, -Mn, -n match?  Changes to.properties files?  If it continues to fail, contact support Source/target db log files AI files pmmgr.properties file –arguments=-logging 2

30. © 2008 Progress Software Corporation30 OPS-24: Success with OpenEdge Replication Agenda  Terminology and architecture - visual  Replication performance hotspots  Replication availability hotspots  Source and target management

31. © 2008 Progress Software Corporation31 OPS-24: Success with OpenEdge Replication Managing Replication – DSRUTIL Monitor and Status  Not immediately obvious when a failure occurs  Proactively verify replication performance and status

32. © 2008 Progress Software Corporation32 OPS-24: Success with OpenEdge Replication DSRUtil Monitor  DSRUTIL source/target –C monitor Attaches to replication shared memory. –Cannot connect to replication shared memory. Status = -1 Checks status of Server and Agent –Server status –Agent status

33. © 2008 Progress Software Corporation33 OPS-24: Success with OpenEdge Replication DSRUtil source –C monitor OpenEdge Replication Monitor Page 1 Database: C:\wrk101c\repl\source Database is enabled as OpenEdge Replication: Source Server is: In Normal Processing Number of configured agents: 1 Delay Interval (current / min / max): 5 / 5 / 500 Recovery information: State: No recovery being performed Agents needing recovery: 0 Agents connected: 0 Agents in synchronization: 0 Transition information: Type: Manual

34. © 2008 Progress Software Corporation34 OPS-24: Success with OpenEdge Replication DSRUtil target –C monitor (page 1) Database: C:\wrk101c\repl\target Database is enabled as OpenEdge Replication: Target Agent: Name: agent1 ID: 1 Host name: State: Normal Processing Ready: Yes Critical: No Method: Asynchronous Agent is waiting for: Nothing Maximum bytes in TCP/IP message: 8500 Server/Agent connection time: Wed May 14 13:48:43 2008 Delay Interval (current / min / max): 5 / 5 / 500 Transition information: Type: Manual The last block received at: Wed May 14 13:51:14 2008 Activity information: Blocks received: 2084 Blocks processed: 2084

35. © 2008 Progress Software Corporation35 OPS-24: Success with OpenEdge Replication DSRUtil target –C monitor (page 2) Blocks acknowledged: 0 Notes processed: 207774 Transactions started: 10269 Transactions ended: 10269 Synchronization points: 33 AI Block Information: Source RDBMS Block (Seq / Block): 2 / 2516 Last Processed Block (Seq / Block): 2 / 2494 Latency Information: Repl Server behind Source DB by: 1 second(s) Current Source Database Transaction: 13906 Last Transaction Applied to Target: 13793 Target Current as of (Target, Source): Wed May 14 13:51:13 2008, Wed May 14 13:51:13 2008 with delta of 000:00:00

36. © 2008 Progress Software Corporation36 OPS-24: Success with OpenEdge Replication DSRUTIL Status  DSRUTIL source/target –C status  Return code indicates current state of replication server or agent.  Good for automated scripts  Grep for return code  Replication User Guide provides details

37. © 2008 Progress Software Corporation37 OPS-24: Success with OpenEdge Replication Managing Replication – OE Management  OpenEdge Management 10.1B02/3.1B02+ remotely monitor log files  10.1B02 AdminServer running on the remote system  Attach OE_DB_Replication Log File Ruleset to the log file monitor  Alert/email thrown when message violating rule set appears.

38. © 2008 Progress Software Corporation38 OPS-24: Success with OpenEdge Replication OE Management – Replication Log File Rule Set

39. © 2008 Progress Software Corporation39 OPS-24: Success with OpenEdge Replication Summary  Terminology and Architecture  Replication performance hotspots  Replication availability hotspots  Source and target management

40. © 2008 Progress Software Corporation40 OPS-24: Success with OpenEdge Replication Success With Replication  Understand the architecture  Know your requirements  Analyze your system  Plan, Configure, Monitor  Succeed

41. © 2008 Progress Software Corporation41 OPS-24: Success with OpenEdge Replication Session ID: Session Title For More Information, go to…  PSDN www.psdn.com/library/kbcategory.jspa?categoryID=21 www.psdn.com/library/kbcategory.jspa?categoryID=334  Knowledge Centrum ( esupport.progress.com) esupport.progress.com 3.1B01/10.1B01 upgrade: P122926, P123418, P123420, P123424, P123426, P123427, P123676 sizing –pica: P121969  Documentation OpenEdge Replication 10.1C docs (www.psdn.com)www.psdn.com OpenEdge Management 3.1C docs (www.psdn.com)www.psdn.com

42. © 2008 Progress Software Corporation42 OPS-24: Success with OpenEdge Replication Relevant Exchange Sessions  OPS-21: Managing multiple sites part time: Replication & OpenEdge Management Case Study

43. © 2008 Progress Software Corporation43 OPS-24: Success with OpenEdge Replication Questions ?

44. © 2008 Progress Software Corporation44 OPS-24: Success with OpenEdge Replication Thank You

45. © 2008 Progress Software Corporation45 OPS-24: Success with OpenEdge Replication