Jos van Lammeren

 Background  GoldenGate Overview Setup Operations Troubleshooting Monitoring Challenges  Q&A

 Using Oracle GoldenGate since 2010  At clients Oracle 9i, 10g near-zero downtime migrations to 11g RAC Oracle 11g RAC setup for DR and EDW load Upgrade OGG 10.4 to 11.2 for Oracle 11g clusters (no app downtime) Oracle 9i, 10g, 11g EDW load SQL Server 2008 to Oracle 11g EDW load  At LamkoDB Oracle 10g, 11g, 12c, SQL Server 2008, 2012, MySQL tests with OGG 11g and 12c

 What GoldenGate does Continuous data replication from one database to another Source and target databases can be of different vendors/releases Non-intrusive, low-impact, sub-second latency Maintains transactional integrity - Resilient against interruptions and failures Data transformation on the fly Few or many tables in a schema DDL replication if desired Used for EDW feeds, DR, DB version upgrades, DB migrations  What GoldenGate does not Is not a replacement for Data Guard

5

 Use of replication

 Source Databases/Platforms supported

 Target Databases/Platforms supported

 Processes/Files involved  Classic Capture or Integrated Capture mode  Nonintegrated Replicat or Integrated Replicat

 Observations Not all data types are supported Need data type translation in heterogeneous environments Encryption of data in files and across network Optional compression of data copied by pump process Cryptic names for processes: 8 characters Cryptic names of trail files e1xxxxxx (2 characters followed by incrementing number) Administrator must like command line tool (similar to sqlplus)

 Installation GG software (source & target server) Download software from Oracle site. Software to use is Oracle DBMS release specific. On Linux/Unix create account that owns the software & add to dba group Create profile to set $ORACLE_HOME, $PATH, etc. for the account Unzip & untar software or use Oracle Universal Installer Configure GLOBALS and mgr.prm files  Installation in DB DB in archive log mode (at least source instance) Turn off recyclebin (in source instance for DDL replication) Put DB in supplemental logging mode (source instance) Run OGG supplied installation scripts (source & target)

 How many schemas/tables to replicate & how much data will be replicated?  Have tables the same structure in source and target database or do target tables have additional columns?  Do tables have PKs?  Are there referential constraints to deal with if not all tables are being replicated?  Delete Cascade constraints  When bi-directional replication Sequences Triggers firing on either database

 Setup replication flow & initial load of data Create/start Extract on source database Create/start Pump process on source database Do initial data load from source to target database Create/start Replicat on target database

 Extract Extract process E1ORA1 E1ORA1 extracts data from MMDBORA1 (Oracle) to local trail files in directory/gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1

EXTRACT E01ORA1.obey ADD EXTRACT E01ORA1, TRANLOG, BEGIN NOW, params /gg/product/11.2.1_11g/ggs/dirprm/MMDBORA1/E01ORA1.prm ADD EXTTRAIL /gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1, EXTRACT E01ORA1 E01ORA1.prm EXTRACT E01ORA1... USERID ggate, PASSWORD abc EXTTRAIL /gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1 megabytes TABLE CHINOOK.*; start E01ORA1

 Pump Pump process P1ORA1 copies local trail file contents to remote trail files on another server in directory C:\gg\dirdat\MMDBORA1\p1

PUMP P01ORA1.obey ADD EXTRACT P01ORA1, EXTTRAILSOURCE /gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1, params /gg/product/11.2.1_11g/ggs/dirprm/MMDBORA1/P01ORA1.prm ADD RMTTRAIL C:\gg\dirdat\MMDBORA1\p1, EXTRACT P01ORA1, MEGABYTES 100 P01ORA1.prm EXTRACT P01ORA1 RMTHOST , MGRPORT 7809 RMTTRAIL C:\gg\dirdat\MMDBORA1\p1, megabytes 100 PASSTHRU TABLE CHINOOK.*; start P01ORA1

 Initial load of data Export/Import Data pump RMAN Load data over a DB link bcp OGG utilities many more …

 Replicat Replicat process R01ORA1 (SQL Server 2012) reads the remote trail file contents and applies rows to the database

REPLICAT R01ORA1.obey DBLOGIN SOURCEDB GGATE, USERID ggate, PASSWORD abc add replicat R01ORA1, exttrail C:\gg\dirdat\MMDBORA1\p1 params C:\gg\dirprm\CHINOOK\R01ORA1.prm R01ORA1.prm REPLICAT R01ORA1 SOURCEDB GGATE, USERID ggate, PASSWORD abc... MAP CHINOOK.ALBUM, TARGET dbo.ALBUM; MAP CHINOOK.ARTIST, TARGET dbo.ARTIST; MAP CHINOOK.CUSTOMER, TARGET dbo.CUSTOMER; MAP CHINOOK.TRACK, TARGET dbo.TRACK; start R01ORA1

 Replication flow complete

 ggsci command line interface info all info * or info E*, etc info E01ORA1 info E01ORA1 detail start E01ORA1 stop E01ORA1 stats E01ORA1 send E01ORA1 status many more …

ggsci commands GGSCI (MMDBLNXDB01) 1> info all Program Status Group Lag at Chkpt Time Since Chkpt MANAGER RUNNING EXTRACT RUNNING E01ORA1 00:00:00 00:00:04 EXTRACT RUNNING E02ORA1 00:00:00 00:00:04 EXTRACT RUNNING P01ORA1 00:00:00 00:00:09 EXTRACT RUNNING P02ORA1 00:00:00 00:00:00 REPLICAT RUNNING R01SQL1 00:00:00 00:00:04 GGSCI (MMDBLNXDB01) 2> info E01ORA1 EXTRACT E01ORA1 Last Started :49 Status RUNNING Checkpoint Lag 00:00:00 (updated 00:00:02 ago) Log Read Checkpoint Oracle Redo Logs :52:56 Seqno 3726, RBA SCN ( )

 Network issues causing pump to crash Usually restart pump Use AUTORESTART in mgr.prm

 Network issues causing pump to crash Usually restart pump Use AUTORESTART in mgr.prm  Server reboot and processes not starting Use AUTOSTART in mgr.prm

 Network issues causing pump to crash Usually restart pump Use AUTORESTART in mgr.prm  Server reboot and processes not starting Use AUTOSTART in mgr.prm  Extract performance bad Use dedicated volumes where redo/archived logs are stored Have separate volumes for local trail files Use multiple extracts

 Network issues causing pump to crash Usually restart pump Use AUTORESTART in mgr.prm  Server reboot and processes not starting Use AUTOSTART in mgr.prm  Extract performance bad Use dedicated volumes where redo/archived logs are stored Have separate volumes for local trail files Use multiple extracts  Pump performance bad Compress data when sending data across Have separate volumes for local/remote trail files Use multiple extracts/pumps

 Replicat performance bad Check for tables with missing PKs Use batchsql in prm file of replicat Have separate volumes for remote trail files Use multiple replicats

 Replicat performance bad Check for tables with missing PKs Use batchsql in prm file of replicat Have separate volumes for remote trail files Use multiple replicats  Disk full for local and/or remote trail files Occasionally need to setup replication again, with or without initial load Monitor disk space usage by storage group or sysadmins Have separate volumes for local and remote trail files Use PURGEOLDEXTRACTS in mgr.prm

 Crashing replicats when tables added/changed Add/modify tables on target database, possibly update data definition file, and restart replicat Make extract process stop when it notices DDL

 Crashing replicats when tables added/changed Add/modify tables on target database, possibly update data definition file, and restart replicat Make extract process stop when it notices DDL  Extract process does not stop Commit open transactions in DB Kill session having open transaction Force stop of Extract process if not many archived logs to re-read at next Extract startup

 Crashing replicats when tables added/changed Add/modify tables on target database, possibly update data definition file, and restart replicat Make extract process stop when it notices DDL  Extract process does not stop Commit open transactions in DB Kill session having open transaction Force stop of Extract process if not many archived logs to re-read at next Extract startup  Crashing replicats because of manual data changes in target database Do more manual changes to fix the data Possibly reload affected tables Educate users

 Where to look for information Report file of process (view report ) Discard file OGG Error log Oracle alert log / SQL Server ERRORLOG  Compare source and target data after fixing issue Veridata select data from source table MINUS select data from target table Oracle DBMS_COMPARISON Other tools (TOAD, …)

 Setup Heartbeat in replication flow Document ID: OGG_Best_Practice_-_heartbeat_table_using_DBMS_SCHEDULER-_V11_0_ID pdf Update heartbeat record in source database (every minute) Extract picks up record in source database, adds information and write to local trail file Pump picks up record, adds information and write to remote trail file Replicat picks up record, adds information and writes to target database (as an insert)

 Monitor using Oracle tool (Director) Need to install Java agent in every GG instance & maintain bunch of servers to keep tool running (Weblogic, …)  (Remote) Monitor using own developed tools Monitor status of processes Monitor number of rows replicated Report on Heartbeat data Queries against heartbeat table in target database Graphs generated based on data in heartbeat table (lag times) ggsci-like GG Dashboard for all GG instances monitored

 OGG Management Pack

 LamkoDB Monitoring

 Historical stats for Extract process

 Lag times for Extract/Pump/Replicat

 GG Dashboard

 GG process detail information

 GG process relationships

 Who supports GoldenGate? DBA Middleware group

 Who supports GoldenGate? DBA Middleware group  What to do about table structure changes software releases need to be evaluated/tested/coordinated

 Who supports GoldenGate? DBA Middleware group  What to do about table structure changes software releases need to be evaluated/tested/coordinated  What do developers, support, clients know about OGG? Educate personnel

 Who supports GoldenGate? DBA Middleware group  What to do about table structure changes software releases need to be evaluated/tested/coordinated  What do developers, support, clients know about OGG? Educate personnel  Monitor what is running and what is broken Have proper monitoring in place with alerting

 Who supports GoldenGate? DBA Middleware group  What to do about table structure changes software releases need to be evaluated/tested/coordinated  What do developers, support, clients know about OGG? Educate personnel  Monitor what is running and what is broken Have proper monitoring in place with alerting  Keeping track of new, existing and changing replication flows. Number of processes quickly becomes a spaghetti bowl. Have automated tool to show relationships

 Questions? Jos van Lammeren