2. GoldenGate Monitoring and Troubleshooting
Lorena Lobato Pardavila - CERN
Replication Technology Evolution for ATLAS Data Workshop

3. Outline Database level Monitoring GoldenGate Troubleshooting Questions
Oracle GoldenGate Director and Oracle GoldenGate Enterprise Manager Plug-in
Database level Monitoring
GoldenGate Troubleshooting
Questions
Director no demo because of web is not completed…
Replication Technology Evolution for ATLAS Data Workshop

4. Outline Database level Monitoring GoldenGate Troubleshooting Questions
Oracle GoldenGate Director and Oracle GoldenGate Enterprise Manager Plug-in
Database level Monitoring
GoldenGate Troubleshooting
Questions
Replication Technology Evolution for ATLAS Data Workshop

5. Oracle GG Director: Overview
Multi-tiered, client-server application that enables the configuration and management of Oracle GoldenGate instances from a remote client
OGG Director Server Domain
GGSCI
OGG Director Web
OGG Director Server Application
GGSCI
OGG Director Client
GGSCI
OGG Director Administrator
In order to administer OGG, you have to open many SSH windows for each server on which GG is running. But there is a monitoring tool, which helps you to administer all processes from one instance.OGG Director is a multi-tiered GUI application that manages the GG processes without the need to login to the GG server and easier than ggsci command line, as it show a view the near real-time lag on each procss and a overview of the whole system. Director consists of several components:
We have an OGG Director server domain which is composed of OGG Director server application, and a monitor agent and this Server Domain is running WebLogic Server. Is not needed running it but it is needed having all their components installed. At the same time, this Domain is connected with a Repository database which stores all the configurations and the statistics in clear text format.
Then, on one hand, we have that Director can link to many manager processes running on OGG instances. The Manager IP address and port number are the only required parameters for the Server to connect with them.
On the other hand, we have set of clients
OGG Director Administrator which is used to set up all the GGSI instance configurations. All of this information is stored in the Director database. From this client is when database connection are established.
Then we have OGG Director Client where we can create change or delete the parameter files and trail files. It also displays online the system overview and it´s the main part of this system considering that you can control processes from here.
You can control and monitors them on Director web too but not manipulate them, indeed .
So after this overview, let´s go to check the system requirements needed for installing OGG Director
Monitor Agent
Clients
OGG Instances
OGG DIRECTOR DATABASE
Replication Technology Evolution for ATLAS Data Workshop

6. System Requirements Oracle GoldenGate Director Server
1 GB of RAM up to 1.5 GB (recommended) of free disk space.
Dedicated http port (default port is 7001)
DYNAMICPORTLIST in each Manager process
Oracle WebLogic Server 11g Standard Edition
10.3.1, , , , or or 12c (12.1.1)
Oracle GoldenGate Director Web
Supports Microsoft Internet Explorer v6.0 or later and Mozilla Firefox v2.0 or later
No software is installed on the client system for the OGG Web
Oracle Director Desktop Client
Same version as that Oracle GoldenGate Director Server
Install Java Runtime Environment (JRE) on each cliente host
Security requirements
Oracle GoldenGate Director Server connects to remote Oracle GoldenGate instances
through each of their Manager ports. If a firewall is present within the Oracle
GoldenGate network, do the following:
1. Use the DYNAMICPORTLIST parameter in each Manager parameter file to specify a
list of available ports. For usage and syntax, see the Oracle GoldenGate Windows and
UNIX Reference Guide or the Oracle GoldenGate HP NonStop Reference Guide, as
appropriate for the platform.
2. Open those ports, plus the Manager port and the Oracle GoldenGate Director
Server http port, through the firewall.
Replication Technology Evolution for ATLAS Data Workshop

7. Oracle GG Director: Overview
OGG Director Administrator Tool
Replication Technology Evolution for ATLAS Data Workshop

8. Oracle GG Director: Overview
OGG Director Desktop Client
Replication Technology Evolution for ATLAS Data Workshop

9. Oracle GG Director: Overview
OGG log
Replication Technology Evolution for ATLAS Data Workshop

10. Oracle GG Director: Overview
Alerts configuration
Replication Technology Evolution for ATLAS Data Workshop

11. Oracle GG Director: Web
Server_domain/acon
Replication Technology Evolution for ATLAS Data Workshop

12. OGG Enterprise Manager Plug-in (Live)
For installing the plug-in:
Enterprise Manager Cloud Control 12c Bundle Patch 1 ( ) and later
Oracle GoldenGate 11g Release 2 ( ) and later
Management features:
Monitor Oracle GoldenGate instances.
Gather configuration data and track configuration changes for Oracle GoldenGate instances.
Raise alerts and violations based on thresholds set on monitored targets and configuration data.
Support monitoring by a remote Agent. A Local Agent is an agent running on the same host as the Oracle GoldenGate instance.
Nice login
Replication Technology Evolution for ATLAS Data Workshop

13. OGG Enterprise Manager Plug-in
Summary
Replication Technology Evolution for ATLAS Data Workshop

14. OGG Enterprise Manager Plug-in (Live Demo)
Replication Technology Evolution for ATLAS Data Workshop

15. Outline
Oracle GoldenGate Director and Oracle GoldenGate Enterprise Manager Plug-in
Database level Monitoring
GoldenGate Troubleshooting
Questions
Replication Technology Evolution for ATLAS Data Workshop

16. Outline Database level Monitoring GoldenGate Troubleshooting Questions
Oracle GoldenGate Director and Oracle GoldenGate Enterprise Manager Plug-in
Database level Monitoring
GoldenGate Troubleshooting
Questions
As databases professionals, one of the most difficult aspects of working with OGG is how to troubleshoot and resolve errors that occur throughout the lifecycle of the implementation. Due to the nature of OGG functionality and operations as enterprise software, it involves all layers of the technology stack from source to target database to network issues and more. The goal of this chapter is to provide the best possible overall framework to identify and resolve problems that occur within the OGG.
We are going to review the most common problems by using a holistic approach to troubleshooting. Because OGG touches multiple areas including database, network, storage and operating systems, we’ll focus on these areas as related
Replication Technology Evolution for ATLAS Data Workshop

17. Troubleshooting Network Hardware Storage/Disk OGG REPORTS
Operating System
Database
Application Server
Let´s go to take a visual approach to OGG root-cause analysis. What does GoldenGate have to trave and identify the source of problems from this analysis.
Storage/Disk
OGG REPORTS
ORACLE DATABASES
LOGDUMP
Replication Technology Evolution for ATLAS Data Workshop

18. OGG Reports Oracle GoldenGate Process Failures
GGSCI> info all
GGSCI> info <process_name>, detail
GGSCI> view report <process_name>
Oracle GoldenGate Trail Files Problems
Trail files that don’t roll over: info exttrail *
Trail files purging issues: PURGEOLDEXTRACTS
The first type of problem within OGG environments are process failures. These may occur on the source, target or on both environments for one or more processes.
Replication Technology Evolution for ATLAS Data Workshop

19. OGG Reports Oracle GoldenGate Error Log Analysis : ggserr.log
GGSCI> view ggsevt
Configuration issues
Incorrect sofware version installed for OGG
Database configuration issues on the source and/or target with OGG
OGG parameter file configuration issues
Operating system configuration issues
Network configuration issues
Configuration issues:
From second points
*Database configuration issues on the source and/or target with OGG, like can be
-No DDL script installed
-No checkpoint table specified for replpicat
-No minimum supplemental loggin is enabled…
*OGG parameter file configuration issues
-Unable to open exttrail file
-Bad written
*Operating system configuration issues
-LD_LIBRARY problems
-ORACLE_HOME
*Network configuration issues
-Network access and connectivy
-Network latency
-Network data-transfer issues
-tnsnames.ora
-firewall closed
Replication Technology Evolution for ATLAS Data Workshop

20. Oracle Database issues with GoldenGate
Extract errors = logminer problems
Replicat issues = streams apply problems
Data-synchronization issues
Missing column and rows errors
Procedures the same like for Streams!
Repeat: there are may problems but we don´t have time to envolve all of them so let´s try to focus in the headers
Extract
Can’t access Oracle database archive and redo logs
Failure condition due to Oracle source database issues
Data Pump errors
Replicat issues
Errors on the Oracle database target system
Hangs on the target system
Experiences an abends failure on the target system
Fails on large transactions
Incompatible record errors with a replicat
Data-synchronization issues
Missing column errors
Fetch failures
Replication Technology Evolution for ATLAS Data Workshop

21. LogDump
GroupID
Tokens
Trail Info
Signature, Compatibility, Character Set, Creation Time, URI, URI History, File Name, Multi-part, Trail Sequence Number, File Size, First CSN, Last CSN, First IO Time, Last IO Time, Log BSN
Machine Info
Operating System Name, Node Name, Release, Version, Hardware
Database Info
Vendor, Name, Character Set, Major Version, Minor Version, Version String, Client Character Set, Client Version String, Client NCharacter Set, Db Locale, Db NCharacter Set, Db Object Metadata, Db Time Zone
Producer Info
Name, Data Source, Major Version, Minor Version, Maintenance Level, Bug Fix Level, Build Number, Version String
Continuity Info
Recovery Mode
OGG LogDump utility is the Swiss Army knife of analysis and troubleshooting. It´s complicated to understand it but it´s a potent tool in this field. By using it. We can view data in transactions present for a trail file, identify missing transactions, and troubleshoot replication issues (as duplicate and lost data transactions).
As you can see here is the Trail File structure. To fully comprehend the data available via Logdump, one must first know something about the OGG Trail and its record layout. The following diagram provides a simplified view of an OGG Trail.
The File Header contains a wealth of information about the physical environment that produced the Trail file and Trail file contents. This information includes (CLICK)
Each source operation captured is encapsulated into a unique transaction record. This record consists of a GoldenGate Header (GHDR), variable length data area, and GGS and User Tokens.
GHDR
The GoldenGate Header, or GHDR, area details information about the source operation and transaction. This includes the source commit timestamp (IO Time token), record length, database operation type (IO Type), whether this record is a before or after data image, and source table name. The GHDR also contains information that denotes each record as the beginning record of a transaction, the ending record of a transaction, a record between the beginning and end of a transaction, or a single record transaction.
Data Area
The variable length data area contains the atomic data for the source database operation. This includes the column index number and data changed for an after image, or original column data for a before image.
GGS Tokens
GoldenGate Tokens written for each transaction record include the source database row id, source database change number (LOGCSN), and source database transaction id (TRANID).
User Tokens
User Tokens are data defined by the user and added to the Trail via the column conversion function.
Now that we know the basic layout for an OGG Trail, lets see what information we can view via Logdump.
Replication Technology Evolution for ATLAS Data Workshop

22. LogDump ./logdump Open /trail_file_full_path count
Standard utility that comes with Oracle GoldenGate
Enables to the user to view and scan for transactions and records
Obtains transaction statistics and save data contained within OGG Trails
Identify missing transactions
Troubleshoot replication issues
Getting Started With Logdump
Logdump is run from a terminal window. When the application starts, a banner is displayed showing the version and copyright information, followed by the Logdump prompt. The command open [trail] will create a shared read file handler to the designated trail file.
The command count will return data for all records contain in the trail. This data includes, total data bytes, average data bytes per record, totals for each type of database operation, the number of before and after image records, average number of transactions in the Trail, average data bytes per transaction, and average number of records per transaction
For the picture
This is great as an overview, but it does not provide any level of real detail about the Extract’s workload. The command detail on tells Logdump to display the totals and averages shown above and then sort the data and display it for each unique table contained within the Trail.
“next”
Replication Technology Evolution for ATLAS Data Workshop

23. THANK YOU! lorena.lobato@cern.ch
Replication Technology Evolution for ATLAS Data Workshop