2. Diagnostics and Verification of Oracle Real Application Clusters 10g Jack Cai Principal Product Manager Oracle Corporation 40248

3. Agenda  Real Application Clusters 10g is for Everyone  Benefits of RAC Diagnostics and Verification  Diagnostics Framework  Cluster Verification Framework  Q&A

4. Oracle RAC 10g for Everyone  Foundation of Oracle grid computing  Much easier to use and manage – Full stack reduces complication  Clusterware  Cluster filesystem  Automatic storage manager – Integral to Enterprise Manager – Workload management with policy based service – Cluster diagnostics and verification

5. Diagnostics & Verification Benefits Improve problem resolution time  Reduce need to reproduce problems  Eliminate problems related to incorrect cluster configuration  Improve quality of services through a more systematic approach

6. Diagnostics Framework  Trace instrumentation of Database Server  Clusterization of oradebug, hang analyzer  Trace loader  Navigator  Equally useful on single instance database  For database server internal diagnostics ONLY – Not for diagnosis of application problems/tuning – Use them ONLY under Oracle support supervisions

7. Diagnostics Framework Goals  Generate enough diagnostic data for first pass analysis  Minimal overhead to database server  Easy to use – Single system view of a cluster – Tool for data gathering, conversion and repository – Tool for display, navigate and analysis

8. DIAG Process Instance 1 SGA Diagnostics Architecture (Cluster View) Off-line Tools On-line Tools Trace Files Oradebug X$VIEWs Trace Navigation Tool Trace Loader DIAG Process Instance 2 SGA DIAG Process Instance 3 SGA

9. Diagnostics Architecture (Instance) Off-line Tools On-line Tools X$ Views Trace Buffer Pi Trace Buffer Pj Process Pi Process Pj DIAG Process SGA Trace Files Instance Trace Navigational Tool Trace Loader Oradebug

10. Flexible Trace Instrumentation  Focus on issues encountered by customers – Instrument trace on events Oracle support handles most frequently – Foundation laid in 9i, more component instrumentation in 10g  Trace control – Trace is event based, events can be anything within a program – Trace is controllable on a per process basis – 1000 event id (10000-10999), 256 opcodes, 256 levels – Trace is generated to SGA buffer rather than directly to files – X$Trace, X$Trace_events can be queried during run time  “Always-on” minimal tracing  Optional trace archiving

11. Trace Control Example  Dynamic control – ALTER TRACING  ON  OFF  ENABLE  DISABLE  FLUSH – Examples  alter tracing enable 12345: 5: 32  alter tracing disable 12345:5:BGS  alter tracing flush 32

12. oradebug Is Cluster Aware  Single system image of the cluster  Commands are clusterized  DIAG based implementation – Damen process – Minimal overhead  HANGANALYZE analyzes the entire cluster  Examples SQL> oradebug -g /ALL SQL> oradebug setinst “3 4”

13. Offline Trace Loader Offline trace loader converts binary trace files into text format, and loads files into Oracle databases for archiving Better data analysis through power of database queries Build a repository for problem tracking Facilitate trace data exchange in cross-platform environments

14. Offline Trace Loader  Extracts trace data from archived trace files  Converts binary trace data to text  Loads trace data into database as trace repository  Simple command line interface – trcldr [option,...] – trcldr mode=append ifile=f1.trw userid=scott/tiger Raw Trace Files (in binary or text format) Converted Trace Files (in text format) Target Database Data ConversionData Loading Trace Loader Trace Loader

15. Trace Navigation Tool  Navigate traces with links and highlights across cluster – send/recc, malloc/free – User Defined Protocol Navigation  Coloring of traces based upon attributes  Well-defined queries against keywords  Cluster wide time-ordered event diagram

16. Navigation Tool Protocol Trace

17. Trace Navigation

18. Navigation Tool Event Diagram

19. Summary of Diagnostics  Diagnostics facilities allow support to resolve issues more quickly – First pass failure diagnosis  Allows developers to find bugs faster  Tools are not meant for end users  End users should only use the tools under guidance from Oracle support personnel

20. Cluster Verification Goal  Eliminate errors through: – Pre/post validation of installation steps and/or configuration changes – Ongoing cluster validation  Flexible – API invoked by other programs  OUI, OEM, DBCA, ASM – Command line interface – Can be run anytime  To be released post Oracle Database 10g

21. Cluster Verification Framework  Verification categories – Stage verification – Component verification  Diagnostic mode  API for vendor verification tools – Utilizing vendor platforms’ verification capabilities

22. Stage Verification  RAC deployment process is divided into stages – Stages are steps of installation/configuration changes – All RAC deployment stages are verified – Verification performs pre-stage and post-stage checks  Example stages – Cluster filesystem setup – Clusterware setup – Database installation – Cluster node addition – Database configuration

23. Component Verification  RAC is divided into key components – Each component can be individually verified  Examples – Node connectivity – Shared storage accessibility – Cluster integrity – System requirements – CFS integrity

24. Diagnostic Mode  Attempt to pinpoint root cause of a verification failure in diagnostic mode  Applicable to stage and component verification  Specific testings are performed when a verification fails  Only triggered by verification failure

25. How Verification is done  Command line tool: cluvfy – cluvfy stage –post HWOS (verifies hardware and OS stage) – cluvfy comp sys –p RDBMS (verifies system components for RDBMS installation)  APIs by other relevant tools

26. Summary Diagnostics speed up problem resolutions Cluster verification eliminates incorrect configuration Ultimately reducing unscheduled system down time and improving system availability  Diagnostics tools should ONLY be used under support guidance  End Users can use cluvfy to perform verifications

27. Next Steps….  Recommended sessions – 40131: Oracle Real Application Clusters 10 g : The Fourth Generation – 40132: Oracle Real Application Clusters 10 g : Manage your cluster Without it Managing You  Recommended demos and hands-on labs – Visit DEMOgrounds under Real Application Clusters  See Your Business in Our Software – Visit the DEMOgrounds for more information.  Relevant web sites to visit for more information – http://otn.oracle.com/products/database/clustering/content.html

28. Reminder – please complete the OracleWorld online session survey Thank you.

29. A Q & Q U E S T I O N S A N S W E R S