2. Donna C. Hamby Sr. Principal Instructor Oracle University Oracle Corporation

3. Oracle9i Real Application Clusters

4. What Is a Cluster? – Interconnected nodes – Cluster software hides the structure – Disks available for read and write by node or nodes requiring the data they contain Node Disks Interconnect

5. What Is Oracle9 i Real Application Clusters? – Database with instances on separate nodes – Physical or logical access to each database file – Software controls access to data Database files Instance on each node

6. Cluster Hardware Components – Nodes – Interconnect – Shared disk subsystem

7. Interprocess Communication Software – Is the second main subcomponent of the OSD clusterware – Controls messages across the interconnect – Transfers messages and data blocks for Real Application Clusters OSD Clusterware Cluster Manager Node monitor IPC OSD Clusterware Cluster Manager Node monitor IPC

8. Disk Access  Shared disk access is required for – Control files – Data files – Online redo log files – A quorum (or voting) disk on some systems – Configuration data – Optionally, a server parameter file (SPFILE) Instance OSD Clusterware Connections to other nodes

9. Disk Access  Optional features requiring shared disk access include: – Configuration information for management tools – Server parameter file – Separate undo tablespace for each instance if using automatic undo management Connections to other nodes Instance OSD Clusterware

10. Global Services Daemon – Coordinates with other utilities and tools to manage Real Application Clusters databases and instances – One GSD supports all databases on a node – Requires no configuration OSD Clusterware Instance GSD

11. Background Processes  Additional background processes for a Real Application Clusters database consist of – LMON – LMD – LMSn – LCK – DIAG Instance LMS0 LMON LCK LMD DIAG OSD Clusterware

12. Global Enqueue Service Monitor – Monitors global enqueues and resources across the cluster – Performs recovery operations for cluster enqueues Instance LMS0LMON LCK LMD DIAG OSD Clusterware

13. Global Enqueue Service Daemon – Manages access to global enqueues and resources – Handles resource requests from other instances Instance LMS0LMON LCK LMD DIAG OSD Clusterware

14. Global Cache Service Processes – Manage access to data blocks across the cluster – Ship block images between the buffer caches of different instances (Cache Fusion) Instance LMS0LMON LCK LMD DIAG OSD Clusterware

15. LCK Process – Assists LMSn processes with requests for resources between instances – Handles cache resource request that do not involve Cache Fusion: dictionary and row cache Instance LMS0LMON LCK LMD DIAG OSD Clusterware

16. Diagnosability Daemon – Captures diagnostic information related to instance process failures – Logs information to the directory specified by BACKGROUND_DUMP_ DEST – Process cannot be configured and should not be stopped Instance LMS0LMON LCK LMD DIAG OSD Clusterware

17. GSD Management – Start a GSD on all the nodes in your Real Applications Clusters database for use by the management tools including  SRVCTL  the DBCA  OEM – Run only one GSD on each node gsdservice -install gsdservice -start gsdservice -remove gsd Successfully started the daemon on the local node. UNIX Windows

18. Server Control Utility  The Server Control Utility (SRVCTL) – Is the preferred tool to administer your Real Application Clusters database environment – Manages cluster database configuration information used by several Oracle tools – Provides cluster database management commands – Requires the GSD to be running Node 2 GSD SRVCTL Agent Node 1 GSD SRVCTL Agent

19. SRVCTL Command Syntax srvctl Usage: srvctl {start|stop|status|config|add|delete|rename| move|get|set|unset} [options] srvctl config -h Usage: srvctl config [-p [-n ]][-D ] [-h] [-V] -p Show configuration for specified cluster database... -n Only show services on named node -D Debug level -h Print usage -V Show version

20. SRVCTL Cluster Database Configuration Tasks  Modify database configuration information – Add and delete cluster databases – Add instances to and delete instances from a cluster database – Rename instances – Move instances – Set and unset the environment for an entire cluster database – Set and unset instance environments

21. Add and Delete Databases  Add database configuration information  Remove database configuration information srvctl add db –p T2 –o $ORACLE_HOME srvctl delete db –p T1

22. Add and Delete Instances  Add instance configuration information  Remove instance configuration information srvctl add instance –p T2 –i T2N3 –n raclust3 srvctl delete instance –p T2 –i T2N1

23. Move and Rename Instances  Move instance to new node in the static configuration  Rename instance in the repository srvctl move instance -p T2 -i T2N2 -n raclust1 srvctl rename instance -p T2 -i T2N2 -e T2N1

24. Store and Remove Environment Data  Store database-wide environment setting  Store instance-specific environment setting  Remove environment setting from single instance srvctl set env -p T2 -t LANG=gb -i T2N1 srvctl set env -p T2 -t LANG=us srvctl unset env -p T2 -t LANG -i T2N3

25. SRVCTL Cluster Database Tasks  Manage cluster database components – Start cluster databases and instances – Stop cluster databases and instances – Start and stop listeners associated with a cluster database – Start and stop listeners associated with a cluster database instance – Obtain the status of a cluster database instance

26. Starting Databases and Instances  Start all instances and listeners  Start two named instances with their listeners  Start listener only on one node srvctl start -p T2 –i T2N1,T2N2 srvctl start -p T2 srvctl start –p T2 –n raclust3 –s lsnr

27. Starting Databases and Instances  Start all instances with nondefault connect string  Start listeners and open instances with special parameter file on two nodes srvctl start -p T2 –n raclust1,raclust2 \ –o pfile=/ora/admin/initADMIN.ora srvctl start -p T2 –c 'admin/adm1n as sysdba'

28. Stopping Databases and Instances  Stop an instance only, not the listener  Shut down instances and listeners on three nodes using a nondefault connect string srvctl stop -p T2 \ –n raclust2,raclust3,raclust4 \ –c admin/adm1n as sysdba srvctl stop –p T2 –i T2N3 –x lsnr

29. SRVCTL Cluster Database Tasks  Manage cluster database components – Start cluster databases and instances – Stop cluster databases and instances – Start and stop listeners associated with a cluster database – Start and stop listeners associated with a cluster database instance – Obtain the status of a cluster database instance

30. Starting Databases and Instances  Start all instances and listeners  Start two named instances with their listeners  Start listener only on one node srvctl start -p T2 –i T2N1,T2N2 srvctl start -p T2 srvctl start –p T2 –n raclust3 –s lsnr

31. Starting Databases and Instances  Start all instances with nondefault connect string  Start listeners and open instances with special parameter file on two nodes srvctl start -p T2 –n raclust1,raclust2 \ –o pfile=/ora/admin/initADMIN.ora srvctl start -p T2 –c 'admin/adm1n as sysdba'

32. Stopping Databases and Instances  Stop an instance only, not the listener  Shut down instances and listeners on three nodes using a nondefault connect string srvctl stop -p T2 \ –n raclust2,raclust3,raclust4 \ –c admin/adm1n as sysdba srvctl stop –p T2 –i T2N3 –x lsnr

33. Inspect Status of Cluster Database  List the active instances  Show the status of two instances  Display the status of the listener on one node srvctl status -p T2 –i T2N1,T2N4 –s inst srvctl status -p T2 srvctl status –p T2 –n raclust2 –s lsnr

34. Inspect Database Configuration Information  List the instances defined for a database  List environment information for a database  List environment information for an instance srvctl get env -p T2 srvctl config -p T2 srvctl get env -p T2 –i T2N4

35. Cache Fusion. Cache Fusion helps provide transparent scalability in a Real Application Clusters database The algorithms enable transportation of block images between instances Cache Fusion services track the current location and status of resources Directory structures within the SGA of each instance store the resource information

36. Cache Fusion Block Transfers: Example Overview Instance A 1008 Instance D Resource master Instance C Instance B

37. Example 1: Read with No Transfer Instance D Resource master Instance C Request to obtain a shared resource on C N 1 1008 Instance B Instance A

38. Example 1: Read with No Transfer Instance D Resource master Instance C N 1 1008 Instance B Instance A 2 The request is granted and the requesting instance is informed N–>S

39. Example 1: Read with No Transfer Instance A Instance D Resource master Instance B Instance C S Read request 1 3 2 1008

40. Example 1: Read with No Transfer Instance A Instance D Resource master Instance B Block image delivered Instance C S 1008 1 2 3 4

41. Example 2: Read to Write Transfer Instance A Instance D Resource master Request to obtain an exclusive resource on B Instance C S 1008 Instance B N 1 1008

42. Example 2: Read to Write Transfer Instance A Instance D Resource master Instance C Instruction to transfer the block to B for exclusive access S 1008 Instance B N 2 1 1008

43. Example 2: Read to Write Transfer Instance A Instance D Resource master Instance C Block and resource status (including C’s plan to close its resource) S–>N 1008 Instance B N 1008 2 1 3

44. Example 2: Read to Write Transfer Resource assumption and status message Instance A Instance D Resource master Instance C N 1008 Instance B N–>X 1009 2 1 3 4 1008

45. Example 3: Write to Write Transfer Instance D Resource master Instance C Instance A N Instance B X 1009 Request to obtain resource in exclusive mode 1 1008

46. Example 3: Write to Write Transfer Instance C Instance A N Instance B X 1009 Instruction to transfer exclusive resource to B 1 2 Instance D Resource master 1008

47. Example 3: Write to Write Transfer Instance D Resource master Instance C Instance B X–>N 1009 Instance A N 1009 Exclusive-keep copy of buffer 1 2 3 1008

48. Example 3: Write to Write Transfer Instance D Resource master Instance C Instance A N–>X 1013 Instance B 1009 N 1 2 3 4 1008

49. Example 4: Write to Read Transfer Instance A Instance D Resource master X 1013 Instance B 1009 N Instance C N Request to obtain resource in shared mode 1 1008

50. Example 4: Write to Read Transfer Instance D Resource master Instance A X 1013 Instance B 1009 N Instance C N Instruction to transfer shared resource to C 1 2 1008

51. Example 4: Write to Read Transfer Instance D Resource master Instance A X–>S 1013 Instance B 1009 N Instance C N 1013 Shared-keep copy of buffer 3 2 1 1008

52. Example 4: Write to Read Transfer Instance D Resource master Instance A S 1013 Instance B 1009 N Resource assumption information Instance C N–>S 1013 2 3 4 1 1008

53. Block Transfers in Real Application Clusters Instance A Node 1 Instance B Node 2 Cache Fusion block transfer Block transfer with forced disk write

54. Notes on Using Multiblock Locks  Use only for query intensive or already- partitioned databases  GC_FILES_TO_LOCKS -Assign the same value to for every instance in a cluster database  Leave no white space in the parameter value between the double quotes: separate the entries for each set of file IDs with a colon

55. Notes on Using Multiblock Locks  Configure Cache Fusion, instead of multiblock locks, for selected files by – Not including their IDs in GC_FILES_TO_LOCKS – Setting their LOCK_COUNT to zero and not assigning a grouping factor

56. Real Application Clusters Guard Architecture Monitors Vendor CMS IP addresses Listeners Oracle instance DGM Application or middleware Monitors Vendor CMS IP addresses Listeners Oracle instance DGM IP (A) IP (B) Node B (Secondary) Node A (Primary) Pack Disk group manager, only needed on some platforms

57. Real Application Clusters Guard Failover Vendor CMS IP addresses Listeners Oracle instance DGM Monitors Vendor CMS IP addresses Listeners Oracle instance DGM Heartbeat monitor Listener monitor Instance monitor Node A Node B (New primary) Pack Application or middleware IP (A) IP (B) Failover Monitors

58. Listeners  Multiple listeners enable – Client load balancing – Connect-time failover – Transparent Application Failover (TAF) Clients Cluster nodes Listeners

59. TNS Connect Descriptors rac1= (DESCRIPTION= (ADDRESS= (PROTOCOL=tcp) (HOST=aaacme1) (PORT=1630)) (CONNECT_DATA= (SERVICE_NAME=rac.us.aaacme.com))) $ sqlplus qualassur1@rac1 SQL*Plus: – Production (c) Copyright 2000 Oracle Corporation. All rights reserved. Enter password:

60. Host-Based Failover  Detect failure by monitoring the heartbeat  Reorganize cluster membership in the Cluster Manager  Transfer disk ownership from the primary node to a secondary node  Restart application and database binaries  Perform application and database recovery  Reestablish client connections to the failover node

61. Real Application Clusters Failover  Detect failure by monitoring the heartbeat  Reorganize instance membership  Perform instance recovery  Reestablish failed client connections

62. TAF in Real Application Clusters  Can be used with spare nodes or with primary/secondary instance configurations  Designed for this environment, but can be used for – Real Application Clusters Guard – Replicated systems – Data Guard

63. TAF Failover Mode Options  Must add failover options manually to TNS configuration files  They are part of the CONNECT_DATA section of a connect descriptor

64. TAF Failover Mode Options  Failover options include – TYPE : Identify the nature of TAF, if any – METHOD : Configure how quickly failover can occur – BACKUP : Identify an alternate net service name – RETRIES : Limit the number of times a reconnection will be attempted – DELAY : Specify how long to wait between reconnection attempts

65. TAF Implementation  The recommended configurations for TAF include: – Connect-time failover with client load balancing – Retrying connections – Pre-establishing connections

66. Connect-Time Failover with Client Load Balancing rac.us.aaacme.com= (DESCRIPTION= (LOAD_BALANCE=on) (FAILOVER=on) (ADDRESS= (PROTOCOL=tcp) (HOST=aaacme1) (PORT=1521)) (ADDRESS= (PROTOCOL=tcp) (HOST=aaacme2) (PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=rac.us.aaacme.com) (FAILOVER_MODE= (TYPE=select)(METHOD=basic))))

67. Retrying Failover Connections rac1= (DESCRIPTION= (ADDRESS= (PROTOCOL=tcp) (HOST=aaacme1) (PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=rac1.us.aaacme.com) (FAILOVER_MODE= (TYPE=select)(METHOD=basic) (BACKUP=rac2) (RETRIES=20) (DELAY=15)))) rac2= (DESCRIPTION=...

68. Preestablishing a Connection for TAF rac1= (DESCRIPTION= (ADDRESS= (PROTOCOL=tcp) (HOST=aaacme1) (PORT=1521)) (CONNECT_DATA= (SERVICE_NAME= rac.us.acme.com) (INSTANCE_NAME=rac1) (FAILOVER_MODE= (BACKUP=rac2) (TYPE=select) (METHOD=preconnect) ))) rac2= (DESCRIPTION= (ADDRESS= (PROTOCOL=tcp) (HOST=aaacme2) (PORT=1521)) (CONNECT_DATA= (SERVICE_NAME= rac.us.acme.com) (INSTANCE_NAME=rac2) (FAILOVER_MODE= (BACKUP=rac1) (TYPE=select) (METHOD=preconnect) )))

69. A Q & Q U E S T I O N S A N S W E R S