1. Manage large RAC Clusters Session# 851 Tom S. Reddy Database Administration, Inc.

2. About the Speaker IOUG Conference Committee 2013 Upgrades, Migrations and New Features Oracle Certified Database Administrator Presented at previous COLLABORATE Conferences Focus on MAA, HA, RAC and performance tuning Chief Technology Officer at Database Administration, Inc.

3. Survey DBAs Developers Sys Admins Managers Others How many use RAC? How many use Exadata? Versions10g? 11g? 12c! Familiar with MAA?

4. Manage Large RAC Clusters Basic Commands Challenges Solutions Workload Management Services Performance Management Standby and Backups

5. Overview of RAC What is RAC? How does RAC work? How many nodes can you have? What are the advantages and disadvantages of having large number of RAC nodes?

6. Overview of RAC …Cont’d High Availability Highly Scalable Commodity Servers Database Cloud vs Pluggable DB’s Oracle® Real Application Clusters Administration and Deployment Guide 11g Release 2 (11.2)

8. What is a large cluster? 4 nodes 8 nodes 12 nodes 16 nodes 32 nodes 100 nodes? Nodes vs CPUs

9. Design of a large cluster Hardware Chassis Network Servers CPU Memory OS Oracle Binaries Disk Layout

10. Sample Specs Servers 16 Core Count 2x6x16 = 192 cores at 3.46Ghz! Memory 48x16 = 768 GB! SGA > 400GB!! PGA > 250GB!!

11. New System Specs Servers 8 Core Count 4x8x8 = 256 cores at 2.7Ghz! Memory 256x8 = 2048 GB! SGA > 1024GB!! PGA > 768GB!! Disk SSD SAN Infiniband Private Network

12. Oracle Binary Setup Installation Binaries on Internal Hard Drives Home1: Grid Infrastructure/ASM Home2: Oracle Database Home3: Grid Control Agent This allows for rolling upgrades/patching OCR & Voting Disk Files on dedicated ASM disk group (Normal or High Redundancy)

13. Challenges Sheer number of nodes How do you build them? How do you install oracle? How do you patch oracle? How do you perform other maintenance?

14. Challenges …Cont’d How do you monitor them? How do you manage them? Performance Monitoring Performance Management Performance Tuning

15. Solutions Oracle Tools OEM crsctl srvctl Other Tools Custom Scripts

16. Solutions

17. Get familiar with the following commands: crsctl -help crsctl status resource -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources --------------------------------------------------------------------------------... ora.LISTENER.lsnr ONLINE ONLINE housrv01 ONLINE ONLINE housrv02 ONLINE ONLINE housrv03 ONLINE ONLINE housrv04 ONLINE ONLINE housrv05 ONLINE ONLINE housrv06 ONLINE ONLINE housrv07 ONLINE ONLINE housrv08 ora.asm ONLINE ONLINE housrv01 ONLINE ONLINE housrv02 ONLINE ONLINE housrv03 ONLINE ONLINE housrv04 ONLINE ONLINE housrv05 ONLINE ONLINE housrv06 ONLINE ONLINE housrv07 ONLINE ONLINE housrv08 ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE housrv05 ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE housrv03 ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE housrv01 ora.r1test.db 1 ONLINE ONLINE housrv01 Open 2 ONLINE ONLINE housrv02 Open 3 ONLINE ONLINE housrv03 Open 4 ONLINE ONLINE housrv04 Open 5 ONLINE ONLINE housrv05 Open 6 ONLINE ONLINE housrv06 Open 7 ONLINE ONLINE housrv07 Open 8 ONLINE ONLINE housrv08 Open

18. Solutions …Cont’d Get familiar with the following commands: srvctl –help Usage: srvctl [ ] commands: enable|disable|start|stop|relocate|status|add|remove|modify|getenv|setenv|unsetenv|config objects: database|instance|service|nodeapps|vip|network|asm|diskgroup|listener|srvpool|server|scan|scan_ listener|oc4j|home|filesystem|gns|cvu For detailed help on each command and object and its options use: srvctl -h or srvctl -h srvctl status database –d R1TEST Instance R1TEST1 is running on node housrv01 Instance R1TEST2 is running on node housrv02 Instance R1TEST3 is running on node housrv03 Instance R1TEST4 is running on node housrv04 Instance R1TEST5 is running on node housrv05 Instance R1TEST6 is running on node housrv06 Instance R1TEST7 is running on node housrv07 Instance R1TEST8 is running on node housrv08

19. Solutions …Cont’d Custom scripts raccmd_serial scp to move files cygwin

20. Solutions …Cont’d raccmd_serial.sh housrv "ps -ef | grep -v grep | grep smon " 01 16 Running command $2 on housrv01 ssh root@housrv01 ps -ef | grep -v grep | grep smon oracle 8885 1 0 Feb11 ? 00:13:53 ora_smon_r1srvc1 root 17366 1 1 Feb07 ? 08:29:30 /oracle/grid/11.2.0/grid/bin/osysmond.bin oracle 18115 1 0 Feb07 ? 00:00:04 asm_smon_+ASM1 Running command $2 on housrv02 ssh root@housrv02 ps -ef | grep -v grep | grep smon oracle 12763 1 0 Feb11 ? 00:17:01 ora_smon_r1srvc2 root 17390 1 1 Feb07 ? 06:31:09 /oracle/grid/11.2.0/grid/bin/osysmond.bin oracle 18271 1 0 Feb07 ? 00:00:02 asm_smon_+ASM2 Running command $2 on housrv03 ssh root@housrv03 ps -ef | grep -v grep | grep smon root 17439 1 1 Feb07 ? 06:40:01 /oracle/grid/11.2.0/grid/bin/osysmond.bin oracle 18343 1 0 Feb07 ? 00:00:02 asm_smon_+ASM3 oracle 23321 1 0 Feb11 ? 00:16:53 ora_smon_r1srvc3 …

21. Solutions …Cont’d raccmd_serial.sh (Courtesy of Joel N) #!/bin/bash USR=root if [ -z $1 ]; then echo No Servername echo SYNTAX raccmd racname ""command"" exit fi if [ -z $2 ]; then echo No command echo SYNTAX raccmd racname ""command"" exit fi if [ -z $3 ]; then echo No start server number echo SYNTAX raccmd racname ""command"" exit fi if [ -z $4 ]; then echo No end server number echo SYNTAX raccmd racname ""command"" exit fi for i in $(seq $3 $4) do if test $i -lt 10 then echo Running command '$2' on $10$i echo ssh $USR@$10$i$5 $2 ssh $USR@$10$i $2 else echo Running command '$2' on $1$i echo ssh $USR@$1$i$5 $2 ssh $USR@$1$i $2 fi done

22. Solutions …Cont’d Cygwin xterm_all_nodes (Courtesy of Kevin L) Opens and xterm on each node in the cluster exe_all_nodes Opens an xterm on each node (including the one you are logged in on) and runs the specified executable

23. Solutions …Cont’d

25. Workload Management Services Load Balancing Resource Manager Performance Management SQL Monitoring

26. Services Enable Workload management by routing work to optimal instances Helps lower Interconnect traffic vs Cache Fusion One or more Services per Instance Typically one per Application or type of Workload Large Database Cloud can be divided into smaller/manageable resources DBMS_SCHEDULER & Job Classes

27. Services R1COMPC ARAisionJob ClassService Name Node1Node2Node3Node4Node5Node6Node7Node8 hourCOMPa01hourCOMPa02hourCOMPa03hourCOMPa04hourCOMPa05hourCOMPa06hourCOMPa07hourCOMPa08 R1COMPC1R1COMPC2R1COMPC3R1COMPC4R1COMPC5R1COMPC6R1COMPC7R1COMPC8 PPAA95ARA_095R1COMPC_ARA_095 AAPP30ARA_030R1COMPC_ARA_030 AAPP60ARA_060R1COMPC_ARA_060 PPAA20ARA_020R1COMPC_ARA_020 PPAA10ARA_010R1COMPC_ARA_010 PPAA50ARA_050R1COMPC_ARA_050 PPAA12ARA_012R1COMPC_ARA_012 PPAA14ARA_014R1COMPC_ARA_014 AAPP63ARA_063R1COMPC_ARA_063 AAPP293ARA_293R1COMPC_ARA_293 AAPP305ARA_305R1COMPC_ARA_305 AAPP72ARA_072R1COMPC_ARA_072 AAPP306ARA_306R1COMPC_ARA_306 AAPP93ARA_093R1COMPC_ARA_093 AAPP91ARA_091R1COMPC_ARA_091 AAPP80ARA_080R1COMPC_ARA_080 AAPP94ARA_094R1COMPC_ARA_094 PPAANAR1COMPC_IDR 95 30,60 20,10,50,12,14 63,293,305,72,3 06,93,91,80,94

28. Solutions

30. Resource Manager Resource throttling based on Consumer Groups Typically used to manage CPU Parallelism Services Basic: Limit user CPU load to 90% Complex: Consumer Groups/additional resources

31. Solutions

32. Performance Management OEM Performance Tab SQL Monitoring Top Activity Tab Top Sessions Top Services

33. OEM Performance Tab

36. AWR Trends DateCPU %User IO %Other % 11/6/2012241759 11/26/2012651718 12/10/2012631621 12/31/2012731215 1/14/2013632215 2/4/2013562321 2/18/2013533512 2/25/2013493813 3/4/2013483616 3/11/2013512821 3/18/2013523117 3/25/2013473617 4/1/2013503020 4/2/2013533413

37. SQL Monitoring Best way to monitor real-time SQL using OEM Real-Time Monitoring of SQL Database or Instance Level SQL must have consumed at least 5 sec of CPU&IO or be run in parallel to be captured!

38. Solutions

39. Top Activity Sessions

40. Top Activity Services

41. Performance Management ADDM addmrt.sql vs addmrpti.sql AWR awrgrpt.sq vs awrrpti.sql awrgdrpt.sql vs awrgdrpi.sql ASH ashrpti.sql

42. ADDM

43. AWR

49. AWR Instance

52. ASH

53. Standby Physical Standby Equal or fewer nodes All standby nodes receive archived logs Only one apply node Switchover/Failover DGMGRL Manual (shutdown all other nodes) and work on one node in each location OEM…doesn’t always work

54. Standby Check Manual Primary select thread#, max(sequence#) from v$archived_log group by thread# order by 1; Standby select thread#, max(sequence#) from v$archived_log where applied = 'YES' group by thread# order by 1; select name, value, time_computed from v$dataguard_stats; DGMGRL show configuration verbose show database verbose psrva show instance verbose psrva1 show database verbose s1srva OEM

55. Standby Check THREAD# MAX(SEQUENCE#) ---------- -------------- 1 19945 2 15671 3 14383 4 14491 5 15028 6 14394 7 14587 8 14868 9 14185 9 14184 10 14202 11 15140 12 14494 13 14383 14 14956 15 15199 16 14383 16 rows selected NAMEVALUETIME_COMPUTED transport lag+00 00:00:004/7/2013 21:42 apply lag+00 00:00:004/7/2013 21:42 apply finish time+00 00:00:00.0004/7/2013 21:42 estimated startup time1224/7/2013 21:42

56. Backups RMAN Image Copy Backups & Incremental Updates Backups at the Standby Create channels by Instances CONFIGURE CHANNEL DEVICE TYPE sbt CONNECT 'SYS/change_on_install@node1' Change Tracking file for Incrementals Managed by OEM Flashback Technologies

57. Duplicates Duplicate from large RAC production environments to equivalent or few nodes Regression or Development Refreshes RMAN Live Duplicate Takes just a few hours on a multi-TB DB RMAN Backup-based Duplicate SAN based Snapshots can be much faster…Testing

58. Summary Basic Commands Challenges Solutions Workload Management Services Performance Management Standby and Backups

59. Manage large RAC Clusters Session# 851 Questions? Please fill out evaluations! www.ioug.org/eval http://coll13.mobile.mapyourshow.com www.ioug.org/eval http://coll13.mobile.mapyourshow.com

60. Manage large RAC Clusters Session# 851 Tom S. Reddy Database Administration, Inc. tom.reddy@dbadmin.com http://www.linkedin.com/in/tomreddy/