1. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t DBA Experience in a multiple RAC environment DM Technical Meeting, Feb 2008 Miguel Anjo et al, CERN Physics Databases LCG

2. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Databases at CERN Physics Databases Infrastructure Administrative DBA experience in a multiple RAC environment - 2

3. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Motivation Big shops may have similar problems: –Users from different backgrounds –Power developers like to see the DB internals –Production systems are accessible by developers –Developers/project managers want reports –Monitoring is a key for fast reaction and problem resolution Share ideas with other RAC and HA environments –Some solutions may be made available –Ideas and improvements are welcome DBA experience in a multiple RAC environment - 3

4. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Standards One version, same patches everywhere (using cloning from a $ORACLE_HOME image) Two Oracle roles – developer and application –Dev: can do DDL Two Oracle profiles – developer and application –Dev: limit 10 sessions, password expiration Three development stage –Development: 8/5 monitoring, limited space –Integration: 8/5 monitoring, dedicated RAC for tests, patched 2 weeks before production, consultancy –Production: 24/7 monitoring, backup to tape every hour, limited number and scheduled interventions Naming convention, logbook DBA experience in a multiple RAC environment - 4

5. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t HA monitoring cluster 2 node mid-range server (similar to production) Runs Oracle clusterware Connected to shared storage Machine alias pointing to a virtual IP Responsible for: –monitoring –Backups –scripts repository –Oracle binaries repository (including $ORACLE_HOME tar images) DBA experience in a multiple RAC environment - 5

6. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t User$, guser$, all$, gall$ views For power users that want DBA privileges Based on v$/gv$ views + groups view create table cern_database_groups (group_name varchar2(30), username varchar2(30)); View to get users from group: create view cern_all_users as select user username from dual union (select cdg2.username from cern_database_groups cdg1, cern_database_groups cdg2 where cdg1.group_name = cdg2.group_name and cdg1.username=user); DBA experience in a multiple RAC environment - 6 v_$sessionv_$open_cursorv_$session_longopsv_$sql v_$sqlareav_$session_waitv_$db_object_cachev_$lock v_$sgastatv_$statnamev_$latchv_$latchname v_$latch_childrenv_$sysstatv_$parameterv_$mystat

7. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t User$, guser$, all$, gall$ views Example simple views: create or replace view user$session as select * from v$session where username=user; create or replace view gall$session as select * from gv$session where username in (select username from cern_all_users); create or replace view gall$lock as select * from gv$lock where (sid,inst_id) in (select sid,inst_id from gall$session); DBA experience in a multiple RAC environment - 7

8. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Session Manager DBA experience in a multiple RAC environment - 8

9. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Session Manager Developers want to know –“what’s happening to my session?” PL/SQL packaged developed in-house Allows users to use a GUI to see his sessions Shows SQL, DML and DDL locks Possible to kill user sessions (buggy) –Creates a job that runs on appropriate instance DBA experience in a multiple RAC environment - 9

10. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Session Manager DBA experience in a multiple RAC environment - 10

11. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Weekly reports DBA experience in a multiple RAC environment – 11

12. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Weekly reports – Main goals Not OEM replacement Targeted to application owners and experiment DBAs Quick overview of application/services usage –spotting behavior changes –application contacts/responsible –Preventive monitoring DBA experience in a multiple RAC environment - 12

13. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Weekly reports - Info shown List projects, users and responsible Sessions, cpu, logical and physical reads/writes per username Top 5 users (CPU, physical reads, writes) Usage per service per hour normalized –100% ~= 1 “DB Time” hour Sessions per day per username Connection errors, space usage Very Short and long sessions –connection pooling Passwords expiring and schemas not used Queries not using bind variables DBA experience in a multiple RAC environment - 13

14. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Weekly reports - Setup AUDIT_TRAIL=DB Base information stored at SYS.DBA_AUDIT_SESSION / SYS.AUD$ Create schema with some dictionary access Procedure to create aggregates - DAILY, WEEKLY, MONTHLY - per username Materialized views for faster report generation PHP page to display tables DBA experience in a multiple RAC environment - 14

15. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Wiki pages, logbook Procedures User documentation Logbook DBA experience in a multiple RAC environment - 15

16. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t RACMon - monitoring DBA experience in a multiple RAC environment - 16

17. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t RAC monitoring Based on python, bash scripts and xml Checks host, service, ASM disks availability Possible to disable host or RAC –Scheduled/unscheduled interventions –Creates blackout in OEM Logs on database Sends SMS after 3 failures Other monitoring tools –Tablespace and user quota alarms –Alert logs’ ORA - errors daily digest email –Streams chain monitoring UI DBA experience in a multiple RAC environment - 17

18. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Service Level Status Monitoring for managers Possible to get RSS feed on status changes DBA experience in a multiple RAC environment - 18

19. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Physics Databases numbers ~20 RACs (2-8 nodes, 6-35TB raw space) 220 CPUs, 440GB RAM, 300TB disk –soon quad-cores (3-4 node RAC) 345 production schemas (feb 07) 9.625 GB production data Largest table: 3.02 billion rows (IOT, non partitioned) Workload –5.2 million sessions/week (week 3/2008) –128 MB/s (average for week 3/2008) –995 CPU hours/week (CPU time avg for week 3/2008) DBA experience in a multiple RAC environment - 19

20. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Questions … and answers! Tools presented here developed mostly by –Dawid Wojcik, Eric Grancher, Jacek Wojcieszuk, Luca Canali, Miguel Anjo, Sebastian Lopienski name.surname@cern.ch Thanks, Miguel.Anjo@cern.ch DBA experience in a multiple RAC environment - 20

21. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t DBA experience in a multiple RAC environment - 21