1. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t ASM and Oracle Service Availability Monitoring LCG 3D Workshop CERN, January 26 th, 2007 Dawid Wojcik, CERN IT LCG

2. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Oracle Architecture at CERN – LCG 3D Workshop, Jan 2007 - 2 The need for monitoring Why build custom monitoring solutions:Why build custom monitoring solutions: –complex setup with many potential point of failures –large number of RAC and ASM instances –difficult customization of existing monitoring products –missing some crucial functionality in other solutions (e.g. dashboard) –simplified deployment

3. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Oracle Architecture at CERN – LCG 3D Workshop, Jan 2007 - 3 RAC monitoring requirements Reliable (24x7), easy to use and configureReliable (24x7), easy to use and configure Provides up to date information (frequent runs)Provides up to date information (frequent runs) Centralized – no configuration or deployment on RAC sideCentralized – no configuration or deployment on RAC side Web interface (RAC monitoring dashboard) – one common place for ASM and RAC's services statusWeb interface (RAC monitoring dashboard) – one common place for ASM and RAC's services status Monitoring of ASM instances (diskgroups and failgroups)Monitoring of ASM instances (diskgroups and failgroups) Monitoring of oracle services and clusterwareMonitoring of oracle services and clusterware Notification send via emails & SMSs to DBAsNotification send via emails & SMSs to DBAs Availability numbersAvailability numbers

4. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Oracle Architecture at CERN – LCG 3D Workshop, Jan 2007 - 4 Architecture Central monitoring script:Central monitoring script: xml configuration for each clusterxml configuration for each cluster generates bash scripts to gather necessary informationgenerates bash scripts to gather necessary information sends files to the destination nodes and executes it remotelysends files to the destination nodes and executes it remotely retrieves output and upload results to the web serviceretrieves output and upload results to the web service sends alerts if necessarysends alerts if necessary PHP webservice:PHP webservice: receives monitoring datareceives monitoring data separates monitoring from presentationseparates monitoring from presentation general overview of status of deployed RACsgeneral overview of status of deployed RACs

5. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Oracle Architecture at CERN – LCG 3D Workshop, Jan 2007 - 5 Dashboard functionality ASM monitoringASM monitoring –diskgroup and ASM instances status –disk space monitoring and diskgroup balance Production services monitoringProduction services monitoring –all production clusters' services CRS servicesCRS services –all clusterware services for deployed RACs Unavailability tracingUnavailability tracing –instances unavailability –clusterware services unavailability Disabling monitoring for machines under scheduled maintenanceDisabling monitoring for machines under scheduled maintenance

6. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Oracle Architecture at CERN – LCG 3D Workshop, Jan 2007 - 6 Screenshots

7. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Oracle Architecture at CERN – LCG 3D Workshop, Jan 2007 - 7 Screenshots... continued

8. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Oracle Architecture at CERN – LCG 3D Workshop, Jan 2007 - 8 Q & A