1. Computing Facilities CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF CF Monitoring: Lemon, LAS, SLS I.Fedorko(IT/CF) IT-Monitoring WG 03/10/2011

2. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Overview Lemon (short summary) LAS (Lemon Alarm System) SLS (Service Level Status) New CF Monitoring We need to address new requirements beyond current design change of environment (e.g. Service Now) 03/10/2011 CF for IT-Monitoring WG 2

3. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Lemon Discussed topic –Scaling: Lemon instance scales well as application server, limited flexibility with multiple instances –Data aggregation at runtime –Statistics: see Lemon poster with overview of data-flow and statistics Not discussed topics –Integration with other monitoring (Windows,Nagios) –New (Lemon) monitoring architecture 03/10/2011 CF for IT-Monitoring WG 3

4. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF LAS - overview Exception Metrics ITCM (Remedy) Lemon-webLAS GUI Lemon Oracle DB LAS Business Logic PL/SQL Operator Administrator We need to address: Application/service monitoring High level objects alarms (exception on data integrated at runtime) cluster CPU load over threshold LAS  integration with windows monitoring With Remedy phase out the ITCM workflow will be migrated to Service Now 03/10/2011 CF for IT-Monitoring WG 4

5. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF LAS – app and service alarms Host 1 Application A HW scan CPU load partitions occupancy is app running log parsing X log parsing Y SMART IPMI CPU load partitions occupancy is app running log parsing SMART IPMI Except. 1 Except. 2 Except. 3 Except. 4 Except. 5 Current LAS view App/Service alarms Sys-admin alarms Host 2 Application A HW scan Application/service monitoring by exception enhancement to address different recipients (e.g. operator, SM) different issue types (e.g. hw, app, service) 03/10/2011 CF for IT-Monitoring WG 5

6. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF LAS – interaction with Service Now Initial activities started –Define scope of Event management –Design exercise LAS +ITCM  new LAS + Event management @ Service Now Event Management @ Service Now –Keep EM@SN generic and define Event interface suitable for various monitoring systems –No replacement of monitoring tools, rather event dispatching and converting to incident, change, etc. LAS CS (Spectrum) AIS LHC control Event Mgmt (Service Now) 03/10/2011 CF for IT-Monitoring WG 6

7. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF SLS: overview SLS-web USER Scripts XMLSDBRRD test/probes LemonDB SLS XML Service definition SDB (Service database) stores service definition Service Catalog @ Service now stores service definitions We prepared various strategies for a SDB-SLS migration to the Service Now depending on the IT monitoring strategy and an implementation of processes in Service Now 03/10/2011 CF for IT-Monitoring WG 7

8. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF New CF Monitoring Only few hints of requirements New CF monitoring shall address correlation of performance, application and service monitoring data New CF monitoring shall adapt to changes in the configuration and the Service Management tool New CF monitoring shall cope with available manpower in IT 03/10/2011 CF for IT-Monitoring WG 8

9. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF New CF Monitoring (example) Node with Lemon-agent Node with Lemon-agent Node with Lemon-agent BUS (data collector) Data processor Exception over cluster Data processor Exception over service BUS (exception collector) Data processor (Lemon-server) Monitoring DB Data presenter Lemon-web Data presenter SLS Data presenter User customized solution Node without Lemon-agent (e.g. Nagios) Windows Monitoring 03/10/2011 CF for IT-Monitoring WG 9

10. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Backup From now on backup 03/10/2011 CF for IT-Monitoring WG 10

11. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Lemon SQL TCP/UDP HTTP Sensor Monitoring Agent Local Cache Oracle Database Repository Backend Application Server Lemon CLI Lemon-host-check Web Browser RRD tool / Python Apache/ PHP (command line tool to access data) (command line tool node exceptions) Measurement Repository User InterfacesNode Monitoring 11 03/10/2011 CF for IT-Monitoring WG 11