INFSO-RI Enabling Grids for E-sciencE Experience with monitoring of Prague T2 site Tomáš Kouba NEC 2007, Varna, Bulgaria

Enabling Grids for E-sciencE INFSO-RI Introduction 2 sites involved in EGEE and WLCG project in Prague: –farm golias (praguelcg2)‏  about 150 nodes with 450 cores  CE, 2xSE, many WNs, PBSPro head node –farm skurut (prague_cesnet_lcg2)‏  CE, SE, regional BDII –core infrastructure for VOs voce and auger:  LFC catalogue, VOMS server, lcg RB, glite WMS flexible monitoring is crucial for reliability (see

Enabling Grids for E-sciencE INFSO-RI Nagios I Why –de facto standard in monitoring –open source –easy to write new sensors –static configuration is not a problem Other competitors –ganglia –cacti –zenoss (built on ZOPE)‏ –zabbix (rapid development, graphing features, less robust)‏ –moodss

Enabling Grids for E-sciencE INFSO-RI Nagios II - addons Addons: –best source is –nuvola (better html and css look for nagios)‏ –nagiosql (web frontend for generating configuration from sql db)‏ –NagiosReport (developed locally, summarizes problems at site, information is gotten from nagios log files and status files)‏ –NagiosGrapher (generates graphs from nagios plugin's output)‏

Enabling Grids for E-sciencE INFSO-RI NagiosReport example Nagios summary generated at 09/11/ :10:02 in seconds. ================================================================================ Hosts in trouble Hosts in downtime (not monitored): golias01, golias15, golias16, golias17, golias59, golias97, golias99 ================================================================================ downtimes: ========== golias01: Host je docasne vyrazen kvuli chybejicim dilum. Mozna bude opraven. golias15: Stroj je nedostupny, protoze funguje jako remore syslog server golias16: Stroj docasne nedostupny kvuli presunuti do jine site - testovani Cisco routeru golias17: Host vyrazen pro testovani Glite 3.1 na SLC4 golias59: Vyrazen na nahradni dily golias97: The node currently does not exist. golias99: The node currently does not exist. ================================================================================ Hosts with problem occured in last 8 hours ================================================================================

Enabling Grids for E-sciencE INFSO-RI Nagios III Plugins –default plugins (part of nagios default installation)‏  ping, disk, procs, load, tcp, swap, ldap, gentoo_glsa, gentoo_service_rc_all –SRCE plugins (developed by Emir Imamagic)‏  cert, dpm, dpns, edg_broker, globus_gram2, gridftp, lfc, srm, srm_ping, voms  executed in SLC3 UI installation in chroot environment –RAL plugins  lcg_same (by Chris Brew)‏ –locally developed  hpacucli, ups, jobs, gstat

Enabling Grids for E-sciencE INFSO-RI Generating configuration I We have hardware database for hw management Extended with service definitions and hw-service relations Definition of WSDL interface to the database:

Enabling Grids for E-sciencE INFSO-RI Generating configuration II python client for generating the actual configuration –uses ZSI python SOAP bindings (older bindings pySOAP, SOAPpy etc. were not sufficient)‏ technically the WSDL file is generated from C header file (it is much easier for developer than writing WSDL by hand)‏ the project's home page is at under name SiteQuerywww.nagiosexchange.com

Enabling Grids for E-sciencE INFSO-RI Future work Present dependencies in the database and limit number of false alerts Create another sensors for our hardware (air condition unit, diesel power unit)‏ Cooperate our work with monitoring group of LCG/Hepix at Present another client for generating cfengine/quattor configuration