Sergio Fantinel, INFN LNL/PD GridICE The eyes of the grid A monitoring tool for a Grid Operation Center by DataTAG WP4 Sergio Fantinel, INFN LNL/PD

GridICE Actual Implemantation Outline Monitoring scenario Collection of info: EDG WP4 Fmon Framework & GLUE/GLUE+ Schema Discovery, info retrival and presentation service Real testbed experience and results Next steps

Monitoring scenario Different layers of info generation Different points of view Computing Element Storage Element Worker Node Resource Broker Information Index Replica Manager Replica Catalog […] SERVICE checks gatekeeper gsiftp gris gdmp RB/LB … “GRID/VO” measurements number of total CPUs number of free CPUs number of running jobs number of waiting jobs SE free disk space LOW LEVEL measurements CPU load memory usage disk usage (per partition) network activity number of processes number of users (UI) …

GLUE+

Central Monitoring Database EDG WP4 FMon Framework web interface ldap query GIIS (GLUE schema) First discovery phase Central Monitoring Database information index Second discovery phase ldap query monitoring server EDG-WP4 fmonserver GRIS (GLUE+ schema) cluster worker node /proc filesystem WP4 sensor run read metric output EDG-WP4 monitoring agent EDG-WP4 monitoring agent cluster worker node /proc filesystem WP4 sensor run read metric output write run ldif output information providers farm monitoring archive read cluster head node

Experiment Specific Measures Integration Possible and easy integration of VO/Experiment measures publication It must be modified the GLUE schema and write the experiment sensors (ex. CMS KIN/SIM event production) EDG-WP4 fmonserver GRIS (GLUE++ schema) cluster worker node /proc filesystem CMS sensor run read metric output EDG-WP4 monitoring agent EDG-WP4 monitoring agent cluster worker node /proc filesystem CMS sensor run read metric output write run ldif output information providers farm monitoring archive read cluster head node

Server Side service layout GRID WEB Discovery 1A 5B 1C GIIS 1B Gfx/Presentation Config 2A 2B GRIS 5A Monitoring DB scheduler 3 4A 4B Check 1: entities discovery 2: generation of config files 3: check scheduling 4: entities info collection 5: DB info rendering Grid Information System LDAP Interface Developed by DataTAG WP4

Discovery service: entities list This is the list of entities currently tracked by the monitoring system: Clusters Storage Services Worker Nodes (CL) Computing Elements (CL) Run Time Environments (CL) Virtual Organizations (CE) Storage Extents (WN) Network Adapters (WN) Storage Space (SE) Storage Protocols (SE) CL = Cluster WN = Worker Node/host SE = Storage Service

Data presentation service (2) The presentation of the date was made addressing different user types : Vo views, for a VO manager Site views, grid manager Single entity grid/site manager (see next slides)

Data presentation service (3)

Data presentation service (4)

Real testbed deployment First tests on the DataTAG tesbed -> march presentation at 1st DatatTAG review First wide deployment on the CMS/LCG-0 testbed (29/07/2003): 12 sites x 150CPUs bo.infn.it, cern.ch, cmsfarm1.ba.infn.it, cnaf.infn.it, hep.ph.ic.ac.uk, in2p3.fr, its.uiowa.edu, lnl.infn.it, mi.infn.it, pd.infn.it, phy.bris.ac.uk, phy.ncu.edu.tw From the beginning of july started on the Certification Testbed of LCG-1 A few days started on the Deployment Testbed of LCG-1

Results Good response from the CMS/LCG-0 testbed The only big problem we found on monitoring big sites is due to time (NTP) synchronization of the machines so not a problem of the GridICE tool Thanks to the collaboration of the LCG people we are improving significantly our monitoring tool

Next steps, short term Check plug-in refactoring: we made some tests with LDAP and to improve the performance we must aggregate the queries (less queries, more date to be transferred). Data reduction with the activation of the thresholds We are thinking to introduce some kind of caching for last data pushed in the DB to less stress the DB DB schema improvement: dynamic discovery of the URL GRIS (at the moment with GlueInformationServiceURL). Introduction of new components: CESEBind, SECEBind. Activation of the service (GRIS, GIIS, gridftp,…) checking

Next steps, short term (2) Grid Collective Service Monitoring (e.g. edg-broker, edg-replica-location-service) Job Monitoring at queue level (some open issues, ex. VO) Native R-GMA support as GIS: we need a working and stable testbed with R-GMA as GIS, extend the CE GIN to support the new metrics. Hosts Role (via GlueHostService) in order to associate service state to proper host state Support for multiple Information Index back-up Information Index