The Grid Information System Maria Alandes Pradillo IT-SDC White Area Lecture, 4th June 2014

Introduction  What is the Information System  What is the GLUE schema  The past  Software errors  Performance issues  Information volatility  Obsolete entries  Site misconfigurations  The present  GLUE validation  The future  Common solution  Cloud resources  Open issues June 2014 White Area Lectures - WLCG Information System 2

WHAT IS THE INFORMATION SYSTEM? June 2014White Area Lectures - WLCG Information System 3

June 2014White Area Lectures - WLCG Information System 4

June 2014White Area Lectures - WLCG Information System 5

June 2014White Area Lectures - WLCG Information System 6

WLCG Information System Architecture June 2014 White Area Lectures - WLCG Information System 7

Every 2 minutes… June 2014 White Area Lectures - WLCG Information System 8

Behind the scenes June 2014 White Area Lectures - WLCG Information System 9 ComponentLOCLanguage bdii-update398Python glite-info-provider-ldap461Perl glite-info-provider-service513Perl dpm-listspaces1426Python info-dynamic-lsf1531Perl fts-info-publisher204C++ Requirements Operating SystemSL5 or SL6 Python2.5 or 2.6 Openldap2.4 RAM4GB for top BDII

Information Providers June 2014 White Area Lectures - WLCG Information System 10 ComponentMaintained by CREAM CEINFN PBSINFN LSFCERN SGELIP/CESGA SLURMINFN DPMCERN dCacheDESY StoRMINFN Castor/EOSCERN FTSCERN glite-info-provider-serviceRAL

Is the BDII the only way to know about WLCG resources?  Not really…  Without taking into account monitoring tools, there are many information sources for different purposes within WLCG June 2014 White Area Lectures - WLCG Information System 11

WLCG Information Sources ToolMaintained byDescription BDIIWLCGStatic and Dynamic information for EGI and OSG services GstatUnmaintained?Visualisation and validation tool based on the BDII information GOCDBEGIEGI Service Registry (static information manually edited by sys admins) OIMOSGOSG Service Registry (static information) AGISATLASATLAS sites and services (collects information from BDII, GOCDB, OIM and allows for manual input by AGIS admins) SiteDBCMSCMS sites, pledges, CEs and SEs (manual input by sys admins and SiteDB admins) DIRACLHCbLHCb sites and services AliENALICEALICE sites and services (manually maintained) REBUSWLCGPledge information, WLCG topology, Capacity (taken from the BDII and manual input by federations) APELEGIAccounting information directly taken from CEs June 2014 White Area Lectures - WLCG Information System 12

WHAT IS THE GLUE SCHEMA? June 2014White Area Lectures - WLCG Information System 13

June 2014White Area Lectures - WLCG Information System 14

A common data model June 2014 White Area Lectures - WLCG Information System 15 GLUE 1 Computing Resources GLUE 2 Computing Resources

LDAP trees in the BDII June 2014 White Area Lectures - WLCG Information System 16

Do you want to have a look at the information system? Jacques Cousteau approach  Get familiar with the GLUE schema  Try with ldapsearch  Examples available in the BDII Sys Admin guide June 2014 White Area Lectures - WLCG Information System 17 National Geographic approach  Use any of the existing clients available in lxplus  GLUE 1.3  lcg-info  lcg-infosites  GLUE 2.0  ginfo

GLUE 1 vs GLUE 2  EGI sites publish GLUE 1 and GLUE 2  OSG sites publish GLUE 1 only  Information providers use the same logic to publish GLUE 1 and GLUE 2  EGI planned to decommission GLUE 1 this year  This requires clients to start consuming GLUE 2 information  Including experiment frameworks and any application that consumes information from the BDII  It will be a long process that hasn’t started yet June 2014 White Area Lectures - WLCG Information System 18

THE PAST June 2014White Area Lectures - WLCG Information System 19

Software Errors June 2014 White Area Lectures - WLCG Information System matches in the LCG-ROLLOUT list for ‘BDII’ since matches only in 2013 and 2014

Performance issues  Openldap 2.3  LDAP performance tuning for GLUE 1  LDAP performance tuning for GLUE 2 June 2014 White Area Lectures - WLCG Information System 21

Information volatility  Cache implemented in top BDII  To be more robust against system instabilities June 2014 White Area Lectures - WLCG Information System 22

Obsolete entries  Obsolete information published in GLUE 2  Due to a bug in the bdii-update script June 2014 White Area Lectures - WLCG Information System 23

Site misconfigurations  Information Providers rely on configurations files that need to be properly defined by the site  Batch system configuration is not easy  Storage systems could also be complex  Smaller sites with less expertise have sometimes difficulties to get things right June 2014 White Area Lectures - WLCG Information System 24

THE PRESENT June 2014White Area Lectures - WLCG Information System 25

Can we trust the information published in the information system? June 2014 White Area Lectures - WLCG Information System 26

GLUE validation  There are 369 sites currently published in the BDII  It’s impossible to control what they publish right now!  But we need a mechanism to measure how ‘good’ is the information they are publishing  Validation activities started in 2013  Checking the information providers and following up issues with the developers  Monitoring sites  Manually at first  Automatically using the Dashboard later  Nagios monitoring presently done by EGI  Automation is key due to the high volume of information  Validation actions targeting specific attributes have also proven to be more useful for experiments like LHCb June 2014 White Area Lectures - WLCG Information System 27

June 2014White Area Lectures - WLCG Information System 28 WLCG Sites Validation Monitoring

Nagios Monitoring June 2014 White Area Lectures - WLCG Information System 29

Specific VO validation  SRM vs BDII storage capacity numbers June 2014 White Area Lectures - WLCG Information System 30

THE FUTURE June 2014White Area Lectures - WLCG Information System 31

Common solution  Experiments have implemented their own information systems  AGIS has been evaluated for CMS  It would be interesting to unify the effort and the work towards a common solution June 2014 White Area Lectures - WLCG Information System 32

Cloud resources  GLUE schema is being extended to define cloud resources  A new version GLUE 2.1 to be expected by the end of the year  No impact on current version, just an extension  It will require an update in the BDII transparent for the users  EGI is already publishing Cloud resources in the BDII using GLUE 2.0 data model June 2014 White Area Lectures - WLCG Information System 33

Open questions  The BDII seems to be now a stable and mature service offering robustness and performance  However, it’s not the best architecture for combining static and dynamic information  Move more static information to GOCDB/OIM  Use messaging for publishing dynamic information  Information quality still requires an on going effort  Reduce the amount of information currently published by the BDII  Enforce good quality before publication and otherwise publish nothing  Experiments have deployed their own information system solutions  Effort is duplicated  We could learn from each other experience and have a more clear idea of how information systems should evolve June 2014 White Area Lectures - WLCG Information System 34

QUESTIONS? June 2014White Area Lectures - WLCG Information System 35