gLite Information System Riccardo Bruno INFN, Sez. CT 02/07/2007 First Internaional Grid School for Industrial Applications

Outline Grid Information Systems Overview gLite Information Systems GLUE Schema Grid elements LCG Information System Hierarchy LDAP model R-GMA Overview Server Architecture Consumer-Producer model User interfaces References Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

Grid Information Systems Overview Collect information of grid resources Discovering new added resources Monitoring load and health status Publish these information Periodically updated Well know data model Used by Users searching a concrete resource WMS allocating and managing jobs Other monitoring services Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

gLite Information Systems Basic data model. Grid Laboratory Uniform Environment (GLUE) Schema. Two architectures in glite3 LCG Information System (BDII) BDII over Globus MDS (Monitoring and Discovery System). OpenLDAP interface. Preserves LCG-2 compatibility. Relational Grid Monitoring Architecture (R-GMA) Based on the GMA (Grid Monitoring Architecture) standard from the Grid Global Forum Information in SQL relational databases Web Services. Future replacement of LCG IS. Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

GLUE Schema Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

GLUE Schema A schema of objects and attributes describing Grid resources and its relationships. Originally a EU-DataTAG and US-iVDGL coordinated effort. Current participants: EGEE, OSG, Globus and NorduGrid. A way to describe Grid info Statically and dynamically supplied Hierarchically represented Independently of the framework (LDAP, XML, SQL…) Actual release (1.2) is mapped into LDAP XML Old ClassAd (used by Condor Matchmaking) Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

Site Element Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

Cluster Element Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

Computing Element Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

LCG Information System Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

LCG Information System Levels Resource level: Grid Resource Information Server (GRIS) One GRIS on top of each CE, SE, RB, MyProxy (no WNs). Sensors and scripts get status of concrete resources statically (e.g. GlueCEUniqueID) or dynamically (e.g. GlueCEStateWaitingJobs) Site level: Grid Index Information Server (GIIS) Compiles all the information of the different GRISes in a site. LCG-2 recommends using a BDII instead of a GIIS Improves robustness and stability. Called the site BDII. Top level: Berkeley DB Information Index (BDII) Keeps all Grid information about the VOs (generally only one). Stores information from local BDIIs or GIISes in its database. Only queries sites that are included in a configuration file. Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

LCG Information System Hierarchy Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

LDAP Model Way of collecting info Users get info with Pull model (higher level servers periodically query lower level servers) All servers are based on LDAP Inherit hierarchical structure (tree-like) LDAP Data Information Format (LDIF) Users get info with Generic applications ldapsearch (MDS(GRIS/GIIS): 2135 and BDII:2170 ports) Graphical (BDII web; LDAP GUIs) Always can get information about specific resources (maybe more up-to-date) by querying directly the site BDIIs, GIISes or GRISes. Querying VO info with lcg-infosites or lcg-info Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

LDAP Browser showing BDII Info http://www.jxplorer.org/ Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

Relational-GMA Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

R-GMA Overview Added from EDG Project Based on the GMA standard from the GGF Information in SQL relational databases (a DB per VO) Query syntax is a SQL subset Simple consumer-producer model Web Services oriented CLI and Web user interface Allows self-logging applications Future replacement of LCG IS Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

RGMA Overview R-GMA offers a global view of the VO information In one large relational DB: virtual database. Registry stores localization tuples (database rows) published by producers: Standard Tables: CE state in GLUE Schema (by R-GMA-GIN) Applications specific tables (e.g. self-logging with Log4j) Access by SQL queries through a WS interface. Always updated Consumers get information directly from the resource Can be cached by a secondary producer from single o multiple producers to improve access. Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

R-GMA Server Architecture R-GMA Servers Server (one or several) Schema Server (There is one common global schema server for the grid) Registry Server (one or several) Browser (shows grid information via web browser) Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

R-GMA Server Description R-GMA Clients Generic Client (generic set of APIs for different languages and CLIs) Servicetool (used by all services to publish its existence and status) Site Publisher Client to publish the existence of a site. Each site must have one R-GMA site publisher. GadgetIN (GIN) Client to extract information from MDS and to republish it to R-GMA. used by the CE. Data archiver (flexible archiver) Client to make the data that is coming from the R-GMA site-publisher, servicetool and GIN constantly available. By default the GLUE tables and service tables are archived, however this can be configured. Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

R-GMA Servers Per Node Type The gLite R-GMA Server should deployed the first as part of a gLite grid, because all services require it to publish service information. Deployment module Contains Used / included by R-GMA Server R-GMA server R-GMA registry server R-GMA schema server R-GMA browser R-GMA site publisher R-GMA archiver R-GMA servicetool MON-box R-GMA client RGMA client APIs UI, WN, SD R-GMA server,VOMS, LB, WMS, CE, DC, FTS, FTA, Hydra, I/O-Server,Torque Server R-GMA GIN R-GMA GadgetIN CE Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

GMA Producer- Consumer model The Producer stores its location (URL) in the Registry. The Consumer looks up producer URLs in the Registry. The Consumer contacts the Producer to get all the data. Data passes directly from the producer to the consumer: (does not through the registry) The user or any user application does not need to know the registry; this will be handled directly by the consumers and producers. Registry Producer Consumer Store Location Look up Location Execute or Stream data Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

Multiple Producers The Consumer will get all the URLs that could satisfy the query. The Consumer will connect to all the Producers. Producers that can satisfy the query will send the tuples to the Consumer. The Consumer will merge these tuples to form one result set. Registry TableName URL 1 URL 2 Producer 1 Producer 2 TableName Value 1 Value 2 TableName Value 3 Value 4 Consumer TableName Value 1 Value2 Value 3 Value 4 Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

Select * from CPULoad CPULoad (Consumer) Country Site Facility Load Timestamp UK RAL CDF 0.3 19055711022002 ATLAS 1.6 19055611022002 GLA 0.4 19055811022002 ALICE 0.5 CH CERN 0.9 0.6 19055511022002 CPULoad (Producer 2) UK GLA CDF 0.4 19055811022002 ALICE 0.5 19055611022002 CPULoad (Producer 1) UK RAL CDF 0.3 19055711022002 ATLAS 1.6 19055611022002 CPULoad (Producer 3) CH CERN ATLAS 1.6 19055611022002 CDF 0.6 19055511022002 Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

Joins Service URI VO type emailContact site gppse01 alice SE sysad@rl.ac.uk RAL atlas gppse02 cms lxshare0404 sysad@cern.ch CERN ServiceStatus URI VO type up status gppse01 alice SE y SE is running atlas gppse02 cms n SE ERROR 101 lxshare0404 Result Set (Consumer) URI emailContact gppse02 sysad@rl.ac.uk SELECT Service.URI Service.emailContact FROM Service S, ServiceStatus SS WHERE (S.URI= SS.URI and SS.up=‘n’) Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

R-GMA Concepts Producers Primary producer The producer creates and stores locally the informatiion Secondary producer The producer stores locally the information coming from primary producers On Demand The producer returns the information only under the request of the consumer Consumer Continuous As soon as new data becomes available it is broadcast to all interested parties Latest Corresponds to the intuitive idea of “current information” History Returns time sequenced data LATEST RETENTION PERIOD (LRP) and HISTORY RETENTION PERIOD (RTP) Allow producers to periodically purge old tuples, and to give a precise meaning to the “current state”. Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

R-GMA user interfaces R-GMA CLI Web interface Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

References GLUE Schema LCG-2 User Guide gLite 3.0 User Guide http://glueschema.forge.cnaf.infn.it/ LCG-2 User Guide https://edms.cern.ch/file/454439//LCG-2-UserGuide.html gLite 3.0 User Guide https://edms.cern.ch/file/722398/1.1/gLite-3-UserGuide.pdf R-GMA home page http://www.r-gma.org/ R-GMA in EGEE http://hepunx.rl.ac.uk/egee/jra1-uk/ Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

Questions… Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007

Hands On (Wiki pages) Browsing Information System (BDII/RGMA) https://grid.ct.infn.it/twiki/bin/view/GILDA/InformationSystems Advanced Queries to the MDS/BDII https://grid.ct.infn.it/twiki/bin/view/GILDA/BDIIMed Application Monitoring with RGMA https://grid.ct.infn.it/twiki/bin/view/GILDA/ApplicationMonitoringRGMA Acitrezza, First Internaional Grid School for Industrial Applications, 02.07.2007