EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org Introduction to R-GMA: Relational Grid Monitoring Architecture.

Slides:



Advertisements
Similar presentations
21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson.
Advertisements

INFSO-RI Enabling Grids for E-sciencE Information and Monitoring Status and Plans GridPP18, Glasgow, Mar 2007.
FP7-INFRA Enabling Grids for E-sciencE EGEE Induction Grid training for users, Institute of Physics Belgrade, Serbia Sep. 19, 2008.
GGF Toronto Spitfire A Relational DB Service for the Grid Peter Z. Kunszt European DataGrid Data Management CERN Database Group.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Introduction to R-GMA: Relational Grid Monitoring Architecture.
INFSO-RI Enabling Grids for E-sciencE Comparison of LCG-2 and gLite Author E.Slabospitskaya Location IHEP.
Introduction on R-GMA Shi Jingyan Computing Center IHEP.
INFSO-RI Enabling Grids for E-sciencE gLite Data Management Services - Overview Mike Mineter National e-Science Centre, Edinburgh.
INFSO-RI Enabling Grids for E-sciencE Federated Network Performance Monitoring for the Grid K. Kavoussanakis, EPCC, The University.
FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Information System (IS) Valeria Ardizzone.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Simply monitor a grid site with Nagios J.
INFSO-RI Enabling Grids for E-sciencE Workload Management System Mike Mineter
1 DIRAC – LHCb MC production system A.Tsaregorodtsev, CPPM, Marseille For the LHCb Data Management team CHEP, La Jolla 25 March 2003.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Information System on gLite middleware Vincent.
Application code Registry 1 Alignment of R-GMA with developments in the Open Grid Services Architecture (OGSA) is advancing. The existing Servlets and.
INFSO-RI Enabling Grids for E-sciencE R-GMA Server Installation Tony Calanducci INFN Catania - Italy First Latin American Workshop.
GLite Information System(s) Antonio Juan Rubio Montero CIEMAT 10 th EELA Tutorial. Madrid, May 7 th -11 th,2007.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks R-GMA Now With Added Authorization Steve.
WP3 Information and Monitoring Steve Fisher / RAL 23/9/2003.
E-infrastructure shared between Europe and Latin America 12th EELA Tutorial for Users and System Administrators gLite Information System.
EGEE is a project funded by the European Union under contract IST R-GMA: Production Services for Information and Monitoring in the Grid John.
WP3 RGMA Deployment Laurence Field / RAL Steve Fisher / RAL.
INFSO-RI Enabling Grids for E-sciencE
E-infrastructure shared between Europe and Latin America FP6−2004−Infrastructures−6-SSA gLite Information System Pedro Rausch IF.
Grid Deployment Enabling Grids for E-sciencE BDII 2171 LDAP 2172 LDAP 2173 LDAP 2170 Port Fwd Update DB & Modify DB 2170 Port.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE Site Architecture Resource Center Deployment Considerations MIMOS EGEE Tutorial.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks MSG - A messaging system for efficient and.
INFSO-RI Enabling Grids for E-sciencE The gLite File Transfer Service: Middleware Lessons Learned form Service Challenges Paolo.
INFSO-RI Enabling Grids for E-sciencE Building a robust distributed system: some lessons from R-GMA WLCG Service Reliability.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Progress on first user scenarios Stephen.
INFSO-RI Enabling Grids for E-sciencE ARDA Experiment Dashboard Ricardo Rocha (ARDA – CERN) on behalf of the Dashboard Team.
 CMS data challenges. The nature of the problem.  What is GMA ?  And what is R-GMA ?  Performance test description  Performance test results  Conclusions.
SAM Sensors & Tests Judit Novak CERN IT/GD SAM Review I. 21. May 2007, CERN.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Grid2Win : gLite for Microsoft Windows Roberto.
FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America R-GMA Server Installation Valeria Ardizzone.
E-infrastructure shared between Europe and Latin America gLite Information System(s) Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida,
INFSO-RI Enabling Grids for E-sciencE Web Services Mike Mineter National e-Science Centre, Edinburgh.
INFSO-RI Enabling Grids for E-sciencE Information System Valeria Ardizzone INFN EGEE NA4 Generic Applications Meeting Catania,
An Active Security Infrastructure for Grids Stuart Kenny*, Brian Coghlan Trinity College Dublin.
EGEE is a project funded by the European Union under contract IST Information and Monitoring Services within a Grid R-GMA (Relational Grid.
FESR Trinacria Grid Virtual Laboratory Relational Grid Monitoring Architecture (R-GMA) Valeria Ardizzone INFN Catania Tutorial per Insegnanti.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Regional Nagios Emir Imamagic /SRCE EGEE’09,
EGEE-II INFSO-RI Enabling Grids for E-sciencE Practical using WMProxy advanced job submission.
INFSO-RI Enabling Grids for E-sciencE R-GMA Gergely Sipos and Péter Kacsuk MTA SZTAKI Credit to Valeria Ardizzone.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks gLite – UNICORE interoperability Daniel Mallmann.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Practical using R-GMA.
INFSO-RI Enabling Grids for E-sciencE gLite Information System: R-GMA Tony Calanducci INFN Catania gLite tutorial at the EGEE User.
CERN 21 January 2005Piotr Nyczyk, CERN1 R-GMA Basics and key concepts Monitoring framework for computing Grids – developed by EGEE-JRA1-UK, currently used.
INFSO-RI Enabling Grids for E-sciencE NPM Security Alistair K Phipps (NeSC) JRA4 Face To Face, CERN, Geneva.
EGEE-II INFSO-RI Enabling Grids for E-sciencE More on gLite: 2 services you have not seen! Mike Mineter.
SAM Status Update Piotr Nyczyk LCG Management Board CERN, 5 June 2007.
INFSO-RI Enabling Grids for E-sciencE File Transfer Software and Service SC3 Gavin McCance – JRA1 Data Management Cluster Service.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Architecture of LHC File Catalog Valeria Ardizzone INFN Catania – EGEE-II NA3/NA4.
1 Copyright © 2008, Oracle. All rights reserved. Repository Basics.
INFSO-RI Enabling Grids for E-sciencE GOCDB2 Matt Thorpe / Philippa Strange RAL, UK.
Relational Grid Monitoring Architecture (R-GMA)
Grid Event Management Using R-GMA Monitoring Framework
R-GMA Command Line Tool
Information System Valeria Ardizzone INFN
The Information System
MON Box Installation and configuration
gLite Information System(s)
R-GMA as an example of a generic framework for information exchange
Practicals on R-GMA Valeria Ardizzone INFN
gLite Information System
gLite Information System(s)
R-GMA (Relational Grid Monitoring Architecture) for monitoring applications “s” gLite and LCG.
Information and Monitoring System
gLite Information System
The gLite Information System
Presentation transcript:

EGEE-II INFSO-RI Enabling Grids for E-sciencE Introduction to R-GMA: Relational Grid Monitoring Architecture

Enabling Grids for E-sciencE EGEE-II INFSO-RI Acknowledgements Slides are taken/derived from the GILDA team Steve Fisher (RAL, UK) and the R-GMA team

Enabling Grids for E-sciencE EGEE-II INFSO-RI What is R-GMA ? Uniform method to access and publish both information and monitoring data. From a user's perspective, an R-GMA installation currently appears similar to a single relational database. GMA (Grid Monitoring Architecture) was developed by the GGF R-GMA (Relational GMA) was created: –To simplify use of GMA (servers “know” about registries, not the client software) –To give a relational view

Enabling Grids for E-sciencE EGEE-II INFSO-RI Introduction to R-GMA Relational Grid Monitoring Architecture (R-GMA) –Developed as part of the EuropeanDataGrid Project (EDG) –Now as part of the EGEE project. –Evolution from the Grid Monitoring Architecture (GMA) Uses a relational data model. –Data are viewed as a table. –Data structure defined by the columns. –Each entry is a row (tuple). –Queried using Structured Query Language (SQL). nameIDbirthGroup SELECT * FROM people WHERE group=‘HR’ Tom HR

Enabling Grids for E-sciencE EGEE-II INFSO-RI R-GMA There is no central repository!!! There is only a “Virtual Database”. Schema is a list of table definitions: additional tables/schema can be defined by applications Registry is a list of data producers with all its details. Producers publish data. Consumers read data published. VIRTUAL DATABASE TABLE 1, Colum defs TABLE 2, Colum defs TABLE 3, Colum defs TABLE 4, Colum defs SCHEMA TABLE 1,Producer P1 details TABLE 2,Producer P1 details TABLE 2,Producer P2 details TABLE 2,Producer P3 details TABLE 3,Producer P2 details TABLE 3,Producer P1 details TABLE 3,Producer P3 details REGISTRY MEDIATOR P1 P2 P3 C1C2 SQL “CREATE TABLE” SQL “INSERT” SQL “SELECT”

Enabling Grids for E-sciencE EGEE-II INFSO-RI Service orientation PRODUCER CONSUMER REGISTRY Store location Lookup location Transfer Data The Producer stores its location (URL) in the Registry. The Consumer looks up producer URLs in the Registry. The Consumer contacts the Producer to get all the data or the Consumer can listen to the Producer for new data.

Enabling Grids for E-sciencE EGEE-II INFSO-RI Consumer Producer 1 Registry Virtual database TableName Value 1Value2 Value 3Value 4 TableName Value 1Value 2 TableNameURL 1 TableNameURL 2 The Consumer interrogates the Registry to identify all Producers that could satisfy the query. Consumer connects to the Producers. Producers send the tuples to the Consumer. The Consumer will merge these tuples to form one result set. Producer 2 TableName Value 3Value 4

Enabling Grids for E-sciencE EGEE-II INFSO-RI Service URIVOtype Contactsite ServiceStatus URIVOtypeupstatus gppse01aliceSEySE is running gppse01atlasSEySE is running gppse02cmsSEnSE ERROR 101 lxshare0404aliceSEySE is running lxshare0404atlasSEySE is running Result Set (Consumer) URI Contact SELECT Service.URI Service. Contact FROM Service S, ServiceStatus SS WHERE (S.URI= SS.URI and SS.up=‘n’) Joins

Enabling Grids for E-sciencE EGEE-II INFSO-RI Roles R-GMA Consumer users: who request information. Producer users: who provide information. Site administrators: who run R-GMA services. Virtual Organizations: who “own” the schema and registry. Consumer Provider Site Admin VO

Enabling Grids for E-sciencE EGEE-II INFSO-RI Security R-GMA Consumer Provider Site Admin VO Mutual Authentication: guaranteeing who is at each end of an exchange of messages. Encryption: using an encrypted transport protocol (HTTPS). Authorization: implicit or explicit.

Enabling Grids for E-sciencE EGEE-II INFSO-RI Deployment Producer and Consumer Services are typically on a one per site basis Centralized Registry and Schema. The Registry and Schema may be replicated, to avoid a single point of failure –… when you use RGMA CLI you will see which are being used

Enabling Grids for E-sciencE EGEE-II INFSO-RI Producer Types Primary Producer Secondary Producer On-Demand Producer No internal storage Queries passed to user code User Code Producer API Producer Service Tuple Storage C Control and inserted tuples Queries Tuples User Code Producer API Producer Service Tuple Storage C Control only Queries Tuples SELECT * Tuples P User Code Producer API Producer Service C Control only Queries Tuples Queries User Code

Enabling Grids for E-sciencE EGEE-II INFSO-RI Query Types Continuous Latest History Static P1 TABLE 1,Producer P1 details TABLE 2,Producer P1 details TABLE 2,Producer P2 details TABLE 2,Producer P3 details TABLE 3,Producer P2 details TABLE 3,Producer P1 details TABLE 3,Producer P3 details REGISTRY

Enabling Grids for E-sciencE EGEE-II INFSO-RI Continuous Producer Servlet Registry Store location Lookup location Continuous Store table description Producer API SQL “CREATE TABLE” Result Set TableName Value 1Value 2 TableNameURLPredicate Schema TableNameColumn TableName Value 1Value 2 Insert TableName UKRALAlice Consumer ServletConsumer API SQL “SELECT” TableName Value 1Value 2 TableName Value 1Value 2 Query SQL “INSERT”

Enabling Grids for E-sciencE EGEE-II INFSO-RI Query Types Continuous Latest History Static P1 TABLE 1,Producer P1 details TABLE 2,Producer P1 details TABLE 2,Producer P2 details TABLE 2,Producer P3 details TABLE 3,Producer P2 details TABLE 3,Producer P1 details TABLE 3,Producer P3 details REGISTRY

Enabling Grids for E-sciencE EGEE-II INFSO-RI History or Latest Producer Servlet Registry Store location Lookup location Query Store table description Producer API SQL “CREATE TABLE” Result Set TableName Value 1Value 2 TableNameURLPredicate Schema TableNameColumn TableName Value 1Value 2 Insert TableName UKRALAlice Consumer ServletConsumer API SQL “SELECT” TableName Value 1Value 2 TableName Value 1Value 2 Query SQL “INSERT”

Enabling Grids for E-sciencE EGEE-II INFSO-RI Query Types Continuous Latest History Static Latest Retention Period History Retention Period P1 TABLE 1,Producer P1 details TABLE 2,Producer P1 details TABLE 2,Producer P2 details TABLE 2,Producer P3 details TABLE 3,Producer P2 details TABLE 3,Producer P1 details TABLE 3,Producer P3 details REGISTRY P1 Latest-store Continuous&History-store

Enabling Grids for E-sciencE INFSO-RI GridFTP Monitoring (gridView) SA1 have written script to “tail” FTP logs and publish via PP on gridFTP server nodes Continuous query pulls all the data to a central location and writes to an Oracle database for analysis Used for Service Challenge 3 PP C Oracle

Enabling Grids for E-sciencE INFSO-RI Job Monitoring (L&B) Reads L&B logs on the resource broker nodes. Publishes data on state of jobs A database secondary producer is used to aggregate the data as well as a gridView consumer. CMS dashboard – PP C Oracle SP C C C

Enabling Grids for E-sciencE INFSO-RI Job Monitoring (WN) On the WNs, the Job Wrapper (if enabled by JDL) periodically publishes information about the state of the process running the job and its environment. A database secondary producer is used to aggregate the data. – GMABrowser/Browser.do/queryTable?selectQueryType=latest& duration=20&tableName=JobMonitorhttps://rgma13.pp.rl.ac.uk:8443/R- GMABrowser/Browser.do/queryTable?selectQueryType=latest& duration=20&tableName=JobMonitor PP SP C C C

Enabling Grids for E-sciencE INFSO-RI NPM Frameworks: e2emonit Network performance data important: –to detect and resolve network problems. –to intelligently schedule jobs based on network load and reliability. active measurements between end-sites, using tools such as –iperf, –udpmon –ping. PP SP C NM-WG WS /npm-dt/query.jsphttps://egee.epcc.ed.ac.uk: 28443/npm-dt/query.jsp

Enabling Grids for E-sciencE INFSO-RI NPM Diagnostic Tool

Enabling Grids for E-sciencE INFSO-RI NPM DT Scenario - results

Enabling Grids for E-sciencE INFSO-RI Intrusion Detection OnDemandProducer API PrimaryProducer API The Grid intrusion detection work is now within the Interactive European Grid ( grid.eu ) project, as part of the JRA workpackage, and is known as Active Security ( grid.euhttp://

Enabling Grids for E-sciencE INFSO-RI Service Discovery Questions to answer: –“I am at CERN, in 'dteam' VO. Where is a MyProxy server?” –glite-sd-query -t myproxy -s CERN-PROD Service Discovery offers: –client API (library) to hide the differences –plug-in architecture to simplify dependencies –uses the subset of Glue schema as data model –simple API, no complex queries –CLI for other tools and testing Plug-ins for: –BDII –R-GMA –MDS4 (not yet) –File (only for testing)

Enabling Grids for E-sciencE INFSO-RI Service Discovery

Enabling Grids for E-sciencE INFSO-RI TCD R-GMA related projects TCD: Trinity College Dublin gridFS: a grid filesystem InfoGrid: a grid using an information model Keith Rochford's work on grid service monitoring Adaptive eLearning: R-GMA is the first course Shared memory for grids (SMG)

Enabling Grids for E-sciencE EGEE-II INFSO-RI R-GMA APIs APIs exist in Java, C, C++, Python. –For clients (servlets contacted behind the scenes) They include methods for… –Creating consumers –Creating primary and secondary producers –Setting type of queries, type of produces, retention periods, time outs… –Retrieving tuples, inserting data –… You can create your own Producer or Consumer.

Enabling Grids for E-sciencE EGEE-II INFSO-RI Overview of practical We will use a client that gives command-line interfaces to both consumers and producers We will explore the tables on the R-GMA service provided on GILDA Use a table that is set up for training purposes to produce and consume data Now please follow the “more information” link

Enabling Grids for E-sciencE EGEE-II INFSO-RI R-GMA practical html page

Enabling Grids for E-sciencE EGEE-II INFSO-RI Batch Mode The command line tool can be used in batch mode in three ways: – rgma –c Executes and exits. The –c option may be specified more than once. – rgma –f Executes commands in sequentially then exits. Each line should contain one command. –Embedded in a shell script

EGEE-II INFSO-RI Enabling Grids for E-sciencE R-GMA Browser

Enabling Grids for E-sciencE EGEE-II INFSO-RI Table description

Enabling Grids for E-sciencE EGEE-II INFSO-RI R-GMA Browser as Consumer

Enabling Grids for E-sciencE EGEE-II INFSO-RI Query from R-GMA Browser

Enabling Grids for E-sciencE EGEE-II INFSO-RI Query Results

Enabling Grids for E-sciencE EGEE-II INFSO-RI More information R-GMA overview page. – R-GMA in EGEE – R-GMA command line tool – R-GMA Browser Home Page –