INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org www.sztaki.hu/~ghermann/szemelyes/ELTE_2006.

Slides:



Advertisements
Similar presentations
21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson.
Advertisements

INFSO-RI Enabling Grids for E-sciencE Information and Monitoring Status and Plans GridPP18, Glasgow, Mar 2007.
FP7-INFRA Enabling Grids for E-sciencE EGEE Induction Grid training for users, Institute of Physics Belgrade, Serbia Sep. 19, 2008.
INFSO-RI Enabling Grids for E-sciencE EGEE Middleware The Resource Broker EGEE project members.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Introduction to R-GMA: Relational Grid Monitoring Architecture.
INFSO-RI Enabling Grids for E-sciencE Comparison of LCG-2 and gLite Author E.Slabospitskaya Location IHEP.
Introduction on R-GMA Shi Jingyan Computing Center IHEP.
INFSO-RI Enabling Grids for E-sciencE Information System Valeria Ardizzone INFN Singapore, 1st South East Asia Forum -- EGEE tutorial.
FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Information System (IS) Valeria Ardizzone.
INFSO-RI Enabling Grids for E-sciencE Logging and Bookkeeping and Job Provenance Services Ludek Matyska (CESNET) on behalf of the.
Computational grids and grids projects DSS,
PHP and MySQL CS How Web Site Architectures Work  User’s browser sends HTTP request.  The request may be a form where the action is to call PHP.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Information System on gLite middleware Vincent.
E-infrastructure shared between Europe and Latin America Usage of R-GMA and Grid Application Monitoring Valeria Ardizzone INFN-Catania 1° EELA Grid School.
Application code Registry 1 Alignment of R-GMA with developments in the Open Grid Services Architecture (OGSA) is advancing. The existing Servlets and.
INFSO-RI Enabling Grids for E-sciencE R-GMA Server Installation Tony Calanducci INFN Catania - Italy First Latin American Workshop.
GLite Information System(s) Antonio Juan Rubio Montero CIEMAT 10 th EELA Tutorial. Madrid, May 7 th -11 th,2007.
EGEE is a project funded by the European Union under contract IST Outstanding design issues Stephen Hicks 23/06/04
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks R-GMA Now With Added Authorization Steve.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Using gLite API Vladimir Dimitrov IPP-BAS “gLite middleware Application Developers.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks AMGA PHP API Claudio Cherubino INFN - Catania.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Next steps with EGEE EGEE training community.
An information and monitoring system for static and dynamic information about grid resources, applications, networks … RDBMS Servlet aware of API during.
E-infrastructure shared between Europe and Latin America 12th EELA Tutorial for Users and System Administrators gLite Information System.
WP3 Authorization and R-GMA Linda Cornwall WP3 workshop 2-4 April 2003.
CLRC and the European DataGrid Middleware Information and Monitoring Services The current information service is built on the hierarchical database OpenLDAP.
EGEE is a project funded by the European Union under contract IST R-GMA: Production Services for Information and Monitoring in the Grid John.
Enabling Grids for E-sciencE Astronomical data processing workflows on a service-oriented Grid architecture Valeria Manna INAF - SI The.
WP3 RGMA Deployment Laurence Field / RAL Steve Fisher / RAL.
E-infrastructure shared between Europe and Latin America FP6−2004−Infrastructures−6-SSA gLite Information System Pedro Rausch IF.
INFSO-RI Enabling Grids for E-sciencE Building a robust distributed system: some lessons from R-GMA WLCG Service Reliability.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Progress on first user scenarios Stephen.
FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America gLite Information System Claudio Cherubino.
FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America R-GMA Server Installation Valeria Ardizzone.
E-infrastructure shared between Europe and Latin America gLite Information System(s) Manuel Rubio del Solar CETA-CIEMAT EELA Tutorial, Mérida,
The impact of R-GMA (upon WP1 and WP4). EDG (Paris) 6 Mar James MagowanImpact of R-GMA Grid Monitoring Architecture (GMA) We use it not only for.
INFSO-RI Enabling Grids for E-sciencE /10/20054th EGEE Conference - Pisa1 gLite Configuration and Deployment Models JRA1 Integration.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Practicals on RGMA Valeria Ardizzone INFN.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks APEL CPU Accounting in the EGEE/WLCG infrastructure.
INFSO-RI Enabling Grids for E-sciencE Information System Valeria Ardizzone INFN EGEE NA4 Generic Applications Meeting Catania,
INFSO-RI Enabling Grids for E-sciencE VOMS & MyProxy interaction Emidio Giorgio INFN NA4 Generic Applications Meeting 10 January.
EGEE is a project funded by the European Union under contract IST Information and Monitoring Services within a Grid R-GMA (Relational Grid.
FESR Trinacria Grid Virtual Laboratory Relational Grid Monitoring Architecture (R-GMA) Valeria Ardizzone INFN Catania Tutorial per Insegnanti.
INFSO-RI Enabling Grids for E-sciencE R-GMA Gergely Sipos and Péter Kacsuk MTA SZTAKI Credit to Valeria Ardizzone.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Practical using R-GMA.
INFSO-RI Enabling Grids for E-sciencE gLite Information System: R-GMA Tony Calanducci INFN Catania gLite tutorial at the EGEE User.
CERN 21 January 2005Piotr Nyczyk, CERN1 R-GMA Basics and key concepts Monitoring framework for computing Grids – developed by EGEE-JRA1-UK, currently used.
EGEE is a project funded by the European Union under contract IST R-GMA Security Stephen Hicks UK Cluster Security Middleware Security Group.
EGEE-II INFSO-RI Enabling Grids for E-sciencE More on gLite: 2 services you have not seen! Mike Mineter.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Introduction to R-GMA: Relational Grid Monitoring Architecture.
FESR Trinacria Grid Virtual Laboratory gLite Information System Muoio Annamaria INFN - Catania gLite 3.0 Tutorial Trigrid Catania,
EGEE-II INFSO-RI Enabling Grids for E-sciencE Architecture of LHC File Catalog Valeria Ardizzone INFN Catania – EGEE-II NA3/NA4.
Relational Grid Monitoring Architecture (R-GMA)
INFSO-SSA International Collaboration to Extend and Advance Grid Education Information System Valeria Ardizzone INFN Catania Corso di Grid Computing.
Grid Event Management Using R-GMA Monitoring Framework
R-GMA Command Line Tool
Information System Valeria Ardizzone INFN
The Information System
Hands-on on R-GMA Tony Calanducci INFN Catania
MON Box Installation and configuration
gLite Information System(s)
Practicals on R-GMA Valeria Ardizzone INFN
R-GMA Server Installation (v. 1.4)
gLite Information System
gLite Information System(s)
R-GMA (Relational Grid Monitoring Architecture) for monitoring applications “s” gLite and LCG.
Information and Monitoring System
RELATIONAL GRID MONITORING ARCHITECHTURE
gLite Information System
Information Services Claudio Cherubino INFN Catania Bologna
Presentation transcript:

INFSO-RI Enabling Grids for E-sciencE

INFSO-RI Enabling Grids for E-sciencE Information System Valeria Ardizzone INFN EGEE NA4 Generic Applications Meeting Catania, January 2006

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI Outline  Introduction to R-GMA and Grid Monitoring Architecture (GMA).  R-GMA within Testbed  R-GMA in depth: - Schema, Registry, Producer and Consumer - Query and Storage Types - R-GMA Browser  A use case of R-GMA for user application.

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI Introduction to R-GMA Relational Grid Monitoring Architecture (R-GMA) –Developed as part of the EuropeanDataGrid Project (EDG) –Now as part of the EGEE project. –Based the Grid Monitoring Architecture (GMA) from the Global Grid Forum (GGF). Uses a relational data model. –Data are viewed as tables. –Data structure defined by the columns. –Each entry is a row (tuple). –Queried using Structured Query Language (SQL).

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI Grid Monitoring Architecture(GMA) PRODUCER CONSUMER REGISTRY Store location Lookup location Transfer Data The Producer stores its location (URL) in the Registry. The Consumer looks up producer URLs in the Registry. The Consumer contacts the Producer to get all the data or the Consumer can listen to the Producer for new data.

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI R-GMA within Testbed

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI R-GMA: Schema-Registry-Mediator VIRTUAL DATABASE TABLE 1, Colum defs TABLE 2, Colum defs TABLE 3, Colum defs TABLE 4, Colum defs SCHEMA TABLE 1,Producer P1 details TABLE 2,Producer P1 details TABLE 2,Producer P2 details TABLE 2,Producer P3 details TABLE 3,Producer P2 details TABLE 3,Producer P1 details TABLE 3,Producer P3 details REGISTRY MEDIATOR R-GMA Server MEDIATOR: a set of rules for deciding which data providers to contact for any given query. REGISTRY: It holds the details of all producers that are publishing to tables in the virtual database and it also holds the details of “continuous” consumers. SCHEMA : it holds the names and definitions of all of the tables in the virtual database, and their authorization rules.

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI R-GMA: Producer-Consumer VIRTUAL DATABASE TABLE 1, Colum defs TABLE 2, Colum defs TABLE 3, Colum defs TABLE 4, Colum defs SCHEMA TABLE 1,Producer P1 details TABLE 2,Producer P1 details TABLE 2,Producer P2 details TABLE 2,Producer P3 details TABLE 3,Producer P2 details TABLE 3,Producer P1 details TABLE 3,Producer P3 details REGISTRY MEDIATOR P1 P2 P3 C1C2 SQL “INSERT” SQL “SELECT” Producers: are the data providers for the virtual database. Writing data into the virtual database is known as publishing, and data is always published in complete rows, known as tuples. There are three types of producer: Primary, Secondary and On-demand. Consumer: represents a single SQL SELECT query on the virtual database. The query is matched against the list of available producers in the Registry. The consumer service then selects the best set of producers to contact and sends the query directly to each of them, to obtain the answer tuples. R-GMA Server

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI Query and Storage Types Continuous: as soon as new data becomes available it is broadcast to all interested parties. Latest: correspond to intuitive idea of current information. History: return time sequenced data. TABLE 1,Producer P1 details TABLE 2,Producer P1 details TABLE 2,Producer P2 details TABLE 2,Producer P3 details TABLE 3,Producer P2 details TABLE 3,Producer P1 details TABLE 3,Producer P3 details REGISTRY P1 Latest-store Continuous&History-store P1 LATEST RETENTION PERIOD (LRP) and HISTORY RETENTION PERIOD (RTP) allow producers to periodically purge old tuples, and to give a precise meaning to the “current state”. Tuple-store can be in Memory or Database

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI Producer Types Primary Producer Secondary Producer On-Demand Producer User Code Producer API Producer Service Tuple Storage C Control only Queries Tuples SELECT * Tuples P User Code Producer API Producer Service C Control only Queries Tuples Queries User Code Producer API Producer Service Tuple Storage C Control and inserted tuples Queries Tuples

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI Continuous Producer Servlet Registry Store location Lookup location Continuous Store table description Producer API SQL “CREATE TABLE” Result Set TableName Value 1Value 2 TableNameURLPredicate Schema TableNameColumn TableName Value 1Value 2 Insert TableName UKRALAlice Consumer ServletConsumer API SQL “SELECT” TableName Value 1Value 2 TableName Value 1Value 2 Query SQL “INSERT”

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI History or Latest Producer Servlet Registry Store location Lookup location Query Store table description Producer API SQL “CREATE TABLE” Result Set TableName Value 1Value 2 TableNameURLPredicate Schema TableNameColumn TableName Value 1Value 2 Insert TableName UKRALAlice Consumer ServletConsumer API SQL “SELECT” TableName Value 1Value 2 TableName Value 1Value 2 Query SQL “INSERT”

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI R-GMA use case for User Application Table in Schema User Application User Producer and Consumer Testbed: GILDA JDL and script files USE CASE TIMELINE: To submit the JDL file from the GENIUS portal and monitoring its status. In the meantime, from RGMA Browser, monitoring the table and if there is any producers that are publishing tuples. If there is one, to send a query with a predicate to obtain the answer tuples. INGREDIENTS: WN Virtual Database R G M A Browser UICE USER APPLICATION …….. User Submit a Job that also contains its Producer executable Job is running Start User Producer with application data to publish C P P declare itself C select query Producer’s list C select query Query results

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI Create a table in Schema

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI R-GMA Browser as Consumer

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI User Producer and Consumer API available for Java, C, C++ and Python Users may by-pass API if they wish, but API is the easiest way to use R-GMA services

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI import org.glite.rgma.PrimaryProducer; //..... import org.glite.rgma.stubs.ProducerFactoryStub; public class PrimaryProducerExample { public static void main(String[] args) { //.... try { ProducerFactory factory = new ProducerFactoryStub(); TimeInterval ti = new TimeInterval(60, Units.MINUTES); ProducerProperties props = new ProducerProperties(Storage.MEMORY, 0); PrimaryProducer pp = factory.createPrimaryProducer(ti, props, null); String predicate = "WHERE userId = ’" + args[0] + "’"; TimeInterval historyRP = new TimeInterval(60, Units.MINUTES); TimeInterval latestRP = new TimeInterval(25, Units.MINUTES); pp.declareTable("userTable", predicate, historyRP, latestRP); String insert = "INSERT INTO userTable (userId, aString, aReal, anInt)" + " VALUES (’" + args[0] + "’, ’Java producer’, , 42)"; pp.insert(insert); pp.close(); } catch (RemoteException e) { /-.....*/} }

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI Producer Application (in Java)(1)..... ProducerProperties producerProps = null; if (producerType.equals("CONTINUOUS")) {producerProps = new ProducerProperties(Storage.MEMORY, 0);} else if (producerType.equals("LATEST")) {producerProps = new ProducerProperties(Storage.DATABASE, ProducerProperties.LATEST); } else if (producerType.equals("HISTORY")) { producerProps = new ProducerProperties(Storage.DATABASE, ProducerProperties.HISTORY); } else { System.err.println("Invalid producer type (" + producerType + ")."); System.exit(1); } Producer Properties Type: Primary Storage type: Database Termination Interval: 300 (seconds) Predicate: Where … Query type: HISTORY Latest Retention Period: 300 (seconds) History Retention Period: 300 (seconds)

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI PrimaryProducer pp = null; ResourceEndpoint endpoint = null; Try { ProducerFactory pf = new ProducerFactoryStub(); TimeInterval ti = new TimeInterval(terminationInterval, Units.SECONDS); pp = pf.createPrimaryProducer(ti, producerProps, null); endpoint = pp.getResourceEndpoint(); String predicate = "WHERE ID = '" + Id + "'"; pp.declareTable(tableName, predicate, new TimeInterval(historyRP, Units.SECONDS), new TimeInterval(latestRP, Units.SECONDS)); Producer Application (in Java)(2)

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI Producer Application (in Java)(3)..... String insert = "INSERT INTO "+ tableName + " (ID, JobDone, Param, HostCE, Owner) VALUES ('" + Id + "','" + per + "','" + i + "','" + hostce + "','" + owner + "')"; pp.insert(insert);..... pp.close();

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI JDL with User Producer Application [ Type = "Job"; JobType = "Normal"; Executable="startPP.sh"; Arguments = "100 HISTORY Valeria_Ardizzone"; StdOutput="stdout.log"; StdError="stderr.log"; InputSandbox={"startPP.sh","pp.class"}; OutputSandbox={"stdout.log","stderr.log"}; ……. ]

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI JDL Submission from GENIUS

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI User Producer in R-GMA Browser

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI Query from R-GMA Browser

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI Query Results

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI Job Output in GENIUS

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI Security: Requirements R-GMA Consumer users: who requests information. Producer users: who provides information. Site administrators: who runs R-GMA services. Virtual Organizations: who “owns” the schema and registry. Consumer Provider Site Admin VO

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI Security: Solution R-GMA Consumer Provider Site Admin VO Mutual Autentication: guaranteeing who is at each end of an exchange of messages. Encryption: using an encrypted transport protocol (HTTPS). Authorization: implicit or explicit.

EGEE NA4 Meeting - Catania, January 2006 Enabling Grids for E-sciencE INFSO-RI More information R-GMA overview page. – R-GMA documentation in EGEE – R-GMA in GILDA –