R-GMA as an example of a generic framework for information exchange

Slides:



Advertisements
Similar presentations
21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson.
Advertisements

INFSO-RI Enabling Grids for E-sciencE Information and Monitoring Status and Plans GridPP18, Glasgow, Mar 2007.
INFSO-RI Enabling Grids for E-sciencE Building a robust distributed system: some lessons from R-GMA CHEP-07, Victoria,
EGEE is a project funded by the European Union under contract IST R-GMA: Status and Plans Antony Wilson / RAL GridPP 12 - Brunel
Data Management Expert Panel. RLS Globus-EDG Replica Location Service u Joint Design in the form of the Giggle architecture u Reference Implementation.
CoreGRID Workpackage 5 Virtual Institute on Grid Information and Monitoring Services Authorizing Grid Resource Access and Consumption Erik Elmroth, Michał.
Distributed Databases Logical next step in geographically dispersed organisations goal is to provide location transparency starting point = a set of decentralised.
OSG Logging Architecture Update Center for Enabling Distributed Petascale Science Brian L. Tierney: LBNL.
How WebMD Maintains Operational Flexibility with NoSQL Rajeev Borborah, Sr. Director, Engineering Matt Wilson – Director, Production Engineering – Consumer.
Module 18 Monitoring SQL Server 2008 R2. Module Overview Monitoring Activity Capturing and Managing Performance Data Analyzing Collected Performance Data.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Introduction to R-GMA: Relational Grid Monitoring Architecture.
The EU DataGrid – Information and Monitoring Services The European DataGrid Project Team
Oracle10g RAC Service Architecture Overview of Real Application Cluster Ready Services, Nodeapps, and User Defined Services.
Use of R-GMA in BOSS Henry Nebrensky (Brunel University) VRVS 26 April 2004 Some slides stolen from various talks at EDG 2 nd Review (
SOS EGEE ‘06 GGF Security Auditing Service: Draft Architecture Brian Tierney Dan Gunter Lawrence Berkeley National Laboratory Marty Humphrey University.
Introduction on R-GMA Shi Jingyan Computing Center IHEP.
CERN IT Department CH-1211 Genève 23 Switzerland t Internet Services Job Monitoring for the LHC experiments Irina Sidorova (CERN, JINR) on.
Application code Registry 1 Alignment of R-GMA with developments in the Open Grid Services Architecture (OGSA) is advancing. The existing Servlets and.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks R-GMA Now With Added Authorization Steve.
13 May 2004EB/TB Middleware meeting Use of R-GMA in BOSS for CMS Peter Hobson & Henry Nebrensky Brunel University, UK Some slides stolen from various talks.
WP3 Information and Monitoring Steve Fisher / RAL 23/9/2003.
Steve Traylen PPD Rutherford Lab Grid Operations PPD Christmas Lectures Steve Traylen RAL Tier1 Grid Deployment
Towards a Global Service Registry for the World-Wide LHC Computing Grid Maria ALANDES, Laurence FIELD, Alessandro DI GIROLAMO CERN IT Department CHEP 2013.
EGEE is a project funded by the European Union under contract IST R-GMA: Production Services for Information and Monitoring in the Grid John.
CERN IT Department CH-1211 Geneva 23 Switzerland t GDB CERN, 4 th March 2008 James Casey A Strategy for WLCG Monitoring.
INFSO-RI Enabling Grids for E-sciencE Information and Monitoring Status and Plans Plzeň, 10 July 2006 Steve Fisher/RAL.
WP3 RGMA Deployment Laurence Field / RAL Steve Fisher / RAL.
E-infrastructure shared between Europe and Latin America FP6−2004−Infrastructures−6-SSA gLite Information System Pedro Rausch IF.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks MSG - A messaging system for efficient and.
INFSO-RI Enabling Grids for E-sciencE Building a robust distributed system: some lessons from R-GMA WLCG Service Reliability.
MySQL and GRID status Gabriele Carcassi 9 September 2002.
DATABASE REPLICATION DISTRIBUTED DATABASE. O VERVIEW Replication : process of copying and maintaining database object, in multiple database that make.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI How to integrate portals with the EGI monitoring system Dusan Vudragovic.
CERN IT Department CH-1211 Genève 23 Switzerland t CERN IT Monitoring and Data Analytics Pedro Andrade (IT-GT) Openlab Workshop on Data Analytics.
EGEE is a project funded by the European Union under contract IST Information and Monitoring Services within a Grid R-GMA (Relational Grid.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Practical using R-GMA.
INFSO-RI Enabling Grids for E-sciencE gLite Information System: R-GMA Tony Calanducci INFN Catania gLite tutorial at the EGEE User.
CERN 21 January 2005Piotr Nyczyk, CERN1 R-GMA Basics and key concepts Monitoring framework for computing Grids – developed by EGEE-JRA1-UK, currently used.
DataTAG is a project funded by the European Union International School on Grid Computing, 23 Jul 2003 – n o 1 GridICE The eyes of the grid PART I. Introduction.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Introduction to R-GMA: Relational Grid Monitoring Architecture.
CERN IT Department CH-1211 Genève 23 Switzerland t CERN Agile Infrastructure Monitoring Pedro Andrade CERN – IT/GT HEPiX Spring 2012.
INFSO-RI Enabling Grids for E-sciencE Information and Monitoring Status and Plans GridPP-DB, IC London, 12 July, 2007.
The Mediator: What Next? Talk by: Andy Cooke Collaborators: Alasdair Gray, Lisha Ma, and Werner Nutt Heriot-Watt University.
Enabling Grids for E-sciencE Claudio Cherubino INFN DGAS (Distributed Grid Accounting System)
Network and Server Basics. Learning Objectives After viewing this presentation, you will be able to: Understand the benefits of a client/server network.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Nagios Grid Monitor E. Imamagic, SRCE OAT.
Introduction to Oracle Forms Developer and Oracle Forms Services
OGSA Information Abdeslem Djaoui OGSA Information, OGSA-WG #11
James Casey, CERN IT-GD WLCG Workshop 1st September, 2007
Grid Event Management Using R-GMA Monitoring Framework
Introduction to Oracle Forms Developer and Oracle Forms Services
Observer Design Pattern
Introduction to Oracle Forms Developer and Oracle Forms Services
Chapter 13 The Data Warehouse
Grid Portal Services IeSE (the Integrated e-Science Environment)
Practicals on R-GMA Valeria Ardizzone INFN
A Messaging Infrastructure for WLCG
Choosing the Discovery Model Martin Forsberg
Northbound API Dan Shmidt | January 2017
Concept of VLAN (Virtual LAN) and Benefits
SDMX Reference Infrastructure Introduction
Patterns.
Author: Laurence Field (CERN)
Cloud computing mechanisms
Software models - Software Architecture Design Patterns
R-GMA (Relational Grid Monitoring Architecture) for monitoring applications “s” gLite and LCG.
Chapter 2: Operating-System Structures
RELATIONAL GRID MONITORING ARCHITECHTURE
Chapter 2: Operating-System Structures
Information Services Claudio Cherubino INFN Catania Bologna
Presentation transcript:

R-GMA as an example of a generic framework for information exchange HPDC 2007 Workshop on Grid Monitoring, Monterey, USA, 25 June, 2007 Steve Fisher/RAL on behalf of JRA1-UK

Overview GMA and R-GMA Generic framework – pros and cons Custom solution – pros and cons Fixing the problems … Steve Fisher/RAL

GMA Defined by the GGF 3 Components Now OGF 3 Components Producer Consumer Registry Real system needs to tie down message formats This has been done by R-GMA Producer Registry Consumer Register Locate Query Data The INFOD-WG at GGF IBM, Oracle and others have defined a GMA compliant specification Relational (tabular) Looks similar to a a single RDBMS Tables can be dynamically added and removed from a schema Producers of information can be dynamically created and deleted Steve Fisher/RAL

R-GMA Relational implementation of the GGF’s GMA Provides a uniform method to publish and access both information and monitoring data Registry is hidden It is intended for use by: Other middleware components End users Easy for individuals to define, publish and retrieve data All data has a timestamp, enabling its use for monitoring Producer Registry Consumer Register Locate Query Data Relational (tabular) Looks similar to a a single RDBMS Tables can be dynamically added and removed from a schema Producers of information can be dynamically created and deleted Steve Fisher/RAL

R-GMA Producers Primary – source of data Secondary – republish data Co-locate information to speed up queries Reduce network traffic PP PP SP Query support Latest History Storage Memory: best performance for continuous queries Database: best performance for joins PP PP – Primary Producer SP – Secondary Producer PP Steve Fisher/RAL

R-GMA: Some Applications gridFTP monitoring (GridView) Job monitoring (GridView) Network performance monitoring Intrusion detection Application monitoring APEL Steve Fisher/RAL

R-GMA Patterns Secondary Producer Proprietary RDBMS PP C PP SP C PP C Steve Fisher/RAL

Benefits of R-GMA generic framework R-GMA will have been configured for you at each site No need to worry about firewalls Very easy to get started Define table(s) Publish from job or infrastructure No knowledge of consumers Consume from job or infrastructure No knowledge of producers Can set up one or more secondary producers to hold latest or historical information Steve Fisher/RAL

Problems with R-GMA generic framework You may not like the chosen data model Relational in the case of R-GMA Applications can interfere with each other Registry can be overloaded by unsuitable code running on a large number of worker nodes This is a GMA issue Steve Fisher/RAL

Benefits of custom solution For any one application you can almost always do better with something you write yourself It may even be possible to do this quite quickly Many applications have simple star topology e.g. APEL sends data to GOC Can replace knowledge of registry location by knowledge of central repository and just push data there Steve Fisher/RAL

Problems with the custom solution Pushing data is easy to configure but what do you do when data are blocked by firewall? Your single central node may not be able to cope; how do you then partition the data? How do you combine results from custom solution A with custom solution B? Beware of the difference between a prototype and deployment on 200 sites R-GMA has addressed all above Steve Fisher/RAL

Fixing the problems … Murphy’s law applies very well to distributed system One or more sites will be misconfigured or dead Must avoid single points of failure Obvious Must avoid evil interactions Less obvious Ultimately it is a plumbing problem All blockages are a problem Steve Fisher/RAL

Avoiding single points of failure Multiple Virtual Databases Separate registries for different communities Rogue user will only affect his own community Registry replication Registry failure is not then a problem Steve Fisher/RAL

Avoiding evil interactions Avoid queues of messages waiting to be sent Registry does not notify Prefer pull to push Slaves poll master Design system to not require responses to internal messages CHEP talk being prepared to go into many of the details Steve Fisher/RAL

Summary Generic solutions are hard to develop Generic distributed systems are very hard to develop Simple ad-hoc solutions seem attractive – but they must not be allowed to grow complex Simple solutions can turn out to be too restrictive Might solve by making it more complex Admit that problem is harder than anticipated If you feel that a custom solution is right for you then keep it very simple – and avoid feature creep Steve Fisher/RAL