21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson

21 Sep 2005LCG's R-GMA Applications A unified approach to I & M Provides information on both: –The Grid itself Mainly for the middleware packages, but the user may also query it to understand the status of the Grid –Grid applications For users Flexible infrastructure –All data has a timestamp enabling its use for monitoring –Dynamic addition and deletion of information producers –Dynamic creation and deletion of tables in schema –Able to cope with failures in a distributed environment

21 Sep 2005LCG's R-GMA Applications R-GMA – one cloud A relational implementation of GMA (from GGF) –Powerful data model and query language All data modelled as tables SQL can express complex queries in one expression Creates impression that you have one RDBMS per VO Producer Consumer R-GMA

Applications

21 Sep 2005LCG's R-GMA Applications Job Monitoring Requirements –Allow grid users to monitor the progress of their jobs –Allow VO administrators to get an overview of what is happening on the grid The Problem –The location where a grid job will run is not known –Worker nodes are behind firewall so are not accessible The Solution –R-GMA provides transport across firewalls The wrapper approach Instrumenting the code –Using a logging service –Calling R-GMA directly

21 Sep 2005LCG's R-GMA Applications

21 Sep 2005LCG's R-GMA Applications LCG Job Monitoring Deployed by LCG No changes needed to user code R-GMA Primary Producer SP R-GMA Secondary Producer PP Job wrapper PP User code Job wrapper PP User code SP Database SP

21 Sep 2005LCG's R-GMA Applications LCG Job Monitoring R-GMA Primary Producer SP R-GMA Secondary Producer PP Deployed by LCG Job wrapper PP User code Job wrapper PP User code SP Database SP L&B Server PP

21 Sep 2005LCG's R-GMA Applications Producers PP SP PP - Primary Producers Initial source of data Data published by user code stored by Producer Service SP- Secondary Producers Used to republish data in order to: co-locate information to speed up queries reduce network traffic use different producer properties

21 Sep 2005LCG's R-GMA Applications Producer Properties Producers may use: –Memory Gives best performance for continuous queries –Database Best performance for joins Primary and Secondary Producers must support –Continuous queries as soon as new data becomes available it is broadcast to all interested parties Primary and Secondary Producers may support –History Queries over time sequenced data –Latest Queries correspond to intuitive idea of current information

SEE DEMO Job Monitoring Web Interface

21 Sep 2005LCG's R-GMA Applications Application Monitoring Users can define their own tables C, C++, Java and Python apis available log4j appender available PP User code PP User code R-GMA Primary Producer SP R-GMA Secondary Producer PP SP Database SP

21 Sep 2005LCG's R-GMA Applications Network Monitoring Deployed by EGEE Data obtained from network monitoring sensors are published via R-GMA R-GMA Primary Producer SP R-GMA Secondary Producer PP Network sensor PP Network sensor SP Database SP

21 Sep 2005LCG's R-GMA Applications

21 Sep 2005LCG's R-GMA Applications GridFTP Monitoring Deployed by LCG Data are extracted from GridFTP logs and published via R-GMA R-GMA Primary Producer C R-GMA Consumer PP C Oracle RDBMS PP GridFTP log monitor PP GridFTP log monitor

21 Sep 2005LCG's R-GMA Applications Consumer Service Query types: –History Queries return time sequenced data –Latest Queries correspond to intuitive idea of current information –Continuous queries as soon as new data becomes available it is broadcast to all interested parties

SEE DEMO Grid View GridFTP Monitor

21 Sep 2005LCG's R-GMA Applications Network Monitoring Deployed by Grid Ireland TCP logs are queried in response to queries and the results are returned via R-GMA R-GMA Consumer ODP R-GMA On-Demand Producer C C User code C SP TCP Log ODP

21 Sep 2005LCG's R-GMA Applications On Demand Producers Provides a Producer interface to non relational data sources User has to write code to handle SQL Has been used to provide access to TCPdump data

21 Sep 2005LCG's R-GMA Applications R-GMA plans Will be taken into EGEE-II –Change from Servlets to Web Services –Add multi-VDB support –Support more than MySQL –Add authorization Conform to GGF/INFOD specification

21 Sep 2005LCG's R-GMA Applications Further Information R-GMA web site – JRA1-UK web site –

21 Sep 2005LCG's R-GMA Applications R-GMA – more detail Producer Service Registry Service Consumer Service API Mediator Schema Service Consumer application Producer application Publish Tuples Send Query Receive Tuples Register Locate Query Tuples SQL CREATE TABLE SQL INSERT SQL SELECT