CrossGrid Task 3.3 Grid Monitoring Trinity College Dublin (TCD) Brian Coghlan Paris MAR-2002

DataGrid WP3 CrossGrid Simulation Kernels Simulation Kernels Grid Visualisation Kernel Grid Visualisation Kernel User Interaction Service User Interaction Service User InteractiveSurgeryFloodManagement HEP Interactive Distributed Data Access HEP Data Mining on Grid HEP High LevelTrigger Weather Forecasting

Paris MAR-2002 DataGrid WP3 CrossGrid Architecture Applications And Supporting Tools Applications Development Support Grid Common Services Grid Visualisation Kernel Data Mining on Grid Data Mining on Grid Interactive Distributed Data Access Globus Replica Manager Roaming Access Grid Resource Management Grid Resource Management Grid Monitoring MPICH-G Distributed Data Collection User Interaction Service DataGrid Replica Manager DataGrid Replica Manager Datagrid Job Manager GRAM GSI Replica Catalog GASS MDS GridFTP Globus-IO Resource Manager Resource Manager CPU Resource Manager Resource Manager Resource Manager Resource Manager Secondary Storage Resource Manager Resource Manager Scientific Instruments (Medical Scaners, Satelites, Radars) Resource Manager Resource Manager Detector Local High Level Trigger Detector Local High Level Trigger Resource Manager Resource Manager VR systems (Caves, immerse desks) VR systems (Caves, immerse desks) Resource Manager Resource Manager Visualization tools Optimization of Data Access Tertiary Storage Local Resources Biomedical Application Biomedical Application Portal Performance Analysis MPI Verification Metrics and Benchmarks HEP High LevelTrigger Flood Application Flood Application HEP Interactive Distributed Data Access Application HEP Data Mining on Grid Application HEP Data Mining on Grid Application Weather Forecast application Weather Forecast application Copyright Marian Bubak et al CrossGrid DataGrid GLOBUS EXTERNAL

Paris MAR-2002 DataGrid WP3 Grid Services Copyright Marian Bubak et al Grid Visualisation Kernel Data Mining on Grid Data Mining on Grid Interactive Distributed Data Access Globus Replica Manager Roaming Access Grid Resource Management Grid Resource Management Grid Monitoring Distributed Data Collection User Interaction Service DataGrid Replica Manager DataGrid Replica Manager Datagrid Job Manager GRAM GSI Replica Catalog GASS MDS GridFTP Globus-IO

Paris MAR-2002 DataGrid WP3 CrossGrid WP2 Info Flows CrossGrid Technical Annex Fig. WP2-1 MPI verification (2.2) Benchmarks (2.3) Applications (WP1) executing on Grid testbed (WP4) Performance analysis (2.4) Automatic analysis Performance measurement Analytical model Visualization Application Source Code Grid Monitoring (3.3) Not now needed

Paris MAR-2002 DataGrid WP3 CrossGrid WP3 Info Flows CrossGrid Technical Annex Fig. WP3-1 WP3 Portals (3.1) Roaming Access (3.1) Grid Resource Management (3.2) Optimisation of Data Access (3.4) Tests and Integration (3.5) Applications WP1 End Users WP1, WP2, WP5Testbed WP4 Performance evaluation tools (2.4) Grid Monitoring (3.3)

Paris MAR-2002 DataGrid WP3 WP3 Grid Resource Management (3.2) Grid Monitoring (3.3) Optimisation of Data Access (3.4) Performance evaluation tools (2.4) input input input control result result result WP2 CrossGrid Task 3.3: External Info Flows

Paris MAR-2002 DataGrid WP3 CrossGrid Task 3.3: Subtasks Task (CYFRONET) Invasive Monitoring: ‘Autonomous monitoring system for on-line and automatic performance analysis ‘ performance analysis ‘ Task (TCD) Non-invasive Monitoring: ‘SQL-query-based tool support and interfaces to Grid application programming environment’ application programming environment’ Task (ICM) Jiro: ‘Jiro-based services for Grid infrastructure monitoring’

Paris MAR-2002 DataGrid WP3 CrossGrid Task 3.3: Internal Info Flows Trace info WP3 Jiro Services (3.3.3-ICM) OMIS Service Manager + Perf Tools (3.3.1-CYFRO) Result info Non-invasive Monitoring (3.3.2-TCD) input OMIS Application Monitor + Local Monitor (3.3.1-CYFRO) Jiro info OMIS info Applications Instruments Infrastructure input ctrl skt Performance Information Post-processing (3.3.1-ICM) input dB

Paris MAR-2002 DataGrid WP3 WP3 Jiro Services (3.3.3-ICM) R-GMA (DataGrid) OMIS Service Manager + Perf Tools (3.3.1-CYFRO) result R-GMA info (includes Jiro, MDS, Trace, OMIS & Result) input input input Non-invasive Monitoring (3.3.2-TCD) input MDS (Globus) OMIS Application Monitor + Local Monitor (3.3.1-CYFRO) MDS info Jiro info OMIS info Applications Instruments Infrastructure Static info input ctrl skt Performance Information Post-processing (3.3.1-ICM) input result dB CrossGrid Task 3.3: Can it use R-GMA ?

Paris MAR-2002 DataGrid WP3 Task (CYFRONET, Krakow, Poland) Invasive Monitoring: ‘Autonomous monitoring system for on-line and automatic performance analysis ‘ performance analysis ‘ Derived from APART research For performance analysis of parallel programs CrossGrid Task 3.3.1: Invasive Monitoring

Paris MAR-2002 DataGrid WP3 CrossGrid Task 3.3.1: Invasive Monitoring Autonomous monitoring system for on-line and automatic performance analysis RGMA Consumer API OMIS Interface Application Application Monitor Synchronous Control/Data via Globus sockets ? dB PerformanceInformationPost-processing(3.3.1-ICM) Invasive Monitoring Local Monitor RGMA Producer API Producer Servlet Performance Data Storage Consumer Servlet(s) Archiver Servlet Consumer Servlet RGMA Archiver API Service Manager Performance Tools Asynchronous Information via R-GMA ?

Paris MAR-2002 DataGrid WP3 CrossGrid Task 3.3.2: Non-invasive Monitoring High-Speed Data Capture Relational Trace Database Bulk Database Import Task (Trinity College Dublin, Ireland) Non-invasive Monitoring: ‘SQL-query-based tool support and interfaces to Grid application programming environment’ application programming environment’

Paris MAR-2002 DataGrid WP3 CrossGrid Task 3.3.2: Non-invasive Monitoring SQL-query-based tool support and interfaces to Grid application programming environment OMIS Interface Synchronous Control/Data via Globus sockets ? dB PerformanceAnalysisPost-processing(3.3.2-TCD) Non-invasive Monitoring File Producer Log Files RGMA Archiver API RGMA/OMIS interface Performance Tools Non-invasive Monitoring RGMA Consumer API RGMA Producer API Producer Servlet Consumer Servlet(s) Archiver Servlet Consumer Servlet RGMA Archiver API Asynchronous Information via R-GMA ?

Paris MAR-2002 DataGrid WP3 CrossGrid Task 3.3.3: Jiro Monitoring Task (ICM, University of Warsaw) Jiro: ‘Jiro-based services for Grid infrastructure monitoring’ New technology For distributed resource management Part of ‘Federated Management Architecture’

Paris MAR-2002 DataGrid WP3 CrossGrid Task 3.3.3: Jiro Monitoring Jiro-based services for Grid infrastructure monitoring’ Jiro Monitoring … Jiro Connections dB RGMA Consumer API RGMA Producer API Producer Servlet Consumer Servlet(s) Archiver Servlet Consumer Servlet Jiro Information via R-GMA ?

Paris MAR-2002 DataGrid WP3 Problems with R-GMA: 1. Sensor data MUST be pushed to Producer servlet e.g.for debugging, queries may focus on small section of logfiles, yet complete multi-GB logfiles must be move to Producer servlet 2. Each query MUST instantiate new objects & connections e.g. when debugging, have constant interaction with same producer, yet each query requires fresh instantiation of Consumer servlet + http connections

Paris MAR-2002 DataGrid WP3 Problem 1: dB RGMA Producer API Hidden Canonical Producer MySQL Query Engine JDBC log files Sensor Code database files DBProducer code implicit interface queriesresult-set queries result-set Canonical Producer API RGMA Canonical Producer log files Sensor Code Canonical Query Engine data seeks

Paris MAR-2002 DataGrid WP3 Problem 2: Synchronous Control/Data via Globus sockets OMIS Interface Application Application Monitor Local Monitor Performance Data Storage RGMA Consumer API dB RGMA Producer API Producer Servlet Consumer Servlet(s) Archiver Servlet Consumer Servlet Asynchronous Information via R-GMA ? OMIS Interface Service Manager Performance Tools Maybe: Query RGMA to find Local Monitor THEN set up Control/Data connection

Paris MAR-2002 DataGrid WP3 CrossGrid Task 3.3 THE END