BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 CrossGrid Architecture Marian Bubak and TAT Institute of Computer Science & ACC CYFRONET AGH, Cracow, Poland

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Overview –Applications and their requirements –Tools for X# applications development –New grid services –X# architecture

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Biomedical Application –Input: 3-D model of arteries –Simulation: LB of blood flow –Results: in a virtual reality –User: analyses results in near real-time, interacts, changes the structure of arteries

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Steering in the Biomedical Application CT / MRI scan Medical DB Segmentation Medical DB LB flow simulation VE WD PC PDA Visualization Interaction HDB 10 simulations/day 60 GB 20 MB/s

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 VR-Interaction

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Asynchronous Execution of Biomedical Application

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Interaction in Biomedical Application

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Cascade of Flood Simulations Data sources Meteorological simulations Hydraulic simulations Hydrological simulations Users Output visualization

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Basic Characteristics of Flood Simulation –Meteorological intensive simulation (1.5 h/simulation) – HPC large input/output data sets (50MB~150MB /event) high availability of resources (24/365) –Hydrological Parametric simulations - HTC Each sub-catchment may require different models –Hydraulic Many 1-D simulations - HTC 2-D hydraulic simulations need HPC

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Distributed Data Analysis in HEP –Objectives Distributed data access Distributed data mining techniques with neural networks –Issues Typical interactive requests will run on o(TB) distributed data Transfer/replication times of order of 1h Data transfers once and in advance of the interactive session. Allocation, installation and set up the corresponding database servers before the interactive session starts

Weather Forecast and Air Pollution Modeling –Distributed/parallel codes on Grid Coupled Ocean/Atmosphere Mesoscale Prediction System STEM-II Air Pollution Code –Integration of distributed databases –Data mining applied to downscaling weather forecast

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Key Features of X# Applications –Data Data generators and data bases geographically distributed Selected on demand –Processing Needs large processing capacity; both HPC & HTC Interactive –Presentation Complex data require versatile 3D visualisation Support interaction and feedback to other components

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Modules of Tool Environment Grid Monitoring (Task 3.3) Performance Prediction Component High Level Analysis Component User Interface and Visualization Component Performance Measurement Component Benchmarks (Task 2.3) Applications (WP1) executing on Grid testbed Application source code G-PM RMD PMD Legend RMD – raw monitoring data PMD – performance measurement data data flow manual information transfer

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Tools Environment and Grid Services Applications Portals (3.1) Portals (3.1) G-PM Performance Measurement Tools (2.4) G-PM Performance Measurement Tools (2.4) MPI Debugging and Verification (2.2) MPI Debugging and Verification (2.2) Metrics and Benchmarks (2.4) Metrics and Benchmarks (2.4) Grid Monitoring (3.3) (OCM-G, RGMA) Grid Monitoring (3.3) (OCM-G, RGMA)

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 User Interaction Service (1/3) –Synchronization between the simulation and visualization –Control of data flow between various Grid components –A plug-in mechanism for the required components –Software interfaces to: Resource broker (both DataGrid and Globus) Condor-G system for High Throughput Computing Nimrod/G tool for advanced parameter study Grid monitoring service and Globus GIS/MDS

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 User Interaction Service (2/3) User Interaction Services User Interaction Services User Interaction Services User Interaction Service Resource Broker Resource Broker Resource Broker Resource Broker Scheduler User Interaction Services User Interaction Services User Interaction Services Service Factory Visualization/ Interaction In VE Running Simulation Agent Visualization Agent Interaction Agent Event Notification Mechanism Other connections Data transfer Network Bandwidth Reservation Job Submission Service Portal

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 User Interaction Service (3/3) –Factory - creates UIS instances –UIS - event channel based service –Running Simulation – simulation sw –Simulation Agent – orchestrates simulation sw –Visualization/Interaction – virtual environment sw –Visualization Agent – orchestrates visualization sw –Interaction Agent – orchestrates interaction sw

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Portals and Roaming Access Applications Portals (3.1) Portals (3.1) Roaming Access Server (3.1) Scheduler (3.2) Scheduler (3.2) GIS / MDS (Globus) GIS / MDS (Globus) Grid Monitoring (3.3) Grid Monitoring (3.3) – Allow access user environment from remote computers – Independent of the system version and hardware – Run applications, manage data files, store personal settings Remote Access Server user profiles authentication, authorization job submission Migrating Desktop Application portal

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Monitoring System

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 OMIS Approach to Grid Monitoring –Application oriented –on-line –data collected immediately delivered to tools –normally no storing for later processing –Data collection based on run-time instrumentation –enables dynamic choosing of data to be collected –reduced monitoring overhead –Standardized interface between tools and the monitoring system – OMIS

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Service Managers and Monitors –Service Managers –one or more in the system –request distribution –reply collection –Local Monitors –one per node –handle local objects –actual execution of requests –Application Monitors –buffering data –filtering of instrumentation –monitoring requests

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Optimization of Grid Data Access Applications Portals (3.1) Portals (3.1) Optimization of Grid Data Access (3.4) Scheduling Agents (3.2) Scheduling Agents (3.2) Replica Manager (DataGrid / Globus) Replica Manager (DataGrid / Globus) Grid Monitoring (3.3) Grid Monitoring (3.3) GridFTP Service consists of Component-expert system Data-access estimator GridFTP plugin –Different storage systems and applications’ requirements –Optimization by selection of data handlers

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Building Blocks of the CrossGrid CrossGrid DataGrid GLOBUS EXTERNAL To be developed in X# From DataGrid Globus Toolkit Other

BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Overview of the CrossGrid Architecture Supporting Tools 1.4 Meteo Pollution 1.4 Meteo Pollution 3.1 Portal & Migrating Desktop Applications Development Support 2.4 Performance Analysis 2.4 Performance Analysis 2.2 MPI Verification 2.3 Metrics and Benchmarks 2.3 Metrics and Benchmarks App. Spec Services 1.1 Grid Visualisation Kernel 1.3 Data Mining on Grid (NN) 1.3 Data Mining on Grid (NN) 1.3 Interactive Distributed Data Access 3.1 Roaming Access 3.1 Roaming Access 3.2 Scheduling Agents 3.2 Scheduling Agents 3.3 Grid Monitoring 3.3 Grid Monitoring MPICH-G Fabric 1.1, 1.2 HLA and others 3.4 Optimization of Grid Data Access 3.4 Optimization of Grid Data Access 1.2 Flooding 1.2 Flooding 1.1 BioMed 1.1 BioMed Applications Generic Services GRAM GSI Replica Catalog GIS / MDS GridFTP Globus-IO DataGrid Replica Manager DataGrid Replica Manager DataGrid Job Submission Service Resource Manager (CE) Resource Manager (CE) CPU Resource Manager Resource Manager Resource Manager (SE) Resource Manager (SE) Secondary Storage Resource Manager Resource Manager Instruments ( Satelites, Radars) Instruments ( Satelites, Radars) 3.4 Optimization of Local Data Access 3.4 Optimization of Local Data Access Tertiary Storage Replica Catalog Globus Replica Manager Globus Replica Manager 1.1 User Interaction Services 1.1 User Interaction Services

BOF at GGF5, Edinburgh, Scotland, July 21-24,