M.Kunze, NEC2003, Varna The European CrossGrid Project Marcel Kunze Institute for Scientific Computing (IWR) Forschungszentrum Karlsruhe GmbH

M.Kunze, NEC2003, Varna Outline u Project Overview u Applications u Toolbox u Experience (Testbeds)

M.Kunze, NEC2003, Varna CrossGrid Collaboration Poland: Cyfronet & INP Cracow PSNC Poznan ICM & IPJ Warsaw Portugal: LIP Lisbon Spain: CSIC Santander Valencia & RedIris UAB Barcelona USC Santiago & CESGA Ireland: TCD Dublin Italy: DATAMAT Netherlands: UvA Amsterdam Germany: FZK Karlsruhe TUM Munich USTU Stuttgart Slovakia: II SAS Bratislava Greece: Algosystems Demo Athens AuTh Thessaloniki Cyprus: UCY Nikosia Austria: U.Linz 21 institutes 11 countries

M.Kunze, NEC2003, Varna GRIDLAB GRIA EGSO DATATAG DATAGRID GRIP EUROGRID DAMIEN Science IST Grid Project Space Industry/Business Science Applications Middleware Fabric CROSSGRID

M.Kunze, NEC2003, Varna Mission Statement Development of a Grid Environment for Interactive Applications Make the Grid more user friendly, secure and efficient Timeline:

M.Kunze, NEC2003, Varna Project Phases requirements definition and merging design, initial prototypes, refinement of requirements integration of components, additional prototypes complete integration, final versions of software components demonstration and documentation Months: first development phase: third development phase: final phase: second development phase:

M.Kunze, NEC2003, Varna Workpackages u WP1 – Application Development u WP2 – Grid Application Programming Environment u WP3 – New Grid Services and Tools u WP4 – International Testbed Organisation u WP5 – Project Management

M.Kunze, NEC2003, Varna Applications Middleware & Tools Fabric (Infrastructure) Architecture: Layered Structure

M.Kunze, NEC2003, Varna WP1 –Application Development 1.Interactive simulation and visualization of a biomedical system 2.Flooding crisis team support 3.Distributed data analysis in HEP 4.Weather forecast and air pollution modeling WP2 – Programming Environment 1.MPI code debugging and verification 2.Metrics and benchmarks 3.Interactive performance evaluation tools Visualization Kernel Data Mining WP3 – New Services Globus Middleware WP4 – Testbeds - Fabric Infrastructure Grid Services: DataGrid GriPhyN Portals and roaming access 2. Grid resource management 3. Grid monitoring 4. Optimization of data access High Level Architecture Technical Components

M.Kunze, NEC2003, Varna Detailed Architecture Supporting Tools 1.4 Meteo Pollution 1.4 Meteo Pollution 3.1 Portal & Migrating Desktop Applications Development Support 2.4 Performance Analysis 2.4 Performance Analysis 2.2 MPI Verification 2.3 Metrics and Benchmarks 2.3 Metrics and Benchmarks App. Spec Services 1.1 Grid Visualisation Kernel 1.3 Data Mining on Grid (NN) 1.3 Data Mining on Grid (NN) 1.3 Interactive Distributed Data Access 3.1 Roaming Access 3.1 Roaming Access 3.2 Scheduling Agents 3.2 Scheduling Agents 3.3 Grid Monitoring 3.3 Grid Monitoring MPICH-G Fabric 1.1, 1.2 HLA and others 3.4 Optimization of Grid Data Access 3.4 Optimization of Grid Data Access 1.2 Flooding 1.2 Flooding 1.1 BioMed 1.1 BioMed Applications Generic Services GRAM GSI Replica Catalog GIS / MDS GridFTP Globus-IO DataGrid Replica Manager DataGrid Replica Manager DataGrid Job Submission Service Resource Manager (CE) Resource Manager (CE) CPU Resource Manager Resource Manager Resource Manager (SE) Resource Manager (SE) Secondary Storage Resource Manager Resource Manager Instruments ( Satelites, Radars) Instruments ( Satelites, Radars) 3.4 Optimization of Local Data Access 3.4 Optimization of Local Data Access Tertiary Storage Replica Catalog Globus Replica Manager Globus Replica Manager 1.1 User Interaction Services 1.1 User Interaction Services CrossGrid DataGrid Globus

M.Kunze, NEC2003, Varna CrossGrid Applications

M.Kunze, NEC2003, Varna Key Features of CrossGrid Applications u Data n Data sources and data bases geographically distributed n To be selected on demand u Processing n Large processing capacity required; both HPC & HTC n Interactive u Presentation n Complex data requires versatile 3D visualisation n Support for interaction and feedback to other components

M.Kunze, NEC2003, Varna Biomedical Application CT / MRI scan Medical DB Segmentation Medical DB LB flow simulation VE WD PC PDA Visualization Interaction HDB 10 simulations/day 60 GB/simulation > 20 MB/s

M.Kunze, NEC2003, Varna Bypass Surgery: Simulated Treatment Planning

M.Kunze, NEC2003, Varna Interactive Treatment Planning The vascular geometry can be modified using a library of models Draw interactivelyComputational geometry

M.Kunze, NEC2003, Varna Sample pulsatile Flow Simulation

M.Kunze, NEC2003, Varna Flood Simulation Data sources Meteorological simulations Hydraulic simulations Hydrological simulations Users Output visualization

M.Kunze, NEC2003, Varna 3D Visualization

M.Kunze, NEC2003, Varna Flood Simulation: Flow and Water Depth

M.Kunze, NEC2003, Varna Distributed Data Analysis in HEP u Objectives n Distributed data analysis n Distributed data mining techniques with neural networks u Issues n Typical interactive requests will run on o(TB) distributed data n Transfer/replication times for the whole data about one hour n Data transfers once and in advance of the interactive session n Allocation, installation and set-up of corresponding database servers before the interactive session n Integration of user-friendly interactive access (based on PROOF)

M.Kunze, NEC2003, Varna Parallel ROOT Facility: PROOF Local Remote Selection Parameters Procedure Proc.C PROOF CPU TagD B RD B DB 1 DB 4 DB 5 DB 6 DB 3 DB 2

M.Kunze, NEC2003, Varna u Distributed/parallel codes on the Grid n Coupled Ocean/Atmosphere Mesoscale Prediction System n STEM-II Air Pollution Code u Integration of distributed databases u Data mining applied to downscaling weather forecast Weather Forecast and Air Pollution Modeling

M.Kunze, NEC2003, Varna Weather Forecast and Air Pollution Modeling

M.Kunze, NEC2003, Varna CrossGrid Toolbox

M.Kunze, NEC2003, Varna Migrating Desktop u Idea n Save and resume a user grid session n Look and feel of a windows desktop u Implementation n Roaming Access Server and Clients n Java Web Services (Portability) u Integration of Tools n Job submission wizard n Job monitoring dialog n GridExplorer dialog n GridCommander dialog

M.Kunze, NEC2003, Varna User Login Dialog

M.Kunze, NEC2003, Varna Grid Commander Tool

M.Kunze, NEC2003, Varna Grid Explorer Tool

M.Kunze, NEC2003, Varna Job Submission Dialog

M.Kunze, NEC2003, Varna Job Monitoring/ Visualisation Plugin

M.Kunze, NEC2003, Varna CrossGrid Testbeds Example sites: Valencia GoG farm, Santander (GridWall), FZK Various instances: Production, Validation, Test

M.Kunze, NEC2003, Varna Testbed Monitoring Mapcenter grid monitoring framework. Mapcenter was developed by DataGrid and adapted to CrossGrid.

M.Kunze, NEC2003, Varna Production Resource Broker Statistics Total users61 Submitted2903 Accepted2716 Matching2554 Sent by JSS2514 Jobs run2355 Jobs done2299 Graphics with RB statistics available from Mapcenter EDG 1.4

M.Kunze, NEC2003, Varna Production RB Statistics (continued) u Most of the failures are related with: n Authorization n Matchmaking n Testbed sites Submitted2903 Accepted2716 Matching2554 Sent by JSS2514 Jobs run2355 Jobs done Not accepted 162Matching failures 40Not submitted 159Didn’t run 56Didn’t reach the end 604Jobs failed EDG 1.4

M.Kunze, NEC2003, Varna The Future

M.Kunze, NEC2003, Varna 1980s: Internet 1990s: Web 2000s: Grid u Where do we need to get to ? n Applications to support an “e-society” (“Cyber-Infrastructure”) n A Grid infrastructure which hides the complexities from the users (“Invisible Computing”) n A powerful and flexible network infrastructure (GEANT 2) u Where do we need to invest ? n Applications targeted at realistic problems in “e-science” n Prototypes of Grid infrastructures n Maintain and improve the GEANT network u EU FP6 proposal: EGEE n Vision: Create European e-Infrastructure n 70 Partners, 31.5 MEuro n Start: April 2004