1 Development of GRID environment for interactive applications Jesús Marco de Lucas Instituto de Física de Cantabria, IFCA Consejo Superior de Investigaciones Científicas, CSIC, Santander, SPAIN DATAGRID DISSEMINATION DAY 14-V-2003 BARCELONA

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 2 The EU CrossGrid Project European Project ( ~5 M€, 3 year project started March 2002 ) proposed to CPA9, 6th IST call, V FP Polish (Cracow & Poznan) / Spanish (CSIC & CESGA) / German (FZK) initiative with the support of CERN ( thanks to Fab!) CYFRONET (Cracow) is the coordinator of the project (Michal Turala, project leader) Objectives: Extension of GRID in Europe, assuring interoperability with DataGrid Interactive Applications (“human in the loop”): Environmental fields (meteorology/air pollution, flooding crisis management) High Energy Physics (interactive analysis over distributed datasets) Medicine (vascular surgery preparation) Need: Develop corresponding middleware and tools Deploy on a pan-european testbed Partners: Poland (CYFRONET, PSNC, ICM, INP, INS), Spain (CSIC: IFCA, IFIC, RedIRIS, UAB, USC), Germany (FZK, USTUTT, TUM), Slovakia (II SAS), Ireland (TCD), Portugal (LIP), Austria (U.Linz), The Nederlands(UvA), Greece (DEMO, AuTH), Cyprus (UCY) Industry: Datamat (I), Algosystems (Gr)

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 3 Surgical Planning Problem: vascular diseases Solution: placement of a bypass by a surgeon Planning for intervention is based on 3D images obtained from MRI or CT scans. The attainable improvement in blood flow should determine which possibility is the best for a particular patient. A 3D arterial model is built on the basis of the images, and presented to the surgeon in an inmersive intuitive environment A CT scanner Stenosis (narrowing of an artery) Viewing the arterial structure in an immersive 3D environment Observation

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 4 Surgical Planning Goal: Simulate vascular reconstruction Method: Interactive Virtual Reality Environment to View scanned data Define proposed interventions View simulation results Advanced fluid code to simulate flows Arterial structures from scans with proposed bypasses Simulated flows Need Grid in interactive mode (the surgeon should not wait long…) Access distributed computational resources for flow simulation and visualization, so get a high performance environment at low cost Distribute simulations for different bypass configurations

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 5 Flood management Problem: flooding crisis in Slovakia Solution: monitoring, forecasting, simulation, real-time actions Precipitation forecasts based on meteorological simulations of different resolution from the meso-scale to the storm- scale. For flash floods, high-resolution (1 km) regional atmospheric models have to be used along with remote sensing data (satellite, radar) From the quantitative precipitation forecast, hydrological models are used to determine the discharge from the affected area. Then hydraulic models simulate water flow through various river structures to predict the impact of the flood Crisis management teams should consult various experts, before making any decisions. The experts should be able to run simulations with different parameters and analyze the impact (“what-if” analysis). monitoring forecasting simulation

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 6 Flood management Goal: Flooding risk prediction Method: Cascade of simulations Meteorological Hydrological Hydraulic Virtual Organization Need Grid in interactive mode (simulation results for “what-if” ) seamlessly connect together experts, data and computing resources needed for quick decisions highly automated early warning system, based on hydro-meteorological (snowmelt) rainfall- runoff simulations

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 7 Flood management Web portal for access Job submission Visualization See DEMO outside

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 8 HEP interactive analysis The next Large Hadron Collider (LHC) at CERN, will accelerate protons to an energy enough to produce a particle hundreds times heavier: the Higgs Boson, the last piece in the Standard Model, key for understanding the origin of the mass. Problem: All collisions will be recorded by sophisticated detectors, and the information stored in distributed databases with a volume of millions of gigabytes. But only few of those complex collisions will produce a Higgs Boson… Solution: On-line filtering techniques + sophisticated mathematical algorithms for physics analysis, like neural networks Physicists across the world are collaborating in this search… level 1 - special hardware 40 MHz (40 TB/sec) 75 KHz (75 GB/sec) 5 KHz (5 GB/sec) 100 Hz (100 MB/sec) data recording & offline analysis level 2 - embedded processors level 3 - PCs

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 9 HEP interactive analysis Goal: Physics analysis on large distributed databases Method: Distributed computing for Access to databases Complex algorithms, like Neural Networks Use Web Portal as GUI Need Grid in interactive mode (physicists try different hypos) Reduce the waiting time to test a new algorithm or a new hypothesis from hours down to minutes by processing in distributed mode (DEMO TODAY)

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 10 Meteo and Air Pollution Problem: Improve local predictions and refine air-pollution modeling close to a thermical power plant. Solution: data-mining on databases of outputs from atmospheric circulation models, to improve downscaling Typical database ( ERA-15, ECMWF) Daily forecasts on a reticule covering the globe from Atmospheric circulation pattern: v=( T( 1ooomb ), T( 850mb ),...,Z,H... ) The dimension can reach 10 4

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 11 Meteo and Air Pollution Goal: Data-mining on databases and improvement on air- pollution prediction Method: Distributed computing for Data-Mining algorithm SOM Air-Pollution STEM II Need Grid in interactive mode (so the power plant reacts on time) Try different air-pollution estimations according to meteo predictions Atmospheric circulation pattern: v=( T( 1ooomb ), T( 850mb ),...,Z,H... ) The dimension can reach 10 4 SIMILAR PATTERNS close in the grid and in the CPs space!! 2/1/1979 1/1/1979

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 12 Application development Good interaction with final user community (clear use cases) Vascular Surgery: Leiden Hospital Flooding crisis management: authorities in Slovakia HEP interactive physics analysis: LHC physicists Meteo and Air Pollution: power plant managers Middleware and Tools (significative effort) : Basic middleware: Globus 2 + DataGrid Distributed computing using MPI: MPICH-G2 Support for correct use of MPI: profiling interface (MARMOT) Benchmarking on a grid context and performance prediction Optimization of data access Monitoring: the application itself, the network use, and the hardware Scheduling: Support for allocation with priority of resources needed for MPI Portals and Roaming Access Web Portal + VNC (Migrating Desktop) Testbed: Support development, test and deployment of applications, tools, and middleware

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 13 Migrating Desktop Multiple Sites Application OCM-G Data Access Portal and Roaming Access Infrastructure Monitoring Scheduling Agent DataGrid Job Management DataGrid Data Management Benchmark Globus Toolkit User Interaction Services Grid Visualization Kernel Tool (Parallel) Application Running Simulation Output Architecture

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 14 The CrossGrid Testbed 16 sites (small & large) in 9 countries, connected through Géant + NReNs + Grid Services: EDG middleware (based on Globus) RB, VO, RC… UCY Nikosia DEMO Athens Auth Thessaloniki CYFRONET Cracow ICM & IPJ Warsaw PSNC Poznan CSIC IFIC Valencia UAB Barcelona CSIC-UC IFCA Santander CSIC RedIris Madrid LIP Lisbon USC Santiago TCD Dublin UvA Amsterdam FZK Karlsruhe II SAS Bratislava Géant

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 15 Using the Testbed Parallel Jobs (HEP Prototype using MPICH-G2) Running Across Sites Grid Services (LIP) Site 1 Site i … network II JSS LB Globus

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 16 Testbed Status

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 17 User Support Software repository Customized GNU Savannah (based on SourceForge ) CVS browsable repository Main current usage: ca web-hits per day 7000 files, 356MB, code-lines, doc- lines doc/pdf-files

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 18 Integration work…

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 19 IST Demonstration CrossGrid has participated in the World Grid demonstration involving European and US sites from CrossGrid, DataGrid, GriPhyN and PPDG, that took place in November It was the largest grid testbed in the world. Applications from the CERN/LHC experiments CMS and Atlas CrossGrid participated with 3 sites: LIP- Lisbon FZK- Karlsruhe IFIC- Valencia

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 20 Extending the GRID in Europe Close collaboration and complementarity with DataGrid Interactive and parallel applications Extending the GRID into new countries and communities Keeping interoperability, in particular for the testbed Outreach and dissemination (visit our booth outside !!!): High impact at the national research level: See Poland, Germany, Spain, Greece examples ACROSSGRID conference in Santiago de Compostela, great success! Dissemination effort to new communities (i.e. SouthEast Europe, Latin America) New application areas start to be interested Reforcing effort via GridStart (concertation meeting in June, 18-19) Starting to establish company and final user contacts: Companies interested in middleware and tools Institutions and companies interested as final users Involved in proposals for new 6th FP: HealthGrid FloodGrid RT Grids…

Jesús Marco de Lucas DataGrid Dissemination Day (Barcelona 14-V-2003) 21 Extending the GRID in Europe …and pushing for a common grid infrastructure for e-Science in Europe: EGEE Keep in contact with us: Thanks in advance for your interest!