EDG Applications The European DataGrid Project Team

EU DataGrid - Applications 2 EDG Application Areas High Energy Physics Biomedical Applications Earth Observation Science Applications

EU DataGrid - Applications 3 High Energy Physics 4 Experiments on LHC CMS ATLAS LHCb ~6-8 PetaBytes / year ~10 8 events/year ~10 3 batch and interactive users

EU DataGrid - Applications 4 Europe: 267 institutes, 4603 users Elsewhere: 208 institutes, 1632 users CERN’s Network in the World

EU DataGrid - Applications 5 Data Flow in LHC

EU DataGrid - Applications 6 Example: CMS Monte Carlo Production

EU DataGrid - Applications 7 CMS jobs description CMKIN : MC Generation of the proton- proton interaction for a physics channel (dataset) CMSIM: Detailed simulation of the CMS detector, processing the data produced during the CMKIN step CMKIN Job CMSIM Job Output data Output data Grid Storage Write to Grid Storage Element Write to Grid Storage Element Read from Grid Storage Element * PIII 1GHz 512MB  46.8 SI95 size/eventtime * /event CMKIN ~ 0.05MB~ sec CMSIM ~ 1.8 MB ~ 6 min

EU DataGrid - Applications 8 CMS production components interfaced to EDG middleware u Production is managed from the EDG User Interface with IMPALA/BOSS u CMS Virtual Organization server at NIKHEF (Amsterdam) CMSEDG SE CE CMS software BOSS DB Workload Management System JDL RefDB parameters Push data or info Pull info UI IMPALA/BOSS CE CMS software CE CMS software CE SE

EU DataGrid - Applications 9 CMSEDG SE CE CMS software BOSS DB Workload Management System JDL RefDB parameters data registratio n input data location Push data or info Pull info UI IMPALA/BOSS Replica Manager CE CMS software CE CMS software CE WN SE CE CMS software SE X CMS production components interfaced to EDG middleware u CMKIN jobs running on all EDG Testbed sites with CMS software installed u CMSIM jobs running on CE close to the input data u produced data: scripts for batch replication to a dedicated SE

EU DataGrid - Applications 10 CMS production components interfaced to EDG middleware u Job monitoring and bookkeeping: BOSS DBs, EDG Logging & Bookkeeping service CMSEDG SE CE CMS software BOSS DB Workload Management System JDL RefDB parameters data registratio n Job output filtering Runtime monitoring input data location Push data or info Pull info UI IMPALA/BOSS Replica Manager CE CMS software CE CMS software CE WN SE CE CMS software SE

EU DataGrid - Applications 11 CMS use of the system (Statistics) CEsSEs Nb. of evts time Events Production within EDG is part of the Official CMS production production/www/html/general/

EU DataGrid - Applications 12 Summary of CMS work and plans for use of EDG middleware u RESULTS n We can distribute and run CMS s/w in the EDG environment n We have generated ~250K events for physics with ~10000 jobs in 3 week period u OBSERVATIONS and PLANNING for the future n We were able to quickly add new sites to provide extra resources n There was a fast turnaround in bug fixing and installing new software n The stress test was labor intensive (since software was developing) n Release EDG 2.0 should fix the major problems and allow for enhanced scalability,and we look forward to evaluating it and using it in our Data Challenge work

EU DataGrid - Applications 13 ESA(IT) – KNMI(NL) Processing of raw GOME data to ozone profiles. 2 alternative algorithms ~28000 profiles/day IPSL(FR) Validate some of the GOME ozone profiles (~10 6 /y) Coincident in space and time with Ground-Based measurements Visualization & Analyze LIDAR data (7 stations, 2.5MB per month) DataGrid environment Level 2 (example of 1 day total O 3 ) Level 1 Raw satellite data from the GOME instrument (~75 GB - ~5000 orbits/y) EDG EO challenge: Processing / validation of 1y of GOME data

EU DataGrid - Applications 14 EO WebMap Portal

EU DataGrid - Applications 15 Web Portal EO Product Catalogue EDG Storage Element EDG User Interface EDG Resource Broker EDG Computing Element EO Replica Catalogue EO Grid Engine EO Product Archive 1. Search Level-1 catalogue 2. Retrieve Level-2 products 3. Level-2 Products already registered in RC? 8. Submit jobs to process Level-1 data 7. Register Level-1 data 11. Register level-2 data 9. Process Level-1 data 10. Transfer Level-2 data to SE 12. Return new Level-2 products Yes? 4. Return available Level-2 products No? 5. Perform GRID processing on-the- fly 6. Transfer Level-1 data from Archive to the Grid Processing Sequence

EU DataGrid - Applications 16 Goals of the DataGrid application validate satellite data with all ground based data available in an easy way:  Comparison of ozone profiles provided by satellite with lidar data in different locations and times (see the web portal)  Statistical comparison and analysis in order to improve algorithms. OZONE LAYER 50 km 10 km ERS/GOME satellite Lidar at the Haute Provence Observatory GOME Ozone Profile Validation

EU DataGrid - Applications 17 Level 2 Catalogue Lidar data catalogue Queries and data information retrieval from the Lidar metadata catalogue GRID Computing Element Storage Elements with Lidar data Queries and data information retrieval from the Gome Level 2 orbit or pixel metadata catalogues When completed comparison between lidar and satellite ozone profiles Satellite data validation Lidar site Level 2 Catalogue GRID Portal Storage Elements with Gome L2 data Submission of the Job in the GRID Validation Processing Sequence

EU DataGrid - Applications 18 Validation Output Figure 1: Estimation of the bias between Gome and Lidar using one month of data. Figure 2 : example of 2 profiles : Comparison between Gome profile and lidar profile for the 2nd October 2000.

EU DataGrid - Applications 19 Perspectives for Biomedical Applications u Grids open new perspectives in large scale genomics analysis n Complete genome annotation n Cross-genomes analysis n Data mining on distributed databases n Pipelining of huge automatic bio-informatics analysis u Medical image processing n Large databases processing n Anatomy and physiology modeling n Epidemiological studies

EU DataGrid - Applications 20 Biomedical Applications u Bio-informatics n Phylogenetics : BBE Lyon (T. Sylvestre) n Search for primers : Centrale Paris (K. Kurata) n Statistical genetics : CNG Evry (N. Margetic) n Bio-informatics web portal : IBCP (C. Blanchet) n Parasitology : LBP Clermont, Univ B. Pascal (N. Jacq) n Data-mining on DNA chips : Karolinska (R. Médina, R. Martinez) n Geometrical protein comparison : Univ. Padova (C. Ferrari) u Medical imaging n MR image simulation : CREATIS (H. Benoit-Cattin) n Medical data and metadata management : CREATIS (J. Montagnat) n Mammographies analysis ERIC/Lyon 2 (S. Miguet, T. Tweed) n Simulation platform for PET/SPECT based on Geant4 : GATE collaboration (L. Maigne) Applications deployed Applications tested on EDG Applications under preparation

EU DataGrid - Applications 21 Medical Imaging Medical images Metadata H 1. query 2. visualisation 3. similarity search 4. scores 5. best results visualisation LFN image patient hospital...

EU DataGrid - Applications 22 Graphic layer Job Monitoring Grid File Browsing File registration and retrieval

EU DataGrid - Applications 23 Graphical Interfaces Image registration Image retrieval Local filesGrid files Metadata Query over metadata Query result

EU DataGrid - Applications 24 Image Registration LFN image patient hospital... Imager SE

EU DataGrid - Applications 25 Similarity search Similarity computation Results visualization Job monitoring Ranked list of images Source image Most similar images Low score images

EU DataGrid - Applications 26 Future: Interfacing medical data with the Grid Client 1 interface Client 2 interface RS interface core grid - server interface header blanking encryption Storage Element Replica Catalog Replication Service RC interface Metadata interface Medical (trusted) site Grid middleware File metadata ACL size checksum... Application metadata ACL encryption key sensitive metadata... Medical server Storage Element MSS Master File Replica Imager

EU DataGrid - Applications 27 Parallel Processing Magnetic Resonance Images simulation using the grid 3 levels of parallelism: Parallel isochromat computations Multi-slice MRI computation Parallel magnetization kernel Magnetisation computation kernel Reconstruction algorithm MRI Image Virtual object MRI sequence

EU DataGrid - Applications 28 Summary u Use Cases n High Energy Physics n Earth Observation n Biomedical Applications

EU DataGrid - Applications 29 Further Information u High Energy Physics u Bio-Informatics u Earth Observation

EU DataGrid - Applications 30