DataGrid is a project funded by the European Commission under contract IST Earth Observation applications in EU DataGrid (EDG) and follow on Luigi Fusco Presented by PierGiorgio Marchetti

CEOS May 04 - n° 2 Outline  EO and DataGrid Objectives and Achievements  Lessons learned  Future activities

CEOS May 04 - n° 3 EO Objectives in DataGrid  Demonstrate how Grid infrastructure can respond to complexity / constraints imposed by EO applications àDevelopment of EO Grid application interfaces and tools àDeployment of EO applications on the EDG framework  Dissemination & promotion of this new technology à To EO/space scientific teams, services providers and operational people àParticipation to outreach activities àPublication of results obtained on Grid in scientific papers

CEOS May 04 - n° 4 Main EO requirements for EDG  Handling of large number and large volume of files from different satellite instruments and ground based measurements àProcessing with complex algorithms (e.g. Neural Network approach, Inversion approach) àCreation and secure access to distributed metadata catalogues and data (e.g. Lidar and EO) àInterface grid MW with the already developed operational satellite tools and infrastructure (MUIS, ESA)  Tested using Ozone data processing and validation use-cases

CEOS May 04 - n° 5  EO Virtual Organisation generation and management (EO-VOMS)  Job management à1000’s of job submissions - single jobs and simultaneous ‘job storm’ tests; different job types - short/long duration, small/large datasets, with/out replica optimisation, MPI, etc.  Replica & Data management àMany thousands entries registered in Replica Catalogue; over 35,000 data files transferred to EDG SEs (including migration to new versions) àTransparent and Secure access to database (metadata) via SPITFIRE  Information System àBoth MDS/LDAP and R-GMA used for application grid interfacing  Fabric management àLGFG, LCFG-NG fabric management tools used for EO site installations  Network management àEDG Network monitoring tools installed & used at EO sites  Security àVOMS tested for EO security use case Achievements: Evaluation of EDG Testbeds

CEOS May 04 - n° 6 Achievements: EDG EO applications  GOME L2 NNO processing (TV-ENEA-ESA) à7 years GOME (30k orbits, 500GB) processed in EDG and in local grid; àIntegration of IDL (cots) licenses.  GOME processing OPERA (KNMI) à7 years of GOME data loaded to the Grid, Use of application metadata DB (Spitfire and RMC) under Grid for large number of files  GOME validation (IPSL) and validation portal (ESA) – 2 versions à7 years from 7 sites LIDAR profiles (NDSC), use of application metadata (Spitfire and RMC). Validation of Opera and NNO  GOMOS reprocessing and validation (ESA-IPSL) à5 months data in ESA local grid. 40k files, 120GB  GREASE: OMI simulation (Dutch Space-ESA) à1 month OMI instrument simulation (including development of Workflow Management System in GRID environment)  CEOS-GRID (ESA-NASA-DS) - ongoing

CEOS May 04 - n° 7 Achievements : EO VO and EO infrastructure  Initial EO VO: ESA-ESRIN, KNMI, IPSL  Present EO VO: extended to research, space related industry, International Space Agencies community àAt present some 25+ people, wish to extend it  The present EO dedicated infrastructure (CE and SE): ESA-ESRIN, IPSL, connection to ENEA proprietary GRID infrastructure  Upgrading infrastructure plans: àextend it to ESA-ESTEC, Dutch Space, CNR, CEOS-NASA, new ESRIN CE+SE àMature elements considered for “operational” deployment

CEOS May 04 - n° 8 ESA(IT) – KNMI(NL) Processing of raw GOME data to ozone profiles. 2 alternative algorithms ~28000 profiles/day IPSL(FR) Validate some of the GOME ozone profiles (~10 6 /y) Coincident in space and time with Ground-Based measurements Visualization & Analyze DataGrid environment Level 2 (example of 1 day total O 3 ) Level 1 Raw satellite data from the GOME instrument (~75 GB - ~5000 orbits/y) The EDG Ozone processing and validation test-case

CEOS May 04 - n° 9 Mauna Loa Table Mountain Haute Provence 1997 validation Green line: mean of [O3(Go)-O3(Li)]/O3(Li) Red line: rms Comparison of Gome (red line) and Lidar data at OHP Validation using several Lidar stations in a short time The latest GOME user results

CEOS May 04 - n° 10 The latest GOME user results S. Casadio – ESA ESRIN (GOME 3D Ozone volume over Antarctica - Sept 02, NNO Level 2 products generate in EDG)

CEOS May 04 - n° 11 Part of near future operational ESA EO environment …other versions? Computing Elements Storage Elements ESA EO GRID Part of present operational ESA EO environment Achievement: “GRID on Demand” Client Applications (JAVA / Linux / Windows) Web Portal Client Applications SOAP OGC Web Services MUIS ESA Catalogue AMS ESA Data Archive WCS / WFS / WMS Catalogue (CSS) GRID Engine geant GLOBUS 2.2 Computing Elements Storage Elements Local Grid DATAGRID Computing Elements Storage Elements European Grid ENVISAT MERIS 10 days mosaic

CEOS May 04 - n° 12 Lessons Learned  Substantial first-hand experience and understanding of Grid technology (as users) à Grid middleware encompassing wide range of standards à Technology now being used for real world applications  Impact of Grid technology on EO community and applications à unprecedented experience for many of the communities involved àNew approach to scientific collaboration, communication and sharing among participants with distinct backgrounds  Testbed evaluation à Still need time, experience and major effort to convince all potential operational users

CEOS May 04 - n° 13 Future & Exploitation: EO community commitments  Continuation of EO Virtual Organization à New EO Applications proposed in EGEE new domains – seismology, oceanography, climate … new instruments / data – SCIAMACHY, MERIS … Modelling - assimilation, application studies, … à NIKHEF and KNMI to maintain the enlarged EO-VO à ISPL/CNRS leads participation to EGEE à ESA-ESRIN and other centres to provide data and infrastructure  New EC project: DILIGENT – Digital Libraries over GRID à integration of EO data with environmental information  Continue widespread dissemination and promotion of Grid solutions in the EO community à Involve new users and applications in EO community à Continue to establish pre-operational Grid services

CEOS May 04 - n° 14 Which operational services? 1.Support to science users Support science communities for focused collaborations, e.g. cal/val, global products, new algorithms 2.Support to application projects àProvide reference application processing environment for generation of products àGeneration of periodic global and regional products for immediate availability at ESRIN 3.Support to Specific Reprocessing àAllow multiple re-processing of same dataset àConsider long term evolution of EO ground segment

CEOS May 04 - n° 15 EO GRID ESA-ESRIN Preparation of “operational environment” Integration of compatible and available ENVISAT software processing tools Upgrading of Research Network Bandwidth, CE and SE Preparation of ESA internal GRID technology plan Consider internal and science operational requirements for coming 3-4 years New short term RTD projects “The VOICE” – e-collaboration environment –funded by ESA (3+ new applications)

CEOS May 04 - n° 16 Thank you!