1. EO GRID Processing on Demand ESA grid activity report on 2007 luigi.fusco, guoqing li ESRIN, European Space Agency (ESA) Presented for WGISS-23, 2007-5-23

2. Summary Brief introduction to ESA EO satellites EO community and GRID requirements EO GRID Processing on Demand (G-POD) –Operational experience –Future vision and plans

3. ENVISAT product’s examples

4. The International Charter on Space and Major Disasters  Initiated by CNES and ESA, joined by CSA, NOAA, ISRO, CONAE, JAXA, USGS and DMC  Unified system of space data acquisition & delivery in case of natural or human-made disasters  Data delivery to civil protection agencies, emergency & rescue services; UN cooperating body since 2003  Operational since 2000: 24 hrs on-duty-operator, data resources from all Charter members  Recent Examples of Charter Activation:  Bam Earthquake 2003  Darfur Crisis 2004  Tsunami Catastrophe 2004/2005  Hurricane Katrina 2005  Cyclone Kyrill 2007 Key international EO programmes

5. GLOBAL MONITORING FOR ENVIRONMENT AND SECURITY (GMES) European independence in critical data sources for environmental monitoring and security and the European contribution to the Global Earth Observation System of Systems (GEOSS) Key international EO programmes

6. The Earth Science world Atmosphere Ocean Biosphere Cryosphere Noisy observations Optimal Trajectory First guess time Field Coupled and inter- disciplinary processes Complex web of sensor Complex data analysis

7. Some Earth Science digital data requirements Global, regional, local applications –Alternative use of the data at different resolution Large historical distributed archives –Long term data and knowledge preservation issues Near real-time access to data –For processing, value adding and dissemination Integration with models to provide long term trends and forecast –Data assimilation Integrate different data sources –Standardisation, Virtual Organisation, … Need to link data to technical information and scientific results –Need to keep/transfer the historical Knowledge

8. Why GRID at ESA for EO ? Operational issues –Envisat generate > 500GBytes of data per day … –EO archives are “ scattered ” –Algorithm evolutions require recurrent reprocessing –More efficient use of existing infrastructures Scientists / user –Need near-real-time access, historical and multi-sources –have the processing algorithms –require large investments to handle the data on their site GRID can solve the equations … –move processors close to the data –reduce dissemination costs and effort –evolutions benefit to all at once GRID as a common shared platform for collaborations

9. EO GRID Processing on Demand (G-POD) Objectives Provide a “user-segment” environment –Put data & processors together –Allow “on-demand” processing of the data Offer scientists a “production” lab –Focus on algorithms –Reuse housekeeping functions (e.g. catalogue, software tools) –Bridge gap from “prototype” to “production” processor Offer scientists a “collaboration” environment –Share tools and functions –reuse output of other processors –IPR is kept by the scientist Open and scalable –host “any” processor –bring more resources (other grids) and data (other archives) together

10. The ESA EO GRID environment Security User certification via ESACERT underway Software resources on-line –IDL, Matlab, BEAT, BEAM, BEST, CQFD, Compilers, public domain image processing utilities –Catalogue queries and data provision functions –Data viewers Computing and Storage Elements –Over 200 Working Nodes –Over 100 TB on-line store –Middleware: LCG 2.6, GLOBUS 4, gLite3 –Link to external CE and SE (e.g. campus, EGEE…) Data Interfaces –GS products Rolling Archives (ENVISAT, MSG) –MODIS NRT products over Europe –Access to AMS SatStore –Some accesses to NASA and other external data providers Network –Gbit LAN –64-192 Mbps to GARR –HiSEEN WAN (e.g. to PACs/PDHS) –MEGALAB (soon 20-40 Gbps in Frascati area)

11. Grid on Demand Infrastructure I/F to external GRID resources via research networks ESA GS Operat’s User access tools Only one environment for operations and new dev

12. EO Data on-line in GRID storage ASAR –ASAR GM: Feb 2004 onwards –ASAR Medium Resolution: June 2005 onwards –Occasional High Resolution products MERIS –Level 1,2 RR: full mission –Samples of FR Level1 AATSR -Level 1: full mission -Level 2: since mid 2005 Atmospheric –GOME Level 1,2: 1996 onwards –GOMOS, MIPAS, SCIAMACHY full archives from D-PAC Third-party data –MSG SEVIRI: last 12 months –MODIS NRT over Europe More to come …

13. Web Portal Interface Temporal/spatial selection of products Job definition, submission and live status monitoring Customisable result visualization interfaces Access to output products and documentation

14. Experience so far Wide ranging applications supported –Systematic productions of high-level products (e.g. level-3) –On-demand Data access and visualisation –Algorithm and data validation activities software test-beds (ESA toolboxes e.g. BEAT, BEAM, BEST) Processor test-beds –Internal Research projects Over 40 projects supported, more to come!

15. Routine Production MERIS Level-3 Products NRT generation –Joint ESA collaboration with ACRI (France), JRC/Ispra (European Commission) and Brockmann Consult (BEAM) –11 monthly products published on-line http://earth.esa.int/meris/level3 Daily ASAR GM mapping of Antarctica –Internal Development in operations since 2005 –Daily Generation of 400-m resolution mosaics and publish to ESA Web Map Server http://ssems1.esrin.esa.int:8080/map/context.php EarthCare Simulator –Collaboration with Estec and Univ. of Valencia (Spain) –Preparatory activity of the mission EarthCARE –2.5 10 5 simulations carried out using ESA and CNR GRIDs River and Lake Processor –ESA/Montfort University (UK) collaboration –Accurate River and Lake heights measurements in NRT from satellite altimetry (RA2) –products published online http://earth.esa.int/riverandlake Meris True-Color Mosaics –9km resolution global Monthly mosaics of MERIS data

16. On demand processing ASAR on Demand –Integrated environment for SAR processing –Binds separate functionality into higher- level applications (flood monitoring, co- registration, mosaicing, etc) Imager –Multi-mission imaging tools –Data selection, user-defined processing, image rendering, geo-coding, visualisation MIRAVI Geo-toolbox Geocoding of MERIS full resolution images produced by MIRAVI real-time service Aeromeris –Fast extraction over user-area of pixels and statistics from the complete MERIS level-2 product archive –Output to Excel, Google Earth, XML

17. Algorithm Validation Volcanoes Monitoring by Infrared –Fast Extraction of AATSR thermal anomalies over > 200 volcanoes –Under validation, Operational service planned for early 2007 GRIMI-2 –EADS/Astrium MIPAS Level-2 reference processor used to validate the operational PDS processor –Test data generation: 300 profiles (100GB) processed in 1hour YAGOP –Alternative GOMOS O3 and Temp level-2 processor (internal research) –Validation against operational ESA products RAIES Processor –Assessment of the 2000 Hz individual echoes of Envisat/RA2

18. New Research applications MERIS/GOME inter-comparison –Validation of Envisat/MERIS cloud fraction and water vapour measurements against ERS-2 GOME Var2Grid –ESA/DMI cooperation (DK) –3DVar data assimilation (T and specific humid.) with CHAMP/GPS products (Background ECMWF ERA40) SCIA-SODIUM –Internal prototype development –Extraction of Sodium profiles in mesosphere from Sciamachy –Collaboration with ESO for the next generation telescope ground site survey ALOS PRISM DEM Prototype Processor –Internal Research (ESA Trainee) –Image co-registration of PRISM views and height reconstruction by image matching between PRISM triplet images Mountains around Shizuoka, Japan

19. Earth Science G-POD Call for Proposals http://eopi.esa.int/G-POD Promote the access and use of EO mission data available at ESA, offering on-line access to products with attached computing infrastructure and tools to assist the generation of “ scientific added value products ” –new earth science algorithms –Build network of competences; promote e-collaboration –transition from science algorithms to new user products … First call published beginning June 2006 on EOPI web 11 Proposals selected in first call, implementation ongoing First very promising results expected mid 2007 Next call in few months

20. Validation period: 11 day ensemble of GOMOS data (Sep., Oct., Dec. 2002) total set of selected occ. events GOMOS: 1719 MIPAS: 4099 ECMWF: 1719 GOMOS to MIPAS temperature profile comparison (relative errors shown in Fig.) Time – Space coincidences criteria for analysis: 3h&300Km YAGOP: Yet Another GOMOS Processor GRID is very good for such analysis: 1 generic grid service

21. near future GRID plans Operational Opportunities –Routine generation of “standard” high level products –Extend infrastructure to other facilities and across establishments Consider potential use of GRID –In future EO missions preparation –GMES Service Elements –Contribute to GEOSS architecture –As a mean to get the science community more involved in ESA programmes Earth Science presence in European e- infrastructure initiatives –Strong participation to new EC FP7 initiatives such as e- infrastructure, digital repositories, knowledge preservation, e-content... Knowledge Infrastructures

22. GRID based RTD plans Grid ES Community Applications DL Applications … … as Knowledge Infrastructure PECS GRID ES Grid on Demand in operation @ ESA CWE Applications

23. Thank you! The ESA EO GRID Team More information at http://eogrid.esrin.esa.int