1. ESA Data Integration Application Open Grid Services for Earth Observation Luigi Fusco, Pedro Gonçalves

2. Topics Earth observation requirements Ozone application Generic application layer (GridEngine) Grid portal for EO services

3. Requirements 100 Gbytes of data per day in ERS missions and 500 Gbytes in ENVISAT Grid Requirements : –enhance the ability to access high level products –allow reprocessing of large historical archives –improve Earth science complex applications (data fusion, data mining, modeling …)

4. Ozone Application Input: Wave spectra data measured by the GOME instrument on the ERS (level 1) Output: Calculation of satellite ozone profiles (level 2 data) Two algorithms: OPERA (MPI) and NOPREGO (Neural Networks) Objective: Data validation using LIDAR measurements Collaboration among different institutes: France (IPSL), Italy (ESA, ENEA, UTV), Holland (KNMI)

5. Ozone Application Requirements Reuse of Legacy data warehouses and catalogues from ESA Constraints: 1.Access to DataGRID storage elements and execution of remote jobs within DataGRID GLOBUS 2.2.4 2.CEOS-GRID GLOBUS 2.4.2 Different certificate authorities Web portal interface

6. Ozone Application Portal Temporal and spatial selection of data Catalogue access and data transfer from ESA data warehouses to the GRID storage elements Job selection and status information Result retrieval and visualization in OWS Remote MySQL access (SOAP) Data validation w/ ground measurements Demonstration of prototype at last WGISS meeting in Toulouse

7. Lessons learned … Joining the services Client Applications (JAVA / Linux / Windows) Web Portal Client Applications SOAP Wrapper OGC Web Services MUIS ESA Catalogue AMS ESA Data Archive WCS / WFS / WMS Catalogue (CSS) GRID Engine DATAGRID Computing Elements Storage Elements GLOBUS 2.2 Computing Elements Storage Elements … other versions Computing Elements Storage Elements

8. GRID Engine Status : First version for DataGrid and GLOBUS 2.2 Connects and manages jobs sessions in multiple GRID environments - (WSDL description and SOAP encoding) Discovery Services Publish jobs availability and current processing status to external services in XML Retrieves and translates job results GRID Engine DATAGRID Computing Elements Storage Elements GLOBUS 2.2 Computing Elements Storage Elements GLOBUS 3 Computing Elements Storage Elements

9. GRID Engine Services are pre-defined applications managed by templates for each GRID that are plugged into the Grid Engine Templates instantiate the necessary scripts for the specific job requests (files names, parameters) Defined by XML schema for future implementation of discovery services

10. GRID Engine - Operations SOAP Services using HTTP(S) Session Manager (SM) –Session and Certificates management –SM_open, SM_authenticate, SM_quit, … Job Manager (JM) –Job commands and status information –JM_submit, JM_list, JM_create, … Storage Explorer (SE) –Storage access –SE_query, SE_move and SE_transfer methods. TBD : the use of SOAP w/ Attachments (W3C sub.)

11. Example of EO specialized Services On-demand geographical orbit projection (using ESAs EnviProj) On-demand orbit projection and mosaicking (using OSSIM) Format Conversion Services: Envisat standard, GeoTiff, HDF-EOS … (using geoTiff, OSSIM and with the future use EnviView libraries) GOMOS/GOME level 1 processing and validation (using ESA data processors)

12. EO data Search and Storage Services Temporal and spatial selection of data Data transfer from data warehouses (AMS) Data transfer to and from the GRID storage elements

13. Auxiliary Grid Services Job selection, launching and status information Result retrieval and visualization in WMS Join personal data folder as a WMS

14. Conclusion The Web Portal Integrates several components of the EO Infrastructure The EO GRID Engine provides a standard way for Web Clients to request GRID operations The Web Portal and the EO Grid Engine allow the complexity of the underlying components to be hidden from the user This Framework can be adapted to provide new tools for the satellite end-users (e.g. integration of ENVISAT specific tools, such as BEAT, BEST, BEAM …)

15. More information luigi.fusco@esa.int http://giserver.esrin.esa.int/