1. Earth Observation and Climate Applications for EGEE
IPSL: M. Petitdidier, C. Boonne, O. Marti, M.A. Foujols
IPGP: JP Vilotte, G. Molguiny, DKRZ: J. Biercamp
KNMI: W. Som de Cerff, ESA-ESRIN: L. Fusco

2. Objectives in DataGrid Lessons learned in DataGrid Achievements EGEE
Outline
Objectives in DataGrid
Lessons learned in DataGrid
Achievements
EGEE
Description of the applications
Conclusion

3. Introduction Earth observation: Geoide, atmosphere and Ocean
Interface between them
Data
acquisition, quality control, validation
Studies
space time evolution
processus
parametrization for modelling
Prediction
short term: meteorology, seismic,
long term: climate

4. Objectives in DataGrid
Demonstrate how Grid infrastructure can respond to the complexity and constraints imposed by applications in the EO domain
By deploying EO applications on the EDG middleware (test and evaluate)
Development of EO Grid application interfaces and tools
Dissemination & promotion of this new technology to other scientific teams, satellite operational services and private companies
Participation to non grid oriented workshops, demonstrating Grid
Publication of results obtained on Grid in scientific papers

5. Earth Observation Application in EDG
Use case Ozone profile processing and validation
Common EO problem: measurement validation
Applies to (almost) all instruments and data products, not only GOME, not only ozone profiles
Validation consists of finding, for example, less than 10 profiles out of 28,000 in coincidence with one lidar profile for a given day

6. GOME Processing and Validation Usecase
Raw satellite data
from the GOME instrument
(~75 GB - ~5000 orbits/y)
ESA(IT) – KNMI(NL)
Processing of raw GOME
data to ozone profiles.
2 alternative algorithms
~28000 profiles/day
IPSL(FR)
Validate some of the
GOME ozone profiles (~106/y)
Coincident in space and time
with
Ground-Based measurements
Visualization
& Analyze
DataGrid
environment
Level 2
(example of 1 day total O3)
Level 1

7. Requirements - Achievements
Handling of large number and large volume of files from different satellite instruments (Gome, Gomos) and ground measurements
Production of about 70,000,000 of Ozone profiles on the Grid stored by orbit (2000 profiles/orbit) and a smaller part by individual files. Not all of them are registered.
Processing with complex algorithms (e.g. Neural approach, inversion approach) using IDL(Interactive Data Language)-runtime
Creation and secure access to metadata catalogues and data
Satellite and Lidar metadata catalogues
Access secure and restricted to data and metadata
Interface Grid MW with the already developed operational satellite tools and infrastructure
MUIS catalogue (ESA proprietary product catalogue)
AMS Archive system (ESA proprietary data archive)

8. GOME Use Case file numbers
1 Year of GOME data :
dataset
Number of files
Size
Level 1
4,724
15 Mb
Level 2
NNO (ESA)
20000 kb
Opera (KNMI)
9,448,000
12 kb
Validation
Lidar (IPSL) 12
2.5 Mb
Total:
9,457,460 files
267 Gb
Just one year out of 8 years of data

9. ACHIEVEMENTS- Metadata
Creation and secure access to metadata catalogues and data
EO Virtual Organization (VO)
Virtual Organization Management Service (VOMS) : permits restricted role to subgroups of a VO
Spitfire, EDG module, transparent and secure access to database (e.g. MySQL) used for Ozone profile files (~60000) and Lidar(<300) metadata
Query the database --> relevant logical filename --> EDG RC : Physical filename --> GridFTP for data retrieval
Efficient to query or insert tuples
Drawback: need to create the database

10. Use of Replica Metadata Catalogue for application
Grid metadata service integrated with the EDG RC : EO RMC: 25 attributes (for level1, level2, Lidar metadata), 32,000 tuples inserted as a test case Example: The Metadata Catalogue is searched to locate the intersecting Profiles : User chooses a LIDAR station, and a time interval (start date, end date) The resulting LFNs are then used to build the JDL scripts for the Validation jobs Improvements needed: Better support for Application Metadata (e.g. DateTime type, Polygon) Restricted access for the different partners of one VO

11. INTERFACE WITH OPS TOOLS:
MUIS (Multimission User Information Service) and
AMS (Archive Management System)
Part of present operational
ESA EO environment
Client Applications (JAVA / Linux / Windows)
Web Portal
Client Applications
SOAP
OGC Web Services
MUIS ESA Catalogue
AMS ESA Data Archive
WCS / WFS / WMS
Catalogue (CSS)
GRID Engine
geant
DATAGRID
Computing Elements
Storage Elements
European Grid

12. Reference EO Applications in EGEE
Extend the Ozone experience:
Partners: KNMI, IPSL, SRON, RIVM, Univ. TorVergata, ESA, .....
Goals: Use of “operational Grid” by the Ozone community
To produce and/or store the retrieved Ozone profiles or columns on the Grid
To store on the Grid the Ozone databases, old and new ones
To extend the processing capabilities (assimilation, new parameters, short term prediction, climatology, process studies...), and obtained new scientific results
To facilitate the collaboration and communication among the community
Important for validation, to exchange data in the emerging large scale European projects (GMES)

13. Ozone Validation: Gome
Validation using several Lidar stations in a short time
Comparison of Gome (red line) and Lidar data at OHP
1997 validation
Green line: mean of
[O3(Go)-O3(Li)]/O3(Li)
Red line: rms
Table Mountain
Haute Provence
Mauna Loa

14. Ozone profile retrieval from GOME (1997-2003)
S. Casadio – ESA ESRIN
(GOME 3D Ozone volume over Antarctica - Sept 02,
NNO Level 2 products generate in EDG)

15. Additional Ozone data sets
Maintain/extend data availability on the Grid
all the level1/2 data and reference data (Lidar, Radiosounding)
Extend GOME experience to other datasets
Envisat Sciamachy, Gomos….
Reprocessing with new algorithm and comparison
Generate level 3 data on the Grid
Assimilation of Ozone data into models (UK)
Study of Ozone holes

16. Potential EO Applications in EGEE: seismology
Model and Tomography (IPGP, France)
Simulation: propagation of waves in the Geoide
Numerous data and computations, access to databases
Test started in DataGrid by accessing a cluster
Orpheus Database (in discussion)
General seismology database located in Netherlands
Test to port a subset of the database on the Grid and to access it

17. Potential EO Applications in EGEE: Climate
Model: Atmosphere, Ocean, Hydrology, Atmospheric and Marine chemistry….
European effort: PRISM, ENES, GEMS
Goal: Post treatment: Comparison of model outputs from different runs and/or institutes
Characteristics of model outputs :
Large volume of data (TB) from different model outputs, and experimental data
Run made on vector computer
Post-treatments: visualisation, statistics, comparison….

18. Climate : Strategy
Secure access to the mass storages where the data are stored (like the ESA portal) (prototype by IPSL+CEA is starting soon)
Difficulty: authorization to implement this access
Metadata Catalogues implemented on the GRID (Spitfire or RMC)
metadata catalogues : one or several?
access to models and data
Demonstration between two institutes (IPSL, DKRZ)
extend to the other institutes
Extension to other European laboratories linked via European projects(PRISM, ENES, GEMS) to develop coupled model, and to compare them.

19. Possible other extensions
In the Space community (scientific and operational), ESA has played an important role to facilitate the access to satellite data via the Grid, to convince the international and operational community of the interest of Grid and to develop some tools to make easier the use of satellite data via the Grid.
Other Satellite data: Envisat ASAR, MERIS, …
build on ongoing ESA funded projects
Integration of science tools and applications
following community interests

20. Material Means Starting point: Node at ESA: cluster, SE, UI
Node at IPSL+IPGP at Paris University Computer Center : 6PC, SE(500Gb), UI
Other at KNMI: 4 PC

21. Requirements Continuation of EO Virtual Organization + VOMS
New EO Applications proposed in EGEE
NIKHEF and KNMI to maintain the EO-VO
ISPL/CNRS could lead participation to EGEE
ESA-ESRIN willness to provide data and infrastructure
Improvements:
data access (restriction)
the replica metadata catalogue
Important questions addressed by the owner of large datasets :
need on the grid secure, controlled and restricted accesses
Porting and query of their metadata catalogue (compatibility)
Update and mirroring

22. Man power effort needed to help this deployment for EO
CONCLUSION
Earth Observation is ready to port new applications today due to the dissemination work towards our colleagues during DataGrid Project
Deployment of EO observations on the Grid will be driven by goals to be attained in the EU projects, Grid is not explicitly mentioned then no funded.
Interaction with the CrossGrid partners about meteorology, pollution, flood prediction…then possibility of new projects
CEOS (Committee of Earth Observation by Satellite) Meeting on June 22 and 23 in London with an invitation to EU representative
Man power effort needed to help this deployment for EO