1. 6-7 May 2002 CEOS GRID Workshop 1 European DataGRID for EO luigi.fusco@esa.int - julian.linford@esa.int ESRIN, 6-7 May 2002 CEOS Workshop on GRID

2. 6-7 May 2002 CEOS GRID Workshop 2 Summary EO applications and GRID requirements ESA EO participation to European GRID projects – DataGrid Ideas for CEOS

3. 6-7 May 2002 CEOS GRID Workshop 3 Earth Observation Community GRID interactive scenario Common access to EO missions catalogues Acquisition plan, order, delivery Parametric data fusion and models integration Collaborative publishing of results On demand high level products generation

4. 6-7 May 2002 CEOS GRID Workshop 4 EO and Networking Computing – which data models ? Distributed Computing –Integration of data from various instruments and missions High-Throughput Computing –Interferometry … On-Demand Computing –Generation of EO user products… Data-Intensive Computing –Archive data re-processing, climate modeling… Collaborative Computing –Scientists application interactions, Instrument cal/val … Ian Foster and Carl Kesselman, editors, “The Grid: Blueprint for a New Computing Infrastructure,” Morgan Kaufmann, 1999

5. 6-7 May 2002 CEOS GRID Workshop 5 High demanding computing Pomona (Cal): subsidence velocity fields 40 ERS1/2 images (92-99), Ambiguity: 28 mm Digital Elevation Model GRID requirements: large data files (10+ GB) stages with intensive processing science driven value adding

6. 6-7 May 2002 CEOS GRID Workshop 6 Science collaborative environment : El Niño November 1997: El NiñoJanuary 1999: La Niña SST anomaly

7. 6-7 May 2002 CEOS GRID Workshop 7 Global fire atlas - ATSR: 1997

8. 6-7 May 2002 CEOS GRID Workshop 8 Global fire atlas - ATSR: 1998

9. 6-7 May 2002 CEOS GRID Workshop 9 Provide a single access point to space systems to emergency & rescue organisations in case of disasters Participating Space Agencies: CNES, CSA, ESA, ISRO, NOAA, … Missions: RADARSAT; ERS, (Envisat); SPOT; IRS; NOAA, …

10. 6-7 May 2002 CEOS GRID Workshop 10 ESRIN MATERA (I) NEUSTREL.ITZ (D) KIRUNA (S) - ESRANGE Earthnet Facilities real time Infrastructure MASPALOMAS (E) TROMSO (N) MATERA (I) METADATA BROWSE WEB SEAWIFS SPOT IRS-P3 LANDSAT 7 TERRA/MODIS STANDARD PRODUCTION CHAINS MULTIMISSION DATABASES FOR REMOTE ACCESS AND USER SERVICES USERS HISTORICAL ARCHIVES USERS PRODUCTS AVHRR

11. 6-7 May 2002 CEOS GRID Workshop 11 ENVISAT FACILITIES ORGANISATION qDecentralised architecture, central co-ordination and supervision. qNational facilities put at ESA’s disposal via MOUs and contracts. qDirect dealing with scientific users (outside ESA operational remit) qCo-operation with value added industry in E.O. promotion and in technology transfer from research to applications. ESRIN I-PAC D-PAC F-PAC UK-PAC E-PAC LRAC/S-PAC F-PAF FIN Co-PAC ESOC

12. 6-7 May 2002 CEOS GRID Workshop 12 No Projects 1-25 Projects 26-50 51-100 100+ Projects Countries P.I. geographic distribution AOs: Stimulating scientific research world-wide 3500+ science Users of ESA data 120 New Cat-1 Projects in 2001 700 Envisat AOs to start in 2002 Stimulating new researchs

13. 6-7 May 2002 CEOS GRID Workshop 13 Why GRID in EO? (1) EO Community: Progressive refinement of data from many sourcess to produce higher quality products Product generation chain involving distributed organisations and users Collaborative: distributed users and data – large international cooperation Discovery: large numbers of products & resources Interoperabiltiy of catalogue and metadata already in operation Web based data services

14. 6-7 May 2002 CEOS GRID Workshop 14 Why GRID in EO? (2) Massive, non-stop data volumes New instruments, sensors & product types Distributed archives Historical dataset reprocessing Complex numerical processing algorithms Near real-time turnover

15. 6-7 May 2002 CEOS GRID Workshop 15 The Grid from a Services View : : E.g., Resource-specific implementations of basic services E.g., Transport protocols, name servers, differentiated services, CPU schedulers, public key infrastructure, site accounting, directory service, OS bypass Resource-independent and application-independent services authentication, authorization, resource location, resource allocation, events, accounting, remote data access, information, policy, fault detection Distributed Computing Toolkit Grid Fabric (Resources) Grid Services Application Toolkits Data- Intensive Applications Toolkit Collaborative Applications Toolkit Remote Visualization Applications Toolkit Problem Solving Applications Toolkit Remote Instrumentation Applications Toolkit Applications Space Science S/C modelling Cosmology Space weather Environment... EO Environment GRID Middleware

16. 6-7 May 2002 CEOS GRID Workshop 16 Needed GRID technologies Resource-independent and application- independent services (middleware) –authentication, authorization, resource location, resource allocation, remote data access, –accounting, security, quality of services, fault detection, real time services, … Specialized protocols, procedures, data standards, operational environments, interfaces to EO legacy systems… EO dedicated portal and user access…

17. 6-7 May 2002 CEOS GRID Workshop 17 Participation to GRID initiatives

18. 6-7 May 2002 CEOS GRID Workshop 18 Participation in European GRID projects EU funded DataGRID – Earth Observation application EGSO – Solar radiance DataTAG – access to Trans Atlantic Connectivity … ESA funded SpaceGRID – vision of GRID systems for space ESA internal GRID initiative …

19. 6-7 May 2002 CEOS GRID Workshop 19 DataGrid EO application objectives  Specification of EO requirements  Bringing Grid-aware application concepts into the Earth Science environment  Adaptation of existing systems and selected EO applications to use the DataGrid infrastructure  Testbed validation through prototyping activity  Activities handled in coordination and synchronisation with other related and relevant work packages  Key partners: ESA-ESRIN, KNMI (NL), IPSL (F)  Associated partners: ENEA (I), BADC (UK)

20. 6-7 May 2002 CEOS GRID Workshop 20 GOME Instrument (1 day coverage) GOME’s Ground track

21. 6-7 May 2002 CEOS GRID Workshop 21 Application of DataGrid in EO One Use Case being studied in detail (GOME) Develop generic components Feedback to DataGrid developers and Architecture Group Re-use components to add new applications Testing in “controlled” GRID environment (ESRIN- ENEA) and in “wide-European” environment

22. 6-7 May 2002 CEOS GRID Workshop 22 Why Grid in EO? L2VAL L1L2RAWL1 ESAESA / KNMI IPSL End User L2 L3 Science Application VAL + An Example: GOME Use Case Process 1 Year of data Regulated Access to Grid processing power Secure access to Grid-registered high-volume data storage L1 L2 4724 files = 66 Gb 9,448,000 files = 108 Gb

23. 6-7 May 2002 CEOS GRID Workshop 23 Architecture Group DataGrid Overview (1/5) Middleware Developers WP1 WP7 WP3 WP2 WP4 WP5 Middleware Packages Integration Testing Integration WP6 Information Index Resource Broker User Interface Computing Element Storage Element Information & Monitoring Replica Management Installation Management Network Monitoring CVS Repository Certificate Authorities EDG Rules EDG Membership Registration Documentation Applications WP8-9-10 Requirements Evaluation & Prototyping SE Site H SE Site G SE Site F SE Site E SE Site D SE Site C SE Site B Site A Installation Management Sites Installation 1. Organization Application Environments

24. 6-7 May 2002 CEOS GRID Workshop 24 DataGrid Overview (2/5) VO LDAP Server Certificate Authorities Users 1. Obtain certificate 2. Join VO 3. Sites subscribe to one or more VOs 4. Publish details Information Index Grid Resource Broker User Interface Search 5. Submit Jobs CESE Site H CESE Site G CESE Site F CESE Site E CESE Site D CESE Site C CESE Site B Grid fabric resources Site A 2. VO registration and information publishing

25. 6-7 May 2002 CEOS GRID Workshop 25 Resource Broker CESE Site H CESE Site G CESE Site F CESE Site E CESE Site D CESE Site C CESE Site B DataGrid Overview (3/5) Certificate Authorities Information Index User Interface JDL script Executable input data input data input data Myjob Search Submit job Request status Check certificate Retrieve result 3. Job submission with local data

26. 6-7 May 2002 CEOS GRID Workshop 26 DataGrid Overview (4/5) User Interface Replica Manager Submit job input data Mydata input data input data input data CESE Site H CESE Site G CESE Site F CESE Site E CESE Site D CESE Site C CESE Site B Replica Catalog Replicate 4. Data replication

27. 6-7 May 2002 CEOS GRID Workshop 27 DataGrid Overview (5/5) 5. Job submission using replicated data Resource Broker CESE Site H CESE Site G CESE Site F CESE Site E CESE Site D CESE Site C CESE Site B Certificate Authorities User Interface JDL script Executable Myjob Submit job Request status Check certificate Search Information Index Replica Catalog Search input data input data input data LFN Retrieve result LFN PFN Logical filename Physical filename LFNPFN :: LFNPFN :: LFNPFN ::

28. 6-7 May 2002 CEOS GRID Workshop 28 DataGrid Activities Testbed validation –writing scripts to test and validate Testbed1 services Develop Use Cases for end-to-end GOME processing and validation demonstration across three sites in Holland, France and Italy) Develop EO Grid Application Interfacing Components –for generic application interfacing High-speed connection to ENEA HPC network Installation of ESRIN DataGrid site –using DataGrid installation tools –installation of 2 CEs: ESRIN cluster using PBS ENEA using LSF/AFS –and 1 SE 0.5TB RAID array on ESRIN cluster populated from ESA AMS MSS archive

29. 6-7 May 2002 CEOS GRID Workshop 29 DataGrid Issues Very-large-scale, complex system with large numbers of participants –Dealing with new concepts and technology –Communication and coordination in large, distributed, multi-cultural, multi- institutional development group –Agressive deployment of middleware releases Driven by needs of HEP –With EO & Biology contributions –Reliant on HEP making the right choices Testbed stability, usability, performance and scalability –Application Grid interfacing layer needs to be developed –After CLIs, need APIs Ongoing rapid prototyping and development –Keeping step with code & documentation Architecture will evolve according to findings –Will take time to make fair assessment

30. 6-7 May 2002 CEOS GRID Workshop 30 Future Directions In general –OGSA and integration of Web Services –Wider uptake of Grid computing concepts In EO –Matrix of common application requirements –Development of Generic Grid platform interface components –Portals-based –Application Frameworks

31. 6-7 May 2002 CEOS GRID Workshop 31 Considerations for CEOS “involvement” in GRID “gridding” of EO emerging technologies and services –Interoperability –EO data format handling –Web-mapping –Archive management Demonstrate GRID applications –International project dimension –collaborative environment –relation with IGOS, WGISS Test Facilities … Support “CEOS standardisation” approach to metadata and data access