Tecnologías Grid aplicadas a Ciencias de la Tierra Manuel López-Puertas (IAA, Granada) Page 1 1ª Reuni ó n e-CA, Granada, Junio 2007

Page 2 1ª Reuni ó n e-CA, Granada, Junio 2007 EU grid project, follow on of DataGrid ( ) 202 sites, 47 countries worldwide CPUs, 11 Petabytes, concurrent jobs Originally from High Energy and Life Sciences => expanded to many other areas (Geosciences) Ideal for scientific research where the time and resources are impractical for traditional IT 25 linked grid projects EGEE  More details of EGEE in posters

Page 3 1ª Reuni ó n e-CA, Granada, Junio 2007 CYCLOPS aims to bridge the gap between Grid and GMES (Global Monitoring for Environment and Security) communities EU-IndiaGrid, funded by EC: an European and Indian Grid-focused project. DEISA (200 teraflops of supercomputing infrastructure). Founded by Europe's National Research and Education Networks (NRENs) and the European Commission. DEGREE (Dissemination and Exploitation of GRids in Earth sciencE) G-POD (Grid Processing on Demand) (ESRIN, ESA) u Access to ESA multi-mission catalog EGEE Earth Science grid projects

Page 4 1ª Reuni ó n e-CA, Granada, Junio 2007 WHY EGEE for GEOSCIENCES ? Grid is very well adapted for Geosciences applications: For statistical approach –intensive computation/storage For rapid solution in case of many independent jobs Sharing and/or processing large datasets in the large-scale European projects

EC Specific Support Action project (10 Institutes) OBJECTIVES: Bridge the Earth Science and GRID communities throughout Europe Ensure that Earth Science requirements are satisfied in next Grid generation Ensure the integration of emerging technologies for managing Earth Science knowledge Demonstrate the interest of Grid for Geosciences with scientific results already obtained. DEGREE: Dissemination and Exploitation of GRids in Earth sciencE

Earth Sciences Applications and Requirements (DEGREE Report) (1/2) Deal with enormous amount of data (size and number of files) (Envisat: 500 Gb daily) Large computational needs Differences from other Science domains: u Deals with Geospatial 4D data u Many different domains u Scattered among all countries and numerous Institutes u Complex work

DEGREE Report on Earth Sciences Applications and Requirements /2/2) Specific requirements: Reliability (good, well established Quality of Service) Real time and Instantaneous access Need to access licensed software (IDL, Matlab, Geocluster, …) Data policies on input/output data (complicated security requirements) Data scattered around various institutes, various formats, metadata in various forms => Data management is essential (Accessibility and Harmonization) is essential => Need for a standardization on Grid service Earth e-science can be an essential improvement in research

Page 8 1ª Reuni ó n e-CA, Granada, Junio 2007 Earth Science domains

Page 9 1ª Reuni ó n e-CA, Granada, Junio 2007 EXAMPLES (1) Grid properties/requirements: Large number of files (~ 40000; ~40000/per algorithm) Metadata Complex algorithm License for IDL GOME Ozone profilesMERIS Global mosaic Grid properties/requirements: Large dataset (size+ # of files) High security access Dataflow Monitoring License for: Globus GT3/4 & Glite and LCG under testing

Page 10 1ª Reuni ó n e-CA, Granada, Junio 2007 EXAMPLES (2) Grid properties/requirements: A full reprocessing needs 4 TB of input data and 1 TB output Full processing time on e.g. 100 nodes: 12 days!. 2 Grid nodes => an NRT service. Globus GT3/4 & Glite and LCG under testing GRIMI-2 (MIPAS)Ozone in polar regions Grid properties/requirements: Currently being ported to EGEE High security and restricted access for data Licensed software

Page 11 1ª Reuni ó n e-CA, Granada, Junio 2007 Other examples SEISSOL, CMT, SpecFem3D, (research into earthquake simulations) KORBA aquifer COMSIMM (looking at current and future climate trends) ICAROS (Chemical Assimilation of Remote Sensing Observations of the Stratosphere) Space weather (SPIDR)

Page 12 1ª Reuni ó n e-CA, Granada, Junio 2007 ES GRID, e-collaborations and SOA portals Survey: 30 portals Analyzed: 17. GRID: 8; Data dissemination: 4; Collaborative: 5

Page 13 1ª Reuni ó n e-CA, Granada, Junio 2007 VOs in Earth Sciences (EGEE) For Geosciences: two VOs: ESR (Earth Science Research) : 50 members in an average, 10 countries, belonging to Academic EGEODE (Expanding GEOscience On DEmand) : 30 members (~15 from Academy) centred on the use of the software, Geocluster, developed by CGG-Veritas (France). Devoted to: Generic seismic plate form

Page 17 1ª Reuni ó n e-CA, Granada, Junio 2007 Summary Deal with enormous amount of data Large computational needs =>Grid is very well adapted for Geosciences applications However, Earth Science community is “very reluctant to deploy their applications” Not many applications ported. Why? Differences from other science fields (Geospatial 4D data, multidisciplinary, scattered, complex work)? Some alternatives (GPOD-ESRIN) seem driven by “costly effective” more than by e-science Others…. NEEDS: Data policies (security reqs., standardization) Earth e-science can be an essential improvement in research

Page 18 1ª Reuni ó n e-CA, Granada, Junio 2007 Our Needs: Retrieval of MIPAS spectra

Our (GAPT-IAA) Needs Earth Observation satellite data (MIPAS/Envisat) Scientific accurate retrievals (not operational) Data volume: 100 scans/orbit*14 orbits/day=1400 scans/day. ~4 years => 2 M scans 20 species from each scan Pre-processing (at home computers) Full calculation: u Place a fixed set of large input files (HITRAN database) + u ~20 Mb (input/output) per scan (job) u A facility to send arbitrarily structured jobs to the batch computer (e.g., LSF, SLURM, etc. u Large RAM memory (1-3 Gb) u Compiler restrictions: Sun compiler to Linux Opteron:OK; PPCs, IBM compilers, Intel: Problems u CPU: 10 min(LTE)-5 hours(NLTE) per species & scan (Dec HP Fortran V5.5) u 1 hour * 20*2 M ~ 4500 years!

Page 20 1ª Reuni ó n e-CA, Granada, Junio 2007