Page 1 Jordi Farres Models for scientific exploitation of EO Data ESRIN, Oct 2012 A Web of EO Processing Services ESA UNCLASSIED - For Offical Use 11/10/2012

Page 2 Agenda 1.Introduction 2.A case study - GPOD 3.A model of scientific exploitation – Earth Web 4.Conclusions ESA UNCLASSIED - For Offical Use 11/10/2012

Page 3 Intro: GSCB meeting [ESRIN, 6/6/2012] PI work more frequently on long-term data series (M.Albani) Sentinel data policy: “data available at no fee within the financial limits for operations costs” (J. Aschbacher) SAR Data provision limited by “data quota based on technical constraints” (H.Laur) ESA processing on-demand (GPOD) is employed at 10% in 2009, peaks to 100% in 2012 (J.Farres) ESA UNCLASSIED - For Offical Use 11/10/2012 The policy may meet expectations but not its implementation High pressure for better dissemination and processing services

Page 4 Intro: Cloud Computing & Large ICT providers The Science Cloud – Helix Nebula SuperSites Exploitation Platform GAIA on Amazon NASA uses Amazon's cloud computing in Mars landing mission ESA UNCLASSIED - For Offical Use 11/10/2012 HPC becomes an utility Opportunities to team with large ICT providers

Page 5 What is the preferred exploitation model? An Earth Box 1.Drop all data on large Dropbox 2.Place all data in the cloud 3.Users / service autonomously develop: Applications, Services and Research An Earth Web 1.Expose data on an exploitation platform 2.Federate services from different EO providers 3.Users develop the value added services on a common framework ESA UNCLASSIED - For Offical Use 11/10/2012

Page 6 Intro: An Earth Web A facility for EO dissemination and processing services geographically distributed in a federated domain largely scalable with reliable QoS ESA UNCLASSIED - For Offical Use 11/10/2012

Page 7 Agenda 1.Introduction 2.A case study - GPOD 3.A model of scientific exploitation – Earth Web 4.Conclusions ESA UNCLASSIED - For Offical Use 11/10/2012

Page 8 A case study: G-POD Service ladderESA ESRIN ESA UK-PAC ESA ESAC* Cloud 6- Portal – User and Info mgt √ 5- Service Mgt Deployment √ 4- Processing Services> Data Mgt Discovery & Catalogue √√ 2- Data Resources ASAR, RA, GOME, GOMOS, MIPAS,... (A)ATSR, MERIS SMOSSAR subset 1- ICT Resources2 Clusters1 Cluster IaaS ESA UNCLASSIED - For Offical Use 11/10/2012

Page 9 A case study: G-POD Service ladderESA ESRIN ESA UK-PAC ESA ESAC* Cloud 6- Portal – User and Info mgt 5- Service Mgt Deployment 4- Processing Services 3- Data Mgt Discovery & Catalogue 2- Data Resources 1- ICT Resources ESA UNCLASSIED - For Offical Use 11/10/2012 PORTAL CATALOGUE DATA COMPUTING

Page 10 A case study: G-POD Service ladderESA ESRIN ESA UK-PAC ESA ESAC* Cloud 6- Portal – User and Info mgt 5- Service Mgt Deployment 4- Processing Services 3- Data Mgt Discovery & Catalogue 2- Data Resources 1- ICT Resources ESA UNCLASSIED - For Offical Use 11/10/2012 PORTAL PROCESSOR COMPUTING PROCESSOR

Page 11 A case study: G-POD / SSEGrid Service ladderESA ESRIN ESA UK-PAC ESA ESAC CloudVITO 6- Portal – User and Info mgt √ 5- Service Mgt Deployment √√ 4- Processing Services> Data Mgt Discovery & Catalogue √√√ 2- Data Resources ASAR, RA, GOME, GOMOS, MIPAS,... (A)ATSR, MERIS SMOSSAR subset SPOT Veg 1- ICT Resources2 Clusters1 Cluster IaaS√ ESA UNCLASSIED - For Offical Use 11/10/2012

Page 12 Agenda 1.Introduction 2.A case study - GPOD 3.A model of scientific exploitation – Earth Web 4.Conclusions ESA UNCLASSIED - For Offical Use 11/10/2012

Page 13 An Earth Web Model – Service ladder Service ladderPartner A Location 1 Partner A Location 2 Partner BPartner C 6- Portal – User and Info mgt √ 5- Service Mgt Deployment √ 4- Processing Services√√ 3- Data Mgt Discovery & Catalogue √√√ 2- Data Resources√√√ 1- ICT Resources√√√√ ESA UNCLASSIED - For Offical Use 11/10/2012 Cloud Provider EO Resources Provider EO Service Provider EO Community Mgt

Page 14 An Earth Web Model – Interoperability Service ladderTechnologiesStandardsHMA-AWG 6- Portal – User and Info mgt SSOSAMLPartial (Identity mgt) 5- Service Mgt Deployment PF Virtual Machines Web Services IPF I/F OVF WPS-T NO 4- Processing ServicesWeb ServicesWPSPartial 3- Data Mgt Discovery & Catalogue Web ServicesCSWYES 2- Data Resourceshttps Web Services Metadata File Formats WCS YES 1- ICT ResourcesIaaS Grid brokers OCCI, EC2NO ESA UNCLASSIED - For Offical Use 11/10/2012

Page 15 An Earth Web Model – Challenges ESA UNCLASSIED - For Offical Use 11/10/ Execute processing X on data Y over server Z (portability) 2.Quantify user authorization in relation processing resources 3.Profile users in relation to QoS 4.Ability to quote “budget” for processing X on data Y 5.Accounting à la cloud for ICT resources 6.Harmonized authorization and accounting 7.…

Page 16 Conclusions 1.An “Earth Web” model is better suited for scientific exploitation than a simple “Earth Box” 2.Facilities/Institutions can federate their own resources/services and get access to the federated pool. 3.Facilities can profile different service layers: ICT, Data, Data Mgt, Processing Services, … 4.BUT Interoperability / Standardization for Processing Services is NOT there ESA UNCLASSIED - For Offical Use 11/10/2012