Page 1 Jordi Farres Grid and Cloud activities 15/3/2011

Page 215/3/2011 Agenda 1.Background 2.Scope 3.Function 4.Requirements 5.Proof of concept

Page 315/3/2011 Summary IaaSPaaS What activities has been done? Or are planned? -MIPAS, ERS-SAR processings -GlobCover -SuperSites-GeoHazards -SSEGrid / ESE -GPOD / CIOP - SSEP What would we like to do?ICT provisioning strategy for PDGS Common ground for Service Providers Support to an EO value added chain

Page 415/3/2011 Background of Cloud Computing at EO Activities –Supersites virtual archive on Level3 (CDN) –2009: Re-processing of ERS SAR Wave products in AWS (Amazon Web Services) –2010: Processing of GlobeCover in AWS –2010: Calvalus project with Hadoop (private cloud) –2011: Cloud PoC with HP (private cloud) –2011: MIPAS re-processing in AWS –2011: Virtual Archive 4 with Hetzner Pros 1.Large resource pool 2.Scalability Up and Down 3.Good connectivity from Internet 4.Good Service Levels 5.Pay per use Cons 1.Large EO data seeding 2.Legacy applications 3.License policies

Page 515/3/2011 Agenda 1.Background 2.Scope 3.Function 4.Requirements 5.Proof of concept

Page 615/3/2011 Scope 1.Identify services/functions requiring: 1.High connectivity 2.Variable processing needs – high peaks on short notice 3.Variable storage needs – high peaks on short notice 4.Intermittent services over time 2.in EarthCare PDGS functional breakdown 3.and constituting a sound subset of functions

Page 715/3/2011 EarthCare: PDGS Breakdown Calibration Facility Instrument & Data Monitoring Facility Reference Planning Facility Processing Facility –NRT processing –Processing –Re-processing Archive –Primary –Back-up On-line Archive User Service Facility FOS

Page 815/3/2011 Candidate functions for new ICT model Calibration Facility Instrument & Data Monitoring Facility Reference Planning Facility Processing Facility –NRT processing –Processing –Re-processing Archive –Primary –Back-up On-line Archive User Service Facility FOS - Punctual in time - Processing intensive - Needs subject to change - Needs beyond mission life TBC - High network throughput - High availability - Needs beyond mission life TBC - Intermittent service

Page 915/3/2011 Agenda 1.Background 2.Scope 3.Function 4.Requirements 5.Proof of concept

Page 1015/3/2011 Function 1.Repository resident in virtual facility and fed by Data Circulation. Repository does not expose any functionality. Back-up Archive Re- processing Repository Data Circulation 2.On-need, the Back-up Archive is activated and can serve data. E.g. the primary archive is down or re- processing in the virtual facility is required. 3.Re-processing is served from Back-up archive. 4.Distributed processing techniques are used to parallelize re-processing over a large volume of products. Re- processing 5.Re-processed data is made available to a transient on-line archive. Access peak is served for a transient on-line archive, hence not overloading nominal on- line archive. Transient On-line Archive 6.Re-processed data is fed back to data circulation and to the primary archive.

Page 1115/3/2011 Other Functions Processor Testbeds –Partial re-processing scenario with a beta processor/configuration –Support to test and qualification process. Support to Projects –(Partial) Re-processing scenario using a 3 rd party processor No data circulation is required for these functions Back-up Archive Transient On-line Archive Repository Data Circulation Re- processing Testbed & Projects

Page 1215/3/2011 Agenda 1.Background 2.Scope 3.Function 4.Requirements 5.Proof of concept

Page 1315/3/2011 Operational Context ICT Provider -Internet connectivity -Large storage -Cloud Computing EO (Facility) Operator -Service Operations and Management -Management of ICT Provision -Management Data Provision -Service Change Mgt Monitor Control Resource Allocation EO Data Virtual Machine Operation User EO Engineering -Application Engineering -Software Maintenance -Open Collaborative Development Application Release Maintenance

Page 1415/3/2011 Virtual ICT facility requirements 1.An ICT Provider shall offer Computing, Network and Storage resources at qualified levels of service. 2.Resources shall be scalable within an order of between peaks and lows. 3.Resources allocation shall take effect within 2 days of request. Resources release shall take effect within 2 hours of request. 4.Costs shall be related to allocated resources (pay-per-use model). 5.Resource management shall be available via standard interface, e.g. OCCI. 6.Specific TBD QoS 7.Specific TBS OS/Hw platforms.

Page 1515/3/2011 Use of Virtual Facility by PDGS 1.No asset loss on destruction of virtual facility - Capability to resume service with another provider in 2 months. Virtual images of all services in the virtual facility are hosted and managed at the PDGS facility. Repository at virtual facility consists of EO data which can be recovered from the Primary archive. Mass extraction from Primary archive will required TBD time. 2.Based on ESA convenience capability to switch virtual facility to a second provider in 3 months. Data circulation can initiate the replication of the repository in an ICT provider to a second provider.

Page 1615/3/2011 Software requirements 1.Software licenses shall be aligned to cloud computing paradigm, i.e. to be employed in a variable number of virtual hosts. 2.Software shall be designed to employ numerous hosts in a distributed manner, e.g. using google’s map-reduce platform. 3.Systems shall be designed to sense resource needs and therefore, be able to trigger the allocation/release of resources.

Page 1715/3/2011 Other Back-up and Transient Archives Server and Network resources shall be allocated based on concurrent demand; i.e. the more products are requested the more servers are serving the requests. Re-processing Server resources shall be allocated based on data-driven principles, i.e. the more we can simultaneously process the more servers are allocated.

Page 1815/3/2011 Agenda 1.Background 2.Scope 3.Function 4.Requirements 5.Proof of concept

Page 1915/3/2011

Page 2015/3/2011

Page 2115/3/2011

Page 2215/3/2011