Status: For Information Only ESA UNCLASSIFIED - For Official Use Cloud Processing at ESA [EO Payload Ground Segment] Cristiano Lopes, ESA CEOS WGISS-40 30/09/2015

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 2 Overview – Cloud Computing Uses Cloud computing and Earth Observation:  Processing,  Archiving,  Distribution,  Discovery… Cloud computing uses in ESA EO Ground-Segment  Past activities  On-going activities  Future activities

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 3 Overview – Cloud Computing Cloud Computing definition:  Cloud Computing, Wikipedia article Cloud Computing  "Emerging Technologies/WGISS" Presentation, Slides 10-15, CEOS Plenary 28 th "Emerging Technologies/WGISS" Presentation  Cloud is simply outsourcing in a new form, Joe McKendrick, Forbes, Oct 18 th 2014 Cloud is simply outsourcing in a new form Traditional cloud service models:  Infrastructure As a Service (IaaS)  Platform As a Service (PaaS)  Software as a Service (SaaS)

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 4 Overview – Cloud “Cloud services” are simply ICT resources that are hosted and managed by someone else and offered as a service (XaaS).

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 5 Distribution – Content Delivery Network EO Data Distribution over Content Delivery Network (CDN): CDN provides dynamic geographical load balancing (local caching); GOCE Gravity field data, distributed over CDN with good success (5 releases); ENVISAT AND ERS SAR data, also distributed with low success (due to data specificities – on-demand data) Plus: Very high user satisfaction – download performance, Transparent caching of data at different locations, Peak loads not affecting ESA infrastructure Minus: Expensive, Only for “stable” and consolidated data Currently not in use – replaced by a managed hosting service.

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 6 Infrastructure – Cloud Deployment EO Data Handling Infrastructure deployment in Cloud Provider: Full deployment of existing infrastructure in a Cloud/Virtualised Environment; “Porting” / Deployment of existing applications as-is (templates): Archive, Processing Management, Distribution, Circulation, Discovery, etc. Initial activities pilots led to full reference and integration platforms deployed in the “Cloud”. Plus: Very easy to deploy existing applications; No noticeable functional issues; Very Flexible – One environment per need (one mission, one project); Pay-per-use (when needed) – minimises costs (resources are only used when needed). Minus: IPR / License issues; Performance impacts (when compared with physical infrastructure, 10x in some cases); Security (Policy, very difficult to demonstrate full compliance for production platforms). Results led to follow-up activities.

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 7 Processing – Cloud Deployment EO Data Processing Pilots/Trials with commercial Cloud Provider: Virtual servers allocated in the cloud – Scheduler and Processing Algorithm. Data transferred off-line via media; ENVISAT MIPAS Reprocessing – 200 VM, 400 CPUs for 5 weeks. 8TiB input, 600GB output. ENVISAT ASAR NRT Processing – 10 VM, 20 CPU; runs with 20 and 50 input products taking ~20 and ~45 minutes. Plus: Very easy to deploy existing processing algorithms; No noticeable data quality issues; Easily scalable (linked to Provided availability). Minus: Relative Expensive (compared to owning hardware); Static infrastructure (full environment started and stopped manually – templates used for processing algorithms); Data transfers cumbersome (Disk In, Network out – costs). Results led to follow-up activities.

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 8 Infrastructure–Managed Hosted Services DISSHARM Project and Platform: Managed hosting service for ESA EO Data Distribution; Single-Tenant virtualised environment with dedicated storage; Located within the ESA EO WAN Backbone and close to its data entry; Currently in Production. Plus: Integrated in ESA EO WAN (data travels locally); Managed as a service (Built-to requirements; Provider responsible for architecture, security); Bandwidth (currently 2Gbps, scalable to 10Gbps contractually). Minus: Limited resources (VMs, storage); No local data redundancy (cost reduction choice); Results led to follow-up activities.

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 9 Infrastructure–Managed Hosted Services Evolution ELVIS Platform  DISSHARM Evolution: Managed hosting service for ESA EO Data Distribution & Processing; Single-Tenant virtualised environment (in this case with more processing resources) with dedicated storage, with a multi-tenant cloud connection (Cloud Bursting); Access to the Provider’s commercial cloud via internal LAN; Currently on early stages of production (cloud access still not present). Example use case - SAR On-the-fly Processing: Process SAR data (ERS, ENVISAT, ALOS) on download request by user; Perform processing on the back-end transparently to the user; Cache output product for next user ; Cloud bursting as needed by demand (to be implemented).

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 10 Processing - Hosted Processing Super Sites Exploitation Platform and Project (SSEP) : Collaborative environment for end-user’s data exploitation, through coordinated access to data, tools and processing power; Virtual workstation on a cloud platform; Integrated information sharing and support tools; Demonstrated on Helix-Nebula.Helix-Nebula Plus: Change in paradigm (bring users to data); Abstract ICT from users (transparently multi-sourced resources); Minus: IPR / License issues (data, tools); Cloud-API Nightmare (different cloud providers using different APIs). Results led to follow-up activities.

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 11 EO Exploitation Platforms Earth Observation Exploitation Platforms Follow to the SSEP concepts On-going activities for Thematic Exploitation Platforms (TEP):  Geohazards (SSEP follow-up),  Coastal, Forestry, Hydrology, Polar and Urban TEP activities are industry driven, with input from ESA on overall architecture:  Open source,  Standards,  Infrastructure Independence Earth Observation Platform - Test bed (Pilot): Open Tender: Solution defined by provider and offered as a service; Hosted processing only (no download of source data); Hosting Sentinel-2, Landsat and ENVISAT MERIS data; ESA provide free (pre-paid) access to limited test and science users

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 12 Lessons learned, challenges, opportunities Cloud Services/Technology Lessons learned:  New solutions for old problems (e.g. Processing, Distribution): Challenges:  Technology maturity  (Numerous APIs, different SLAs, reliability of suppliers)  ESA Industrial Policy  Contract placing;  Security  Compliance with ESA Security Rules. Opportunities  Neutral Environment with open access to anyone, one place to develop, prototype and run applications;  Potentially unlimited resources allowing for new ways to exploit EO data (big data);  Space to define an EO Cloud Architecture.

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 13 Cloud Processing Framework (1) Processing of EO Data in the Cloud. Challenge:  Multiple data sources;  Multiple processing algorithms;  Multiple possibilities of joining processing algorithms;  Quasi-Unlimited resources to control. Opportunity for defining standards and best practices: From existing multiple solutions; From existing or new standards; Tailored to the EO Data Specific Needs.

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 14 Cloud Processing Framework (2) Processing of EO Data in the Cloud. Potential domains: Data Discovery Where  Opensearch? Data access How  OpenDap?, WCS?, HTTP GET [from Object Storage]? Resource management How to provision Cloud Resources  Cloud Scheduler?, Apache jClouds/libCloud? How to do VM Management  Condor?, Torque?, Grid(any)? Workflow management (Algorithms) How to “chain” algorithms  BEPL?, XPDL?

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 15 Cloud Processing – CERN ATLAS. Learning from others  The ATLAS experiment driven by CERN. “The Evolution of Cloud Computing in ATLAS”, Taylor et all, ATL- SOFT-PROC , 27/05/2015.

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 16 Summary In ESA Earth Observation, a number of activities related to cloud technology and services have been performed. The Cloud is an enabler that allows ESA to perform is data exploitation activities in a more efficient way. We believe there is room for collaboration and cooperation in the use of the Cloud with other partners. One such possibility of collaboration is the definition of standards, best-practices and an architecture for EO Data Exploitation. The Cloud Processing Framework is a first example.

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 17 Thank you! Cristiano Lopes Ground Segment System Engineer Earth Observation Programmes Phone: