1. Research and Service Support Resources for EO data exploitation RSS Team, ESRIN, 23/01/2013 Requirements for a Federated Infrastructure

2. Summary –RSS Overview –RSS G-POD process –RSS flexible resources for on-demand processing –Facts & Figures –Requirements for a Federated Infrastructure –Questions & Answers

3. RSS Overview Research and Service Support The ESA Research and Service Support (RSS) service provides resources for supporting Earth Observation (EO) data exploitation RSS users are EO Principal Investigators and EO Service Providers The environments made available to RSS users are: Grid Processing On-Demand (G-POD) Service Support Environment (SSE) E-collaboration environment (Join&Share) OGC services (WMS, WFS, WCS) Virtual Reality Theatre (VRT) Knowledge-based Earth Observation (KEO) Reference Data Sets (RDS) EO Ontology search engine More than 3500 people are registered as RSS users On the industry side Logica delivers the RSS service as part of the ESRIN O&M Framecontract More information available from the RSS Portal (rssportal.esa.int ) Let’s focus on G-POD

4. Research Process

5. Principal Investigators EO algorithms delivery Data type and range indication Output validation On-demand EO data processing Use of produced data (projects delivery, publications, etc) RSS Data are made available in the RSS catalogue Algorithm porting Integration design Implementation Test and validation (involving the PI) On-demand EO data processing Delivery RSS G-POD Process Steps

6. RSS Flexible Resources Flexible Infrastructure satisfies: -HW requirements -Connectivity requirements -SLA (HA, help desk, ticketing systems, etc.) On-demand processing service: Technology Infrastructure Application ESRIN - 172 cores - 400 TB UK-PAC - 88 cores - 300 TB Flexible Infrastructure - 10-200 cores - 1-10 TB EO Scientists Principal Investigators delivery Process EO data

7. RSS Facts & Figures On-demand Processing: team composition and responsibility The G-POD team is composed by 4 selected highly skilled resources Key areas of responsibility are: Interaction with Principal Investigators Algorithm Understanding and proposal evaluation Resources planning (data, storage, integration, capacity, etc) Algorithm integration (porting, design, implementation, test) Data management Processing environment set-up User support (for on-demand processing) Bulk Processing management Key skills are: Remote sensing background / EO data exploitation expertise Software development /software integration Virtualization technology Grid / Cloud technology Operations and Maintenance Scientific goal orientation

8. RSS Facts & Figures On-demand Processing: actual figures in the last 3 years Supported more than 40 active users per year Supported >20 processing/re-processing campaigns (included entire missions, e.g. MERIS, ASAR, SMOS and TPM) Integrated ~10 new algorithms per year Upgraded ~15 algorithms per year Managed >450 TB data farm (ESA and TPM) Provided 3 G-POD testbeds for algorithm validation Set-up flexible (additional) processing capacity in less than 2 working days

9. Design Requirements Design requirements for a federated ground segment, from the Research and Service Support operations point of view, can be summarized in four simple words:  Simple  Flexible  Portable  Scalable These requirements are connected: - a system made of simple parts is usually flexible (because you can replace or intervene on small parts of the system to change its behavior). - a portable system is scalable as well, provided that can be modularly replicated

10. Simple Systems shall be modular (a set of simple generic modules: processor orchestrator, data storage module, data transfer module, etc…). Each module shall be developed and tested separately. The simpler the module, the easier will be verification, operation, etc… Design specifications and deliveries shall contain always input/output working samples for an easy understanding (50 pages of manual describing how to set application configuration files are sometimes less clear than a set of 10 sample files) Systems shall implement well-known protocols and technologies (e.g. HMA protocols, Web-Based protocols, open data formats, no need to re- invent the wheel)

11. Flexible Systems shall have a generic orchestrator, with a set of plugins, extensions, customizable steps, etc... (the more freedom is left to the operator and the developer, the easier it will be to adapt to different and always changing user requirements) Systems shall have generic design specifications, independent by the particular mission, satellite or sensor. If a new satellite/sensor comes with new requirements, new features shall be added to existing design via new or customized modules, not re-designing the entire system. Implementation shall use scripting languages for all the orchestration and integration parts, compiled languages shall be used only when high computing performance are required.

12. Portable Systems shall be independent from the hardware (run on virtualized hardware), they should require only performance and not hardware solutions (e.g. a system can require 10MB/s disk access, but not a SAN storage) System shall be independent from the network configuration (e.g. run on private or public networks, with customizable hostnames/IPs, and any type of security on top) Implementation shall be as much as possible independent from the particular OS, OS distribution version, COTS version, etc… (e.g. not developed for RedHat x64 5.1 2010-01-01 version, but for Linux x64 or Linux RedHat x64)

13. Scalable Load balancing or other workload distribution technologies shall always be implemented for any part of the system. This provides both high availability and capacity to adapt the system to the real usage. Massive processing should be performed always in a “cloud” or “grid” way (meaning that the processing should be sent to different computing nodes, parallelizing the different steps. If you need to do more processing, you just add more nodes) Data should be placed in a distributed file storage (or data farm), which can be part on the cloud, part on local systems, etc... and can be increased easily (e.g. adding new local NAS, new storage blocks in the cloud, etc…). This provides also easier data management, backup, data access, etc…

14. Research and Service Support RSS contacts: RSS Website: rssportal.esa.intrssportal.esa.int Join&Share: wiki.services.eoportal.orgwiki.services.eoportal.org email: giancarlo.rivolta@esa.intgiancarlo.rivolta@esa.int Thank you for your attention!