1. ESA PDGS Data Cube
Andrea Della Vecchia, Damiano Guerrucci, Mirko Albani (ESA)
Simone Mantovani (MEEO)
CEOS WGISS#47
29th April 2019

2. Outline ESA PDGS Data Cube ESA PDGS Collaborative Environment
Use Cases
EO Data Management
In-Situ Data Management
CEOS WGCV
System Scalability
ESA PDGS Collaborative Environment
Next Steps

3. Outline ESA PDGS Data Cube ESA PDGS Collaborative Environment
Use Cases
EO Data Management
In-Situ Data Management
CEOS WGCV
System Scalability
ESA PDGS Collaborative Environment
Next Steps

4. PDGS Data Cube Objectives
Brings the users to the data
Provides standardised and harmonised interfaces to manage EO and in-situ data
Innovative, and complementary, approach to access and visualize data
Permits on-the-fly pixel based processing and bulk asynchronous processing
Integration into ESA Collaborative Environment

5. Outline ESA PDGS Data Cube ESA PDGS Collaborative Environment
Use Cases
EO Data Management
In-Situ Data Management
CEOS WGCV
System Scalability
ESA PDGS Collaborative Environment
Next Steps

6. UC1: EO Data Discovery, Visualization and Pixel-based Access UC2: EO Service Discovery and Processing

7. Web based GUI Jupyter Notebook CLI / REST API
Visualization layer Standardised data access interfaces allow connecting a wide range of user interfaces
Datacube Engine/API VMs The deployment of DAS in front of each data source enables effective pixel-based access services
DAS
OGC WCS Server
ESA
Third Party Missions
ESA
Heritage Missions
ESA
Earth Explorer
Data layer Data remain at their own location (multiple data centers) with the original data format

8. TPMs online data storage @ESA
Mission
Product
(~ 0.6 PB)
Temporal coverage
Data format
Hosted processing
Landsat-5
Level 1 (L1G, L1Gt, GEO_1P, GTC_1P)
(Storage: 355 TB)
04/1984 – 11/2011 (link)
EO-SIP (ZIP):
- single band TIF
@ESA
(WCS link; GUI link)
Landsat- 7
Level 1 (TC_1P)
(Storage: 64 TB)
07/1999 – 12/2003 (link)
Tropforest
Multiple
(Storage: 8TB)
07/2006 – 04/2011 (link)
- ZIP: CEOS format
@ESA (under registration)
SPOT-1-5_ESA
Level 1 / Level 2
(Storage: 6TB)
04/1986 – 11/2015 (link)
@ESA (wcs)
IKONOS-2
Level 1
(Storage: 1TB)
12/2000 – 12/2008 (link)
- ZIP: TIFF
OceanSat2
Level 0 / Level 1 / Level 2
(Storage: 126TB)
10/2015 – 07/2018 (link)
- ZIP: HDF
HDF inside a ZIP currently not supported
Seasat
07/1978 – 10/1978 (link)
- TAR: CEOS format

9. HMs online data storage (Temporary storage@ESA)
Mission
Product
(~ 1.6 PB)
Temporal coverage
Data format
Hosted processing
ENVISAT
ASA_GM1_1P
2002 – 2012 (link)
Level 1: ENVISAT (N1)
@MEEO (wcs)
ASA_IMM_1P
ASA_WSM_1P
ASA_IMS_1P
@MEEO
ASA_IMP_1P
ASA_APS_1P
ASA_APP_1P
ASA_WSS_1P
ERS
SAR_IMS_1P
07/1991 – 07/2011 (link)
Level 1: ERS (E1)
@MEEO (wcs_token) Q2-2019
SAR_IMP_1P
ENVISAT.ATS.LST_2P
24/07/ /04/2012
Level 2::nc
ENVISAT.ATS.LST_3P
Level 3: nc
ENVISAT.UPA-L2P (SST)
2002/05/ /04/08
Level 2: nc
ENVISAT.UPA-L3U (SST)

10. EEs online data storage @ESA
Mission
Product
(~ 1.5 TB)
Temporal coverage
Data format
Hosted processing
SMOS
Collection
(Storage L2 ~1.5TB)
MIR_SMUDP2
2010 – today (link)
DBL / GeoTIFF
@MEEO
(GUI link)
MIR_OSUDP2
MIR_BWSF1C
(link)
@ESA Q2-2019
MIR_BWSD1C
MIR_BWLF1C
MIR_BWLD1C
MIR_SCSF1C
MIR_SCSD1C
MIR_SCLF1C
MIR_SCLD1C
MIR_SC_F1B
MIR_SC_D1B

11. UC1: EO Data Discovery, Visualization and Pixel-based Access
The users:
connect to the GUI (
select the collection(s)
define AOI/TOI
connect to the central jupyter hub
use/edit/create notebooks to manage the full data cycle (discover, access, process, visualize)
Open his/her own terminal
use API / CLI to code and send the access / processing query
UC1: EO Data Discovery, Visualization and Pixel-based Access
The GUI:
connects to
sends the access / processing query to the DAS component
The DAS:
retrieves the data from the online storage
sends back the GUI

12. UC2: EO Service Discovery and Processing
The users:
connect to the GUI (
select the collection(s)
define AOI/TOI
connect to the central jupyter hub
use/edit/create notebooks to manage the full data cycle (discover, access, process, visualize)
Open his/her own terminal
use API / CLI to code and send the access / processing query
UC2: EO Service Discovery and Processing
The jupyter hub provides ( :
notebooks with examples of access / processing query to the DAS
interactive interface for live coding to discover, access, process, visualize the datacube(s)

13. Mission-specific data
F E D E R A T I O N
Web based GUI
Jupyter Notebook
CLI / REST API
Visualization layer Standardised data access interfaces allow connecting a wide range of user interfaces
Datacube Engine/API VMs The deployment of DAS in front of each data source enables effective access services
DAS
DAS
DAS
DAS
DAS
DAS
DAS
Data layer Data remain at their own location (multiple data centers) with the original data format
Mission-specific data
Thematic data
Other geospatial data

14. Federated Infrastructure
Available Collections (~ overall storage +5PB)
High Resolution optical satellite data (Level 1) Sentinel 2 (10m, global, since 2015) Landsat 8 (30m, Europe, since 2013) Land Products (Level 2+) Vegetation Indexes (250m - 10m, 2000) Land Surface Temperature (1Km, 2000) Soil Moisture (100Km - 10km, 1978) Land Cover type (1km, 2000) Atmospheric Products (Level 2+ Numerical Model) Air Temperature ( km, 1979) Precipitation (100km - 4.5km, 1979) Aerosol Optical Thickness (1km, 2000) Ocean products (Level 2+ Numerical Model) Sea Surface TEMPERATURE (5km, Global, 2017) OCEAN Salinity (10Km, Global, 2010) Chlorophyll data (5km, Europe, 2010 – 2013) Sea Surface Height (12km, Europe, 1993 – 2017) Sea Surface Velocity (12km, Europe, 1993 – 2017) ESA TPMs/ HMs/ EEs Landsat 4/7 (30m, Europe, since 1984) SPOT (1986 – 2015) ERS-1/-2 (1991 – 2011, on demand) ENVISAT (2002 – 2012, on demand) SMOS (10km, since 2010)
In BLUE external repositories

15. ESA DC / ODC Federation – Next Step
WGISS-46-28: WGISS-Exec (Robert Woodcock, Andrea Della Vecchia) to prepare one to two slides for the SEO on what WGISS is doing for Data Cubes and propose a way forward for cooperation between WGISS and SEO on the FDA Data Cube topic.
ESA “Data Cube Service” Project will address international interoperability issues … WCS 3.0 Evolution …
Standardised OGC WS Interface
OGC WCS Server
Proprietary APIs
ODC
ESA
Third Party Missions
ESA
Heritage Missions
ESA
Earth Explorer

16. Outline ESA PDGS Data Cube ESA PDGS Collaborative Environment
Use Cases
EO Data Management
In-Situ Data Management
CEOS WGCV
System Scalability
ESA PDGS Collaborative Environment
Next Steps

17. UC3.1: In-situ / EO Service Discovery and Processing (on-going)
UC2.1: In-situ / EO Data Discovery, Visualization and Pixel-based Access
- Atmospheric Correction Inter-Comparison Exercise (ACIX)
- Radiometric Calibration Network (RadCalNet)
UC3.1: In-situ / EO Service Discovery and Processing (on-going)
- Customized UI to support AOT/WV/SR inter-comparison
- Jupyter notebook to support live coding / sharing

18. EO and In-situ Data Management
Preferred - Under development
Preferred
Optional
SQL
ingestion
In-situ
On-the-fly Translator
rasterization EO
WCS
Pre-processing (e.g., ARD):
Re-projection
Re-sampling
Format conversion
OpenSearch

19. Outline ESA PDGS Data Cube ESA PDGS Collaborative Environment
Use Cases
EO Data Management
In-Situ Data Management
CEOS WGCV
System Scalability
ESA PDGS Collaborative Environment
Next Steps

20. Atmospheric Correction Inter-Comparison Exercise (ACIX) http://calvalportal.ceos.org/projects/acix
ACIX aimed to bring together the developers of Atmospheric Correction (AC) processors, to generate who were invited to generate the corresponding Bottom-Of-Atmosphere (BOA) products. The European Space Agency and NASA organise this exercise on AC processors inter-comparison using Landsat-8 and Sentinel-2 imagery of various sites, i.e. agricultural, snow/artic areas, deserts and coastal. A common and harmonised inter-comparison procedure was agreed and followed by all the participants.
For more information about the ACIX workshops, please follow the links to the SPPA web pages: ACIX I 1st WS  ,  ACIX I 2nd WS
Objectives
To elaborate concepts, protocols and guidelines for the inter-comparison and validation of BOA products
To better understand BOA reflectance uncertainty contributors by comparing the outputs of different AC schemes
To identify and review the different uncertainty contributors
To propose further improvements of the available AC schemes
Expected Outcomes
Description of concept, protocols and procedures for inter-comparing and validating products
Assessment of the relative differences among the inter-compared AC processors results
Definition of key regions and key periods for validation and quality assessment
Description of a coordinated plan for inter-comparison and validation activities

21. ACIX Inter-Comparison Exercise
Inter-comparison of BOA radiometric calibration processors
AERONET stations over 19 sites to perform in-situ measurements of AOT, WV and SR
Landsat-8 and Sentinel-2 acquired over sites and converted to BOA with selected processors, to generate AOT, WV and SR
Inter-comparisons performed for all the products on image subset of 9km x 9km centred on every AERONET sites

22. Radiometric Calibration Network (RadCalNet) https://www. radcalnet
RadCalNet is an initiative of the Working Group on Calibration and Validation of the Committee on Earth Observation Satellites. The RadCalNet service provides satellite operators with SI-traceable Top-of-Atmosphere (TOA) spectrally-resolved reflectances to aid in the post-launch radiometric calibration and validation of optical imaging sensor data. The free and open access service provides a continuously updated archive of TOA reflectances derived over a network of sites, with associated uncertainties, at a 10 nm spectral sampling interval, in the spectral range from 380 nm to 2500 nm and at 30 minute intervals. Each individual site is equipped with automated ground instrumentation in order to provide continuous measurements of both surface reflectance and local environmental/atmospheric conditions needed for the derivation of TOA reflectance values. TOA reflectances provided on this portal are derived from the individual sites surface and atmosphere measurements using a common method through a central processing system. Each member site takes responsibility for the quality assurance of the surface/atmosphere measurements provided and is subject to peer review and rigorous comparison to ensure site-to-site consistency and SI traceability.

23. RadCalNet Exercise
Support post-launch TOA radiometric calibration and validation of optical EO data
Ground nadir-view reflectance measurements (converted to TOA), and atmospheric measurements such as surface pressure, columnar water vapour, columnar ozone, aerosol optical depth and the Angstrom coefficient, over 4 sites
Nadir-view top-of-atmosphere reflectance at 30 minute intervals from 9am to 3pm local standard time at 10 nm intervals from 400 nm to 2500 nm

24. ACIX/RadCalNet Inter-Comparison Exercise
Web based GUI
Jupyter Notebook
Data Preparation: in-situ and EO Level 2 data are indexed and registered in the data cube, preserving both radiometric and geometric properties
Data Access: DAS shall offer access to both EO and In-situ data via unified interface (i.e. WCS)
Data Processing: Jupyter allows the users to perform inter- comparison scripts, sharing the code with authorise users, and accessing always the same data via WCS server.
PROs
Centralised data repository, replacing scattered ftp servers
Harmonised and standardised online data access interface, permitting data download
Reproducibility of inter-comparison exercises (via Jupyter notebook), restricted to authorised users
Easy sharing of algorithms and results
DAS
OGC WCS Server
Landsat-8/Sentinel-2
Data
In-Situ: ACIX and
RadCalNet

25. Outline ESA PDGS Data Cube ESA PDGS Collaborative Environment
Use Cases
EO Data Management
In-Situ Data Management
CEOS WGCV
System Scalability
ESA PDGS Collaborative Environment
Next Steps

26. ESA PDGS Data Cube SW Refactoring
Software refactoring currently on-going
Porting all PDGS data cube components to Docker and Kubernetes – fast and easy deployment and horizontal scalability
Faster indexing time
Quicker time response
To support concurrent users
Preserving all functional/interoperability requirements
Operational Environment ready by June 2019
http(s)://datacube.pdgs.eo.esa.int/
http(s)://jupyter.pdgs.eo.esa.int/

27. Interfaces Adherence to standards / Evolution of standards
Discovery – OGC/CEOS OpenSearch BP – to discover collections and products
Access – OGC Web Coverage Service (WCS 2.0) – to provide pixel based access (including sub-setting in any dimension)
Processing
OGC Web Processing Service (WPS 2.0) – to execute processing on data cubes
Jupyther HUB – “The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text” (see

28. Outline ESA PDGS Data Cube ESA PDGS Collaborative Environment
Use Cases
EO Data Management
In-Situ Data Management
CEOS WGCV
System Scalability
ESA PDGS Collaborative Environment
Next Steps

29. Collaborative Environment
EO collaborative environment is a virtual working environment providing discovery and access services to EO data, analysis tools, processors, Information and Communication Technology resources required to work with them, through one coherent interface. A collaborative environment aims to change the paradigm from “data to user“ to “user to data”

30. ESA PDGS Collaborative Environment
To provide the ESA PDGS with a set of interoperable and federated services permitting the users to:
Access to missions/platforms information supported by a common ontology
Discovery and, if possible, direct access to EO and in-situ data:
Copernicus Missions (e.g. Sentinels)
Third Party Missions - TPMs (e.g. SPOT, Landsat, …)
Heritage Missions - HMs (e.g. ERS-1/2, ENVISAT, …)
Earth Explorer – EEs (e.g. SMOS, Cryosat, SWARM, …)
International repositories (e.g., NASA CMR, CEOS IDN)
Discovery and access to exploitation services (e.g. datacube):
Browse/visualization tools for time series analysis
EO data extraction, resampling and reprojection
Hosted Processing for authorised users/communities (e.g. CAL/VAL)

31. Conceptual Model Web User Interfaces
Operational interface
ESA
External
Query and service requests
Web User Interface for pixel-based access, analysis and processing
Web User Interface for granule-based discovery and/or retrieval
Mission oriented GUI
- SWARM
- Cryosat
- SMOS
- External
Multi Mission GUI:
– ESA EOCAT
ESA EO Information Page:
- ESA EO Gateway
International EO Information Page:
- CEOS IDN
……….
ESA PDGS Data Cube GUI, VIRES GUI, etc…
External catalogue application
Conceptual Model
Data access interface
Operational Data Access
Web User Interfaces
Standard services request Interfaces (e.g., restful API)
Protocol for data access
(e.g., OGC, Open Datacube)
OpenSearch
(collections and granules)
(collections
and granules)
Products Access Interface
Knowledge-Base Facility
EO Data & Service Metadata Repository
Thesauri Repository
Standardised Interoperable Interfaces
Metadata management
Centralised Knowledge-Base Facility
Data Repositories
Data Repositories
Standardised Interoperable Services Interfaces
IT Resources provided as services (IaaS, PaaS, SaaS)
VM – Browse Images Generation
VM – DataCube Engine/API
VM – Processors
User Applications
Storage
Temporary
ESA User Applications
Hosted Processing
ESA Missions Data Repositories
External Web Interfaces
Copernicus Data Repositories
External Web Interface
TPMs
HMs
EEs
S-1
S-2
S-3
S-5

32. PDGS DataCube into Collaborative Environment
MEEO
DAS & Jupyter
@ ESA
ESA
(NFS read-only)

33. Outline ESA PDGS Data Cube ESA PDGS Collaborative Environment
Use Cases
EO Data Management
In-Situ Data Management
CEOS WGCV
System Scalability
ESA PDGS Collaborative Environment
Next Steps

34. Next Step – 2019 ESA PDGS Data Cube TTO by Q2 2019, including:
Software refactoring, based on Kubernetes, to boost performance and manage concurrent users
Internal OpenSearch interface aligned with CEOS OS BP 1.2
All ESA TPM/HM/EE collections available
WCS & Jupyter interfaces for ESA Cal/Val initiatives (ACIX, RadCalNet)
Update of ESA by Dec ’19, including:
Cooperation with CEOS WGISS / WGCV / FDA teams
WPS interface available
Interoperability enhancement (e.g., connection with Open Data Cube instances in synergy with ESA Datacube Service Project)
Web User Interface for ESA Cal/Val initiatives (ACIX, RadCalNet)
New ESA missions available