A. Della Vecchia, D. Guerrucci, M. Albani (ESA) Federated Earth Observation (FedEO) CEOS WGISS Meeting #46 A. Della Vecchia, D. Guerrucci, M. Albani (ESA) Yves Coene (Spacebel) 23/10/2018

Outline Introduction Activities & Evolution Metrics Software refactoring ESA Catalogue TTO within Common Service IDN metadata population WGISS Data Asset Metrics

FedEO: Federated Earth Observation Gateway System FedEO = Federated Earth Observation missions access The FedEO system provides a unique entry point to a growing number of scientific catalogues and services.

WGISS Connected Data Assets

Outline Introduction Activities & Evolution Metrics Software refactoring ESA Catalogue TTO within Common Service IDN metadata population WGISS Data Asset Metrics

Software Refactoring – Objectives Optimization of the gateway and the catalog - quicker time response Optimization of the dataset metadata ingestion job – faster ingestion time Porting all FedEO components to Docker and Kubernetes – fast and easy deployment and horizontal scalability Preserving all functional/interoperability requirements

Data Ingestion

Performance Google Cloud Platform (2017) N=3 .. N=9 ESA Cloud Platform (2018) N=4

Time Response 9N Std-2 10M slower than 9N Std-2 1M (0.27sec) (0.8sec) (12M real index files) 3N Std-4 faster than 3N Std-2, 1M entries Interoute (4N std4 - 12M) Google (9N st2 - 10M) Google Cloud Platform ESA cloud

Software Refactoring – Results New FedEO SW boost significantly metadata ingestion (+400% up to 20M entries) and time response (3x up to 10x faster wrt concurrent users). New FedEO SW preserves all the current functional/interoperability requirements New FedEO SW will be deployed at ESA beginning 2019

Outline Introduction Activities & Evolution Metrics Software refactoring ESA Catalogue TTO within Common Service IDN metadata population WGISS Data Asset Metrics

ESA Collaborative Environment To provide the ESA PDGS with a set of interoperable services permitting the users to: Access to missions/platforms information supported by a common ontology Discovery and, if applicable, direct download of EO data: Copernicus Missions (e.g., Sentinels) Third Party Missions - TPMs (e.g. SPOT, Landsat …) Heritage Missions - HMs (e.g., ERS-1/2, ENVISAT instruments …) Earth Explorer – EEs (e.g., SMOS, Cryosat, SWARM, …) International repositories (e.g., NASA CMR, CEOS IDN) Discovery and access to basic services (e.g., datacube): Browse/visualization tools and time series extraction EO data extraction, resampling and reprojection Hosted Processing for authorised users/communities (e.g., CAL/VAL)

Core Services close to the data M2M Interfaces Online Data Storage Distribution Facility Catalogue Clients Web Service Interface (e.g., OADS, ftp, http) Hosted Processing Data/Service Catalogue Remote Desktop Access - CAL-VAL Activities - Access Point for Application Platforms EO-SIP VM – Browse Images Generation VM – DataCube Engine/API Specific Web Portal: - SWARM - SMOS - Cryosat - External Thesauri Service Query Population Multi Mission Portal – ESA eoli Data Access Information Page: - ESA EO Gateway TPMs HMs EEs VM – CAL/VAL Processors International EO Gateway Information Page: - CEOS IDN Core Services close to the data EO Data Visualization and pre-analysis clients ESA PDGS Data Cube

ESA Catalogue TTO by Q1 2019 Core Services close to the data M2M Interfaces Online Data Storage Distribution Facility Catalogue Clients Web Service Interface (e.g., OADS, ftp, http) Hosted Processing Data/Service Catalogue Remote Desktop Access - CAL-VAL Activities - Access Point for Application Platforms EO-SIP VM – Browse Images Generation VM – DataCube Engine/API Specific Web Portal: - SWARM - SMOS - Cryosat - External Thesauri Service Query Population Multi Mission Portal – ESA eoli Data Access Information Page: - ESA EO Gateway TPMs HMs EEs VM – CAL/VAL Processors International EO Gateway Information Page: - CEOS IDN Core Services close to the data EO Data Visualization and pre-analysis clients ESA PDGS Data Cube ESA Catalogue TTO by Q1 2019

Third Party & Earth Explorer & Heritage Missions Visualisation Layer Metadata Layer Data Layer Google/Qwant Search ESA Gateway & Collection Catalogue ERS SAR ENVISAT ASAR SciHub Sentinel-1/2/3 Sentinel Data Repository ESA Third Party & Earth Explorer & Heritage Missions SMOS CRYOSAT-2 LANDSAT SPOT OCEANSAT TROPFOREST SEASAT IKONOS ESA Earth OnLine ESA Map Viewer Copernicus Dataset EO Products Discovery CCMs Repositories Catalogue Clients Catalogue API publicly available: OGC OpenSearch Specification CEOS WGISS OpenSearch Best Practice Distribution Facility Online Data Storage

ESA Catalogue TTO – Results ESA Catalogue shall: be the centralised metadata repository of ESA Collaborative Environment reuse same FedEO SW manage collections Digital Object Identifiers (DOIs) be part of CEOS WGISS Data Asset via FedEO

Outline Introduction Activities & Evolution Metrics Software refactoring ESA Catalogue TTO within Common Service IDN metadata population WGISS Data Asset Metrics

Metadata Export into IDN – Today 20 ESA collections, providing two step search, today on IDN via FedEO DIF-10 generator

FedEO Metadata Mediator – Ongoing Automatic Procedure – beginning ’19 on FedEO operational environment at ESA Partner Metadata repository Metadata Import Harvester tool FedEO Collection Catalogue FedEO Gateway ISO to DIF-10 Metadata Mediator Metadata Preparation IDN Complementary Information gcmd keyword Metadata Preparation IDN guideline for Information Content completeness and consistency ESA Thesauri Service DIF-10 Validator Metadata Export DIF-10 Encoding IDN repository DIF-10 Validation

Metadata Export into IDN – Ongoing A fully automatic metadata mediator is under development and testing. New collections almost ready on development platform at Spacebel http://geo.spacebel.be/opensearch/dif10.html In BLUE collections ready to be uploaded to IDN In RED partners where technical contacts for IDN population update through FedEO procedure need to be started Repository Collections Verified To be Verified ESA 170 20 150 Copernicus Sentinel 5 - DLR 184 66 118 EUMETSAT 734 CNES 8 ROSCOSMOS 28 VITO 31 11 JAXA 45 ESA CCI 125 48 77 CMEMS 2 168

Open Issues with DIF-10 Metadata Preparation (https://idn.ceos.org/subset/idn/defaultDif10/index.html) Completeness Missing values for instrument/platform. Correct DIF-10 values for “project name” Consistency Multiple DIF-10 GCMD platforms/instruments keywords appear, required explicit keywords relation Use of “GOME” (ERS-2) while actually GOME-2 is meant (METOP), METOP-AB instead of specifying METOP-A and/or METOP-B, etc…

Open Issues with DIF-10 NASA DIF-10 Validation (https://gcmd.nasa.gov/qaviewer/QAViewer.html) ISO MIME type “application/xml” valid instead of “application/vnd.iso.19139-2+xml”, recommended by CEOS OpenSearch Best Practice 1.2 [CEOS-BP-012C] NASA extended MIME Type Invalid Keyword Relation Issue with relationship between Platform and Instrument keywords. Error message even with DIF-10 files were OK in the past (e.g., 20 OADS ESA files). GCMD vocabularies do not provide skos attributes to link platform and instrument concepts. In some (rare) cases, the skos:definition contains some formatted text (see ERS-1 Example) which refers to an instrument or platform as text. Even in this case, the GCMD UUID (the only thing which is not ambiguous) is not mentioned. NASA confirmed GCMD vocabularies do not provide platform/instrument relationship. Keyword Management Server (KMS) to be updated to check and manage consistently CEOS platform/instrument relationship , by 2019.

Open Issues with DIF-10 Consistency between DIF-10 Writer Page wrt Validation Tools (DocBuilder / IDN DIF-10) and (QAViewer / CMR validation API). DocBuilder defines Required, Highly Recommended and Recommended fields, but there is some gray area on the subfields. GCMD DocBuilder GCMD DIF Guide IDN DIF10 Guide E.g.: According to IDN guide, Platform field is mandatory, nowhere is specified if “Short Name” and “Long Name” subfields are mandatory. According to DocBuilder, and NASA support, it is assumed that Platform is mandatory, Short name mandatory and Long Name optional (also due to missing values from GCMD vocabularies, e.g., Sentinel-1 example). A clear table listing DIF-10 fields and subfields, pointing to authorized values (e.g., GCMD vocabularies), defining related cardinality (Optional/Mandatory/Sinlge/Multiple values), consistent with validation SW is required for unsupervised DIF-10 production (e.g., FedEO)

DIF-10 Next Steps ESA reports to NASA about understanding of mandatory/optional DIF-10 fields, and identified inconsistencies between DIF-10 Writer Guide and DIF-10 validators FedEO Ingestion tool shall be enhanced (Q1 2019) to generate a log file showing mapping information and let it generate a human readable “Ingestion report”. This will simplify the internal process of metadata owner to make the metadata IDN ready, passing through FedEO. Proceed with systematic European Partner metadata ingestion into FedEO and automatic export into IDN

Outline Introduction Activities & Evolution Metrics Software refactoring ESA Catalogue TTO within Common Service IDN metadata population WGISS Data Asset Metrics

CEOS Connected Data Asset Scenario 1 – M2M Interface APIs allows two steps search to external clients aligned to CEOS OpenSearch BP 1.2 Scenario 2 – GUI Interface CEOS Connected Data Access allow the users to discover/access both collections and products metadata

Scenario 2 – GUI Interface ESA Collection Landing Page Second Step OSDD FedEO Client ISO 19139-2 Metadata

Outline Introduction Activities & Evolution Metrics Software refactoring ESA Catalogue TTO within Common Service IDN metadata population WGISS Data Asset Metrics

Metrics See slide 20 * TotalResults is not returned by catalog. ** Step 2 under construction. See slide 20