A. Della Vecchia, D. Guerrucci, M. Albani (ESA) Federated Earth Observation (FedEO) A. Della Vecchia, D. Guerrucci, M. Albani (ESA) Yves Coene (Spacebel) CEOS WGISS#47 30th April 2019

Outline FedEO ESA PDGS Collaborative Environment SLT Activities Software refactoring IDN metadata population ESA PDGS Collaborative Environment SLT Activities Next Steps

Outline FedEO ESA PDGS Collaborative Environment SLT Activities Software refactoring IDN metadata population ESA PDGS Collaborative Environment SLT Activities Next Steps

FedEO: Federated Earth Observation Gateway System FedEO = Federated Earth Observation missions access The FedEO system provides a unique entry point to a growing number of scientific catalogues and services.

WGISS Connected Data Assets

Outline FedEO ESA PDGS Collaborative Environment SLT Activities Software refactoring IDN metadata population ESA PDGS Collaborative Environment SLT Activities Next Steps

Software Refactoring – Objectives Porting all FedEO components to Docker and Kubernetes – fast and easy deployment and horizontal scalability Optimization of the gateway and the catalog - quicker time response (3x up to 10x faster wrt concurrent users) Optimization of the dataset metadata ingestion job – faster ingestion time (+400% up to 40M entries) Preserving all functional/interoperability requirements

FedEO Architecure FedEO TTO by June 2019, after ESA EO Catalogue TTO. They share same SW with different configuration

Outline FedEO ESA PDGS Collaborative Environment SLT Activities Software refactoring IDN metadata population ESA PDGS Collaborative Environment SLT Activities Next Steps

FedEO Metadata Mediator Automatic Procedure – Available on FedEO reference environment, on Operational environment at ESA by June 2019 Partner Metadata repository Metadata Import Harvester tool FedEO Collection Catalogue FedEO Gateway ISO to DIF-10 Metadata Mediator Metadata Preparation IDN Complementary Information gcmd keyword Metadata Preparation IDN guideline for Information Content completeness and consistency (HERE) ESA Thesauri Service DIF-10 Validator (HERE) DIF-10 Validation Metadata Export DIF-10 Encoding GCMD keyword update, Multiplicity of DIF-10.2 metadata fields and APIs for metadata validation, provided by IDN NASA team (Michael Morhan) IDN repository

DIF-10 Metadata Export To updated with ECV metadata New FedEO Metric web page provides info about collections, granule, reporting about DIF-10 metadata conversion and DIF-10 metadata ready for ingestion into IDN  http://geo.spacebel.be/opensearch/asset/index.html FedEO Metric web page source code is a customization of the CWIC Metric Web Page https://cwic.wgiss.ceos.org/asset/index.html. Source Code kindly made available by CWIC team (Li Lin) To updated with ECV metadata

Open Issues / Next Steps ESA FedEO shall be connected to ESA EO Catalogue after its TTO, by June 2019. All ~200 collections shall be visible and >35M products shall be accessible (further products shall be integrated all along 2019) DLR FedEO currently adopted as translator between DLR and IDN. DLR is preparing all its collection for being exported into IDN, asking to update ESA and GCMD thesauri. DLR shall ask NASA to remove all the old/deprecated collections currently on IDN EUMETSAT A subset of EUMETSAT collections can be ingested passing through FedEO Misalignment between EUMETSAT platform values and GCMD thesauri (e.g., “METOP” assuming both “METOP A” and “METOP B”) VITO/JAXA All available collection correctly encoded in DIF-10 ROSCOSMOS Waiting for formal agreement CCI / CMEMS Waiting for feedbacks GCMD Thesauri update, and synchronization with ESA Thesauri continuously required Yves: could you please list open issues

Open Issues / Next Steps ECV Action from the 10th Meeting of the CEOS/CGMS Working Group on Climate 21st March 2019: To export ESA-ECV metadata from http://climatemonitoring.info/ecvinventory/ to IDN passing through FedEO. ECV collections cab be downloaded from here http://climatemonitoring.info/wp-content/uploads/2019/01/ECV_Inventory_v2.0.xlsx The ECV 2.0 is currently under review, under migration toward release 3.0 Cross check DIF-10 metadata cardinality vs ECV 2.0. Identify mandatory IDN metadata not managed by ECV 2.0 Reporting the results to WGClimate, asking for their integration within ECV 3.0 release Verify, and quantify effort accordingly, to ingest ECV metadata into FedEO to prepare DIF-10   Status: First prototype version of the “ECV” information available on http://geo.spacebel.be/opensearch/asset/index.html To do: Report to ECV team (http://geo.spacebel.be/opensearch/dif10/ingestion-ECV.html) Manage mappings for platforms/instruments/organization names. Names used in the ECV Excel (e.g., NASA, CPOM, NASA JPL) are not identical or missing in the GCMD thesauri Manage missing platforms/instrument relations in the thesauri Manage abstract info, concatenating concatenate some columns to have some content in the “ISO abstract”, and prefix it with the title of the column, as ECV Excel does not contain a column with a proper “abstract” describing the collection.

Outline FedEO ESA PDGS Collaborative Environment SLT Activities Software refactoring IDN metadata population ESA PDGS Collaborative Environment SLT Activities Next Steps

ESA PDGS Collaborative Environment To provide the ESA PDGS with a set of interoperable and federated services permitting the users to: Access to missions/platforms information supported by a common ontology Discovery and, if possible, direct access to EO and in-situ data: Copernicus Missions (e.g. Sentinels) Third Party Missions - TPMs (e.g. SPOT, Landsat, …) Heritage Missions - HMs (e.g. ERS-1/2, ENVISAT, …) Earth Explorer – EEs (e.g. SMOS, Cryosat, SWARM, …) International repositories (e.g., NASA CMR, CEOS IDN) Discovery and access to exploitation services (e.g. datacube): Browse/visualization tools for time series analysis EO data extraction, resampling and reprojection Hosted Processing for authorised users/communities (e.g. CAL/VAL)

Conceptual Model ESA EO Catalogue Web User Interfaces Operational interface ESA External Query and service requests Web User Interface for pixel-based access, analysis and processing Web User Interface for granule-based discovery and/or retrieval Mission oriented GUI - SWARM - Cryosat - SMOS - External Multi Mission GUI: – ESA EOCAT ESA EO Information Page: - ESA EO Gateway International EO Information Page: - CEOS IDN ………. ESA PDGS Data Cube GUI, VIRES GUI, etc… External catalogue application Conceptual Model Data access interface Operational Data Access Web User Interfaces Standard services request Interfaces (e.g., restful API) Protocol for data access (e.g., OGC, Open Datacube) OpenSearch (collections and granules) (collections and granules) Products Access Interface ESA EO Catalogue Centralised Knowledge-Base Facility Knowledge-Base Facility EO Data & Service Metadata Repository Thesauri Repository Standardised Interoperable Interfaces Metadata management Data Repositories Data Repositories Standardised Interoperable Services Interfaces IT Resources provided as services (IaaS, PaaS, SaaS) VM – Browse Images Generation VM – DataCube Engine/API VM – Processors User Applications Storage Temporary ESA User Applications Hosted Processing ESA Missions Data Repositories External Web Interfaces Copernicus Data Repositories External Web Interface TPMs HMs EEs S-1 S-2 S-3 S-5

ESA Catalogue TTO – Results ESA Catalogue shall: be the centralised metadata repository of ESA Collaborative Environment reuse same FedEO SW manage collections Digital Object Identifiers (DOIs) be part of CEOS WGISS Data Asset via FedEO

Outline FedEO ESA PDGS Collaborative Environment SLT Activities Software refactoring IDN metadata population ESA PDGS Collaborative Environment SLT Activities Next Steps

SLT Activities Participation in the review and issues of the following CEOS WGISS documents: WGISS Connected Data Assets Client Partner Guide (OpenSearch), including IDN/CWIC/FedEO Asset WGISS Connected Data Assets FedEO Data Partner Guide (OpenSearch) WGISS Connected Data Assets Error Handling Guide http://ceos.org/ourwork/lessons-learned-best-practices/

WGISS Action WGISS-46-19: WGISS CDA SLT to discuss way forward to implement a single front-end/portal within the IDN to access the WGISS CDA for data discovery and access. M2M Interfaces APIs allows two steps search to external clients aligned to CEOS OpenSearch BP 1.2 GUI Interface The user can be redirected from IDN to CWIC and/or to FedEO client, via DIF-10 metadata links IDN client able to manage two step search (overlap with https://search.earthdata.nasa.gov/ ??)

Scenario 2 – GUI Interface ESA Collection Landing Page Second Step OSDD FedEO Client ISO 19139-2 Metadata

Outline FedEO ESA PDGS Collaborative Environment SLT Activities Software refactoring IDN metadata population ESA PDGS Collaborative Environment SLT Activities Next Steps

Synergy with OGC initiatives Requested to OGC to approve that the related document package enters the so-called “public review” phase (3 weeks). Public comments period was closed in August 2018 (OGC call for public review). All public comments addressed that were received (September 2018-March 2019) and document package resolving comments redelivered to OGC TCC/OAB: OGC 13-026r9: 6 comments OGC 17-003: 24 comments (incl. 18 from DLR – related to JSON-LD Linked Data encoding) OGC 17-047: 6 comments (incl. 4 from VITO). All comments received were processed by the SWG and dispositions agreed with authors of the comments. Presentation to OGC TC (Web conference) scheduled for 30 April 2019 (15h CET) to allow starting final TC Vote for adoption as standard immediately after. OGC 17-084: Most recent update published on SWG Web pages (6 March 2019). Work to resume shortly.

FedEO Evolution ESA GSTP Project “ESE-ERGO” (Sep ‘19 – Mar ‘21) aims to: Adoption of OpenAPI, also encouraged by OGC, as standard interface for its services, in combination with JSON and JSON-LD Analyse and adopt new international guidelines and standards to improve gateway/catalogue functionalities, performance and interoperability. This includes: Emerging GeoDCAT-AP specification to publish EO metadata OGC 13-026r9 to introduce enhanced functionalities (e.g., flexible string search, results ranking, links to EO external services) OGC 17-047 to provide a JSON based encoding response from OpenSearch interface (e.g., quicker catalogue response, ease User interface development) Transition to an internal metadata model of which other metadata models are views, to allow more flexibility and to align to similar approaches applied by NASA (e.g., Unified Metadata Model – UMM/-C/-G/-S/-JSON). This includes: Recent OGC guidelines, in cooperation with CEOS/NASA, about collection and product metadata model based on GEO- JSON(-LD) encoding (e.g., OGC 17-084, OGC 17-003) Generalization of the concept of catalogue entry in order to consider granules (/products), collections (aggregation of granules/products) and services related to both of them (e.g., remote processing, datacube, etc…) seamlessly (e.g., GeoDCAT-AP, ISO 19115-1 & ISO 19115-3, etc…) Web-based metadata management tool to allow authorised users to add/modify collection catalogue entries