Research Data Collections WG Plenary 9 Barcelona

Introduction: Goals and Deliverables Our goal is to facilitate: cross-collection interoperability development of common tools and services for sharing and expanding data collections across repositories and disciplines Our deliverables are: An abstract data model for data collections A Create/Read/Update/List (CRUD/L) API Existing solutions are many but do not offer a full set of generic (CRUD) operations Individual API implementations will be able to declaratively express, via a standard set of capabilities, the operations available for their collections.

Current Status: Completed and In Progress Definitions established and added to RDA DFT Terms Tool Data Model draft completed API Specification draft completed 4 implementations in development Initial datatypes registered in a DTR

API Specification

API: Service Features GET /features

API: Collections - Create/Read/Update/Delete/List GET /collections POST /collections GET /collections/{id} PUT /collections/{id} DELETE/collections/{id}

API: Collections - Set Level Operations Capabilities GET /collections/{id}/capabilities Match POST /collections/{id}/ops/findMatch Intersection GET /collections/{id}/ops/intersection/{otherid} Union GET /collections/{id}/ops/union/{otherid} Flatten GET /collections/{id}/ops/flatten

API: Collection Member - CRUD/L LIST CREATE READ UPDATE DELETE GET /collections/{id}/members POST /collections/{id}/members GET /collections/{id}/members{mid} PUT /collections/{id}/members/{mid} DELETE /collections/{id}/members/{mid}

API: Collection Member - Property Operations READ UPDATE DELETE GET /collections/{id}/members{mid}/properties/{property} PUT /collections/{id}/members/{mid}/properties/{property} DELETE /collections/{id}/members/{mid}/properties/{property}

Data Fabric: RDA Collections API, DTR and PIT Some Collections API properties (Service/Collection/Member) should use registered data types Collections API implementations may delegate PID creation to a PIT API

Data Fabric: RDA Collections in Broker-Driven Workflow

Data Fabric: Collection Types and DTR - Current Status Registered data types for Collection API Models Mapping of properties to formal types started

Implementations: REPTOR Reptor („Repository“) is a data repository which was developed along some RDA recommendations Reptor turns a standard web server into a data repository Implements DTR, DFT and Collection WG recommendations Example instance at: http://dft-rda.esc.rzg.mpg.de/reptor/ http://reptor.thomas-zastrow.de/ Collection API calls: curl -X GET http://dft-rda.esc.rzg.mpg.de/reptor/collections/api.php/collections curl -X GET http://dft-rda.esc.rzg.mpg.de/reptor/collections/api.php/collections/Photos/members ...

Implementations: Tufts (Python + LDP) Python/Flask implementation for Perseids Project at Department of Classics, Tufts University Main payload are linked data annotations Multiple data backends: Filesystem RDF/LDP MongoDB Shared programming interface and data model Demo endpoint: http://collections.perseids.org/ Docker image: http://hub.docker.com/fbaumgardt/rda-collections-api

Use Cases: GEOFON GEOFON (GFZ) is one of the biggest seismological data archives in Europe and member of ORFEUS/EPOS-Seismology. Context: Almost all seismological data centres expose an API for users to request data (an international standard). Only at GEOFON, we have more than 6 million successful requests/year, which are created “on-the-fly” (not predefined). It is not trivial to create a predefined dataset of scientific interest to be shared with users (e.g. seismic waveforms related to an earthquake). Also difficult to create huge datasets due to size limitations for requests from our services (f.i. all data from a station). We cannot afford to keep a copy of every request made in order for it to be frozen and (somehow) reproducible.

Use Cases: GEOFON Aim Be able to define datasets (collections), which can point to data files in our archive. Either via a PIDs or regular URLs. Create big datasets providing all the data related to a project or different (overlapping) subsets from it. Keep a copy of the request definitions made by users in order to be re-created if needed (e.g. if shared with another user). We want to be able to “reproduce” a dataset. Setup restrictions on access to some of the datasets. Provide a single (and simple) link for the collections, so that the users can download the whole collection with one click (an extension to the specification!). Support download resume (related to the previous point). Do it as standard as possible (f.i. within this RDA WG). Current Status The system is already implemented and being tested *today*. We expose around 6000 collections and more than 1.5 million members.

Use Cases: Perseids

Use Cases: DKRZ / Climate data management Needed: Common collection builder solution across multiple use cases, e.g.: ESGF/CMIP6 dataset PID generator + custom collections Climate data processing services - together with CMCC (pre/post Ophidia calls - ECAS) Copernicus Data Services Best option so far: Collection builder (client) library as a reusable open source component Extension of B2HANDLE/pyhandle or add-on library Back end: Handle System + “x”: storage of collection structure still open Front end: The use cases from above - but not all are Python-based (e.g. CMIP6 is Java) PID Kernel Info profile and backing by DTR to be included Offering as a B2HANDLE service for long-term adoption by (unknown) others server-side pathway for e.g. B2SHARE?

Use Cases: CAU Kiel/IGSN Kiel University is setting up the local IGSN registration service for samples in Kiel In collaboration with GEOMAR Helmholtz-Centre for Ocean Research - Central Sample Repository and the Kiel University it is aimed to create a central registration service for all samples A central allocating service will be available to clients from disciplines specific repositories The central service holds the sample birth certificate Collection information (?) Link to detailed record information An initial inventory of samples has been carried out at GEOMAR during the years 2014 to 2015 Various kinds of possible and helpful collection scenarios are already available

Use Cases: CAU Kiel/IGSN Lab1 Collection South Pacific Collection Seamount Collection Cruise1 Collection ... Cruise2 Collection Type1 Collection ... Type2 Collection Event Collection Event Collection Event Collection ... Storage Collection Section Half/liner ... Cruise2 Collection Event Collection Samples Samples Samples Core Geological age Collection

Use Cases: HTRC [ … Beth to Provide]

Next Steps Finalize Data Model and API Specification Incorporate feedback from this session and implementers Examine implications for existing collections solutions and repositories Write final Specification Document Include best practices for workflows and coordination with RDA DTR and PID Types APIs Include best practices for use URL-unfriendly URIs Follow up with existing and potential adopters