EPOS e-Infrastructure Daniele Bailo, and the EPOS IT team EPOS Kick-off meeting – October 5, 2015 - Rome

What is an e-Infrastructure? Short (and ambiguous) answer: an infrastructure with some “electronic” e- e-road e-dam e-railway e-airport

e-Infrastructure for research Interworking of:* people and organisational structures digitally-based technology(hardware and software) resources (data, services, digital libraries,HPC) communications (protocols,access rights,network) GOAL: support modern, internationally leading collaborative research * [Research Councils UK (RCUK)]

EPOS e-Infrastructure people and organisational structures

EPOS e-Infrastructure resources (data, services, digital libraries) 244 Research Infrastructures 138 Institutions 22 countries 2272 GPS receivers 4939 seismic stations 464 TB Seismic data 1.095 PB Storage capacity (seismology) 1.240 PB Storage capacity (GNSS DC) 828 instruments in 118 Laboratories

e-Infrastructure for research What about: digitally-based technology (hardware and softw.) communications (protocols,access rights,network) ?

EPOS ARCHITECTURE ICS and TCS

Architecture Integrated Core Services (ICS) Interoperability layer Thematic Core Services (TCS)

Principles – 1. Interoperability System A  System B communication By means of common language (protocol) and interface to “translate” Protocol System A System B Interface Interface Service A Service B System + interface= service [Newman, Sam.”Building Microservices”,O'ReillyMedia,Inc.,2015]

Principles – 2. metadata Metadata describes an object without actually accessing it Discovery Contextualization Detailed processing * PROESSING INCLUDES SCHEMA INFORMATION TO CONNECT SOFTWARE TO DATA [RDA: the importance of metadata, Keith G Jeffery, Rebecca Koskela]

Integrated Core Services - ICS -

ICS - Integrated Core Services ICS-C and ICS-D ICS-Central Hub Orchestrates external resources (ICS-D) EPOS member states competition for hosting it ICS-Distributed services Provide IT resources not built by EPOS ICS-C Computational resources Visualization resources Others…

Interoperability Layer: ICS-APIs (wrappers) ICS-C EPOS GUI EPOS APIs ICS-D METADATA catalogue System Managing Software IAAA system Workflow Engine Interoperability Layer: ICS-APIs (wrappers) TCSs TCS API TCS API Data set Data products Services Software Data set Data products Services Software National RIs FR Seismic networks FR Seismic networks FR Seismic networks FR Seismic networks FR Seismic networks FR Seismic networks FR Seismic networks FR Seismic networks National RIs National RIs National RIs National RIs

ICS-C Generic Architecture Authentication and Authorization Layer (AAI) GUI Client API METADATA catalogue Accounting System System managing software Workflow Engine Resource connector ICS-D Connector TCS Connector

ICS-C Generic Architecture Authentication and Authorization Layer (AAI) GUI Client API METADATA catalogue Accounting System System managing software Workflow Engine Resource connector ICS-D Connector TCS Connector

System Managing Software It’s the software we have to develop. Bundle of software/technologies, e.g. Interfaces to other ICS modules: Integration with 3rd party tools for: caching tools (rabbitMQ) workflow tools Integrate and orchestrates all ICS components

ICS-C Generic Architecture Authentication and Authorization Layer (AAI) GUI Client API METADATA catalogue Accounting System System managing software Workflow Engine Resource connector ICS-D Connector TCS Connector

Metadata-Elements of EPOS ecosystem Researchers, research managers, innovators, media User Model User interaction with the system: AAAI, interaction workflow, multlinguality Processing Model Describes functions of processing environment etc. Data Model Describes research data and ancillary info (instituions) Resources: 1. data providers, 2. internal to ics, 3. other outside epos delivery framework. We describe: instruments, data storage resources, data processing and visualization Data model: 3 layers. Ongoing work: metadata collection, data taxonomy Processing model: 1.description of the functions available in each processing environment, 2. protocols to connect to them, 3. algorthms which deal with the composing of the processing User model: how a suer interacts with the system: iaaa, interaction workflow, multilinguality Resource Model Representing ICT resources Complete ICT environment for research

Metadata model 3-layers Manage the complexity CERIF (formal syntax, declared semantics) Functions -Discovery, selection -Impact assessment -Manage interoperability Issues (selection of) heterogeneity Many standards to be mapped from TCS Ontologies Complex work Web portal, Spatio-Temporal Search Discovery (DC) and (CKAN, eGMS) Generate Contextual (CERIF metadata model) Point to Detailed (community specific) Esempio carta identità domain specific – data (TCS or Naational Ris)

Contextual (CERIF metadata model) (http://www.eurocris.org/) Common European Research Information Format

Metadata Q&A Do TCSs need to adopt CERIF metadata standard? No, but provide “rich” metadata Do TCSs need to copy their metadata into the ICS metadata catalogue? Short answer: we need translation Long answer  next slides

ICS-C Generic Architecture Authentication and Authorization Layer (AAI) GUI Client API METADATA catalogue Accounting System System managing software Workflow Engine Resource connector ICS-D Connector TCS Connector

AAAI (Authentication, Authorisation, Accounting Infrastructure) Who (name) is accessing your data? How is your data being used? Can you measure the impact of your data on society? If yes, which is its value? What if I download your data and re-sell it? May I please download your data and replicate it on my personal commercial cloud? How do you manage privacy? Who is citing your data?

Great! Only one password to access to 7 systems! AAAI layer Purpose Authenticate users Authorize users Transparent access to TCS and ICS-D data & services Goal Only 1 login – Delegation Issues Many system, sometimes non interoperable EPOS-EGI-CC AARC, EUDAT AAAI ICS-C AAI ICS-D AAI AAI AAI TCS 1 TCS 2 …TCS n

AAAI Q&A Is my EduGAIN/X509/… AAAI system interoperable with ICS? EPOS ICS and other initiatives are working on AAAI interoperability I’m starting from scratch, can ICS help? Of course. Let’s talk with your IT crowd* Do I really need an AAAI? Do you really need a debit card? *

ICS-C Generic Architecture Authentication and Authorization Layer (AAI) GUI Client API METADATA catalogue Accounting System System managing software Workflow Engine Resource connector ICS-D Connector TCS Connector

Workflow engine Purpose Workflow represent a process (Algorithm, service, computation), a composition of atomic functions Tracks data transformation Allows versioning and reproducibility Generates provenance Comments Interoperability among various workflow engines VERCE contribution Rapid validation and monitoring

Workflow & provenance example Two years ago I analyzed 12 of those 341 “*.dat” files in my ./temp/specs/ folder… But ..hey.. I don’t remember WHICH files and HOW I PROCESSED THEM… …and made an image for my paper….

ICS-C Generic Architecture Authentication and Authorization Layer (AAI) GUI Client API METADATA catalogue Accounting System System managing software Workflow Engine Resource connector ICS-D Connector TCS Connector

EPOS Interface GUI & API Human interface (GUI) Website or portal Machine interface API or Web service which execute scripts or queries Returns data/metadata in a given standard Esempio carta identità

“EaaS” EPOS as a Service Metadata & services available: RDF export (“old” ENVRI) OAI-PMH, CKAN, open-search… (EUDAT) other standards CERIF XML APIs Other CRIS EPOS GUI EPOS APIs RDF OAI-PMH EPOS APIs

ICS-C Generic Architecture Authentication and Authorization Layer (AAI) GUI Client API METADATA catalogue Accounting System System managing software Workflow Engine Resource connector ICS-D Connector TCS Connector

Accounting system Purpose Keeps track of Used resources (data, processing) Users actions Goal Load balance Logging Statistics Impact analysis Tailor GUI to user Uses metadata catalogue May I know which resources were accessed by a user and when? Which is the most downloaded data? Most searched data? Is your TCS available or too may users connected?

ICS-C Generic Architecture Authentication and Authorization Layer (AAI) GUI Client API METADATA catalogue Accounting System System managing software Workflow Engine Resource connector ICS-D Connector TCS Connector

Connectors Software Interfaces to RESOURCES ( TCS or ICS-D) EPOS ICS Resource connector ICS-D Connector TCS Connector Service API Service API Service API Service API …….. Service (ICS-D) Service (ICS-D) Service (TCS) Service (TCS)

Thematic Core Services - TCS -

Time for QUIZ Which are the main ingredients of TCS architecture? New building and new hardware New furniture for my office Money System managing the data; metadata; web services Schema describing the data management plan

TCS generic architecture D) System managing the data; metadata; web services National Layer Research Infrastructures: these can vary in different countries within a specific scientific community. They represent the existing DDSS. TCS system: this represents the e-Infrastructure for a specific scientific community. It may include the software used to federate National RIs, or the software to present results on the web (web portal). Metadata Catalogue: this is usually a database where each data object (e.g. file, in case of non-streamed data) is referenced and described by the metadata. It can be used to drive the Web Services

TCS Checklist DDSS / Assets Metadata describing them (standard) Web services (API) exposing them ICS-TCS common work on metadata data policies implemented (AAAI)

TCS Metadata principles Metadata used for Data / Dataproducts, Software, Services, Resources, Users Approved (e.g. OSI,OGC, INSPIRE…) or de facto standard Intra community common strategy Metadata already used by other communities? Common developments? community data (e.g. seismic waveforms, GPS time-series, geological maps), software (e.g. an analysis or visualisation application), services (e.g. use of specialist equipment), resources (e.g. computers, instrumentation, detectors) and users (with their roles, responsibilities and authorities used for AAI).

TCS Web services / APIs principles Each type of Data delivered (and its associated metadata) should be accessible via web services and/or APIs Based on approved (e.g. OSI) or de facto standards Main purposes Discovery (metadata) Data access and retrieval (e.g. download) Data analysis / visualization / processing

ICS-TCS common work Metadata Definition of community, standard-based metadata APIs Definition of APIs for data discovery and for data access/retrieval Match & Map: Match/Mapping of community metadata within the Metadata Catalogue

Metadata match and map Process of mapping Metadata elements to the canonical CERIF ICS –TCS common work in selecting discovery and contextual metadata element (e.g. lat, lon, author,etc)

Metadata & interoperability Do TCSs need to copy their metadata into the ICS metadata catalogue? Metadata dump: Metadata is copied. Requires polling, many entries in ICS catalogue, TCS low “burden” Metadata Runtime access: MD accessed by querying web services. No polling, avoids errors, needs reliable TCS webservices Match / Mapping is required for both options

TCS-ICS Open issues

Computational Earth Sciences (CES) Not reinvent the wheel principle: re-use existing software / available resources TCS contribution: mapped in the assets/requirement collection phase ICS will set up a bid procedure for CES

Identifiers (PID / UID): encourage their adoption by TCS to Unique and reliable reference to Digital Object Reproducibility of data & citation Interlinked with other initiatives (EUDAT, RDA) Ongoing discussion: ICS-C will provide further recommendations

AAAI

“the ITs” We support you!

ICS STAFF - WP6&7 Needed urgently BGS, KNMI, INGV, UiB, GFZ WP Chairs: Keith keith.jeffery@keithgjefferyconsultants.co.uk Kuvvet Kuvvet.Atakan@uib.no Matt mharr@bgs.ac.uk Institutions: BGS, KNMI, INGV, UiB, GFZ Task forces: Architecture Graham agsmith@bgs.ac.uk PID / UID Martin mln@bgs.ac.uk Workflow James jpass@bgs.ac.uk AAAI Mariusz m.sterzel@cyfronet.pl Documentation daniele.bailo@ingv.it IT TCS CONTACTS Needed urgently

Thank You Epos Social www.epos-eu.org WebSite Newsletter www.epos-eu.org/newsletter www.epos-eu.org/ride R.I.D.E. Epos Social

Common questions Do ICS store the data? No but for technical purposes. Data is stored & curated by users. Do ICS assign PIDs? In principle no. Ongoing discussion I feel lost: where should I start from? Define metadata for your data