1. EPOS e-Infrastructure
Daniele Bailo, and the EPOS IT team
EPOS Kick-off meeting – October 5, Rome

2. What is an e-Infrastructure?
Short (and ambiguous) answer: an infrastructure with some “electronic” e-
e-road
e-dam
e-railway
e-airport

3. e-Infrastructure for research
Interworking of:*
people and organisational structures
digitally-based technology(hardware and software)
resources (data, services, digital libraries,HPC)
communications (protocols,access rights,network)
GOAL: support modern, internationally leading collaborative research
* [Research Councils UK (RCUK)]

4. EPOS e-Infrastructure
people and organisational structures

5. EPOS e-Infrastructure
resources (data, services, digital libraries)
244 Research Infrastructures
138 Institutions
22 countries
2272 GPS receivers
4939 seismic stations
464 TB Seismic data
1.095 PB Storage capacity (seismology)
1.240 PB Storage capacity (GNSS DC)
828 instruments in 118 Laboratories

6. e-Infrastructure for research
What about:
digitally-based technology (hardware and softw.)
communications (protocols,access rights,network) ?

7. EPOS ARCHITECTURE
ICS and TCS

8. Architecture Integrated Core Services (ICS) Interoperability layer
Thematic Core Services
(TCS)

9. Principles – 1. Interoperability
System A  System B communication
By means of common language (protocol)
and interface to “translate”
Protocol
System A
System B
Interface
Interface
Service A
Service B
System + interface= service
[Newman, Sam.”Building Microservices”,O'ReillyMedia,Inc.,2015]

10. Principles – 2. metadata
Metadata describes an object without actually accessing it
Discovery
Contextualization
Detailed processing *
PROESSING INCLUDES SCHEMA INFORMATION TO CONNECT SOFTWARE TO DATA
[RDA: the importance of metadata, Keith G Jeffery, Rebecca Koskela]

11. Integrated Core Services - ICS -

12. ICS - Integrated Core Services ICS-C and ICS-D
ICS-Central Hub
Orchestrates external resources (ICS-D)
EPOS member states competition for hosting it
ICS-Distributed services
Provide IT resources not built by EPOS ICS-C
Computational resources
Visualization resources
Others…

13. Interoperability Layer: ICS-APIs (wrappers)
ICS-C
EPOS GUI
EPOS APIs
ICS-D
METADATA
catalogue
System Managing Software
IAAA system
Workflow Engine
Interoperability Layer: ICS-APIs (wrappers)
TCSs
TCS API
TCS API
Data
set
Data
products
Services
Software
Data
set
Data
products
Services
Software
National
RIs
FR Seismic networks
FR Seismic networks
FR Seismic networks
FR Seismic networks
FR Seismic networks
FR Seismic networks
FR Seismic networks
FR Seismic networks
National RIs
National RIs
National RIs
National RIs

14. ICS-C Generic Architecture
Authentication and Authorization Layer (AAI)
GUI Client
API
METADATA
catalogue
Accounting System
System managing software
Workflow Engine
Resource
connector
ICS-D Connector
TCS Connector

15. ICS-C Generic Architecture
Authentication and Authorization Layer (AAI)
GUI Client
API
METADATA
catalogue
Accounting System
System managing software
Workflow Engine
Resource
connector
ICS-D Connector
TCS Connector

16. System Managing Software
It’s the software we have to develop.
Bundle of software/technologies, e.g.
Interfaces to other ICS modules:
Integration with 3rd party tools for:
caching tools (rabbitMQ)
workflow tools
Integrate and orchestrates all ICS components

17. ICS-C Generic Architecture
Authentication and Authorization Layer (AAI)
GUI Client
API
METADATA
catalogue
Accounting System
System managing software
Workflow Engine
Resource
connector
ICS-D Connector
TCS Connector

18. Metadata-Elements of EPOS ecosystem
Researchers, research managers, innovators, media
User Model
User interaction with the system: AAAI, interaction workflow, multlinguality
Processing Model
Describes functions of processing environment etc.
Data Model
Describes research data and ancillary info (instituions)
Resources: 1. data providers, 2. internal to ics, 3. other outside epos delivery framework.
We describe: instruments, data storage resources, data processing and visualization
Data model: 3 layers. Ongoing work: metadata collection, data taxonomy
Processing model: 1.description of the functions available in each processing environment, 2. protocols to connect to them, 3. algorthms which deal with the composing of the processing
User model: how a suer interacts with the system: iaaa, interaction workflow, multilinguality
Resource Model
Representing ICT resources
Complete ICT environment for research

19. Metadata model 3-layers Manage the complexity
CERIF (formal syntax, declared semantics)
Functions
-Discovery, selection
-Impact assessment
-Manage interoperability
Issues (selection of)
heterogeneity
Many standards to be mapped from TCS
Ontologies
Complex work
Web portal, Spatio-Temporal Search
Discovery (DC) and (CKAN, eGMS)
Generate
Contextual (CERIF metadata model)
Point to
Detailed (community specific)
Esempio carta identità
domain specific – data
(TCS or Naational Ris)

20. Contextual (CERIF metadata model)(
Common European Research Information Format

21. Metadata Q&A
Do TCSs need to adopt CERIF metadata standard? No, but provide “rich” metadata
Do TCSs need to copy their metadata into the ICS metadata catalogue? Short answer: we need translation Long answer  next slides

22. ICS-C Generic Architecture
Authentication and Authorization Layer (AAI)
GUI Client
API
METADATA
catalogue
Accounting System
System managing software
Workflow Engine
Resource
connector
ICS-D Connector
TCS Connector

23. AAAI (Authentication, Authorisation, Accounting Infrastructure)
Who (name) is accessing your data?
How is your data being used?
Can you measure the impact of your data on society? If yes, which is its value?
What if I download your data and re-sell it?
May I please download your data and replicate it on my personal commercial cloud?
How do you manage privacy?
Who is citing your data?

24. Great! Only one password to access to 7 systems!
AAAI layer
Purpose
Authenticate users
Authorize users
Transparent access to TCS and ICS-D data & services
Goal
Only 1 login – Delegation
Issues
Many system, sometimes non interoperable
EPOS-EGI-CC
AARC, EUDAT
AAAI
ICS-C
AAI
ICS-D
AAI
AAI
AAI
TCS 1
TCS 2
…TCS n

25. AAAI Q&A
Is my EduGAIN/X509/… AAAI system interoperable with ICS? EPOS ICS and other initiatives are working on AAAI interoperability
I’m starting from scratch, can ICS help? Of course. Let’s talk with your IT crowd*
Do I really need an AAAI? Do you really need a debit card? *

26. ICS-C Generic Architecture
Authentication and Authorization Layer (AAI)
GUI Client
API
METADATA
catalogue
Accounting System
System managing software
Workflow Engine
Resource
connector
ICS-D Connector
TCS Connector

27. Workflow engine Purpose
Workflow represent a process (Algorithm, service, computation), a composition of atomic functions
Tracks data transformation
Allows versioning and reproducibility
Generates provenance
Comments
Interoperability among various workflow engines
VERCE contribution
Rapid validation and monitoring

28. Workflow & provenance example
Two years ago I analyzed 12 of those 341 “*.dat” files in my ./temp/specs/ folder…
But ..hey.. I don’t remember WHICH files and HOW I PROCESSED THEM…
…and made an image for my paper….

29. ICS-C Generic Architecture
Authentication and Authorization Layer (AAI)
GUI Client
API
METADATA
catalogue
Accounting System
System managing software
Workflow Engine
Resource
connector
ICS-D Connector
TCS Connector

30. EPOS Interface GUI & API
Human interface (GUI)
Website or portal
Machine interface
API or Web service
which execute scripts or queries
Returns data/metadata in a given standard
Esempio carta identità

31. “EaaS” EPOS as a Service
Metadata & services available:
RDF export (“old” ENVRI)
OAI-PMH, CKAN, open-search… (EUDAT)
other standards
CERIF XML APIs
Other CRIS
EPOS GUI
EPOS APIs
RDF
OAI-PMH
EPOS APIs

32. ICS-C Generic Architecture
Authentication and Authorization Layer (AAI)
GUI Client
API
METADATA
catalogue
Accounting System
System managing software
Workflow Engine
Resource
connector
ICS-D Connector
TCS Connector

33. Accounting system Purpose Keeps track of
Used resources (data, processing)
Users actions
Goal
Load balance
Logging
Statistics
Impact analysis
Tailor GUI to user
Uses metadata catalogue
May I know which resources were accessed by a user and when?
Which is the most downloaded data?
Most searched data?
Is your TCS available or too may users connected?

34. ICS-C Generic Architecture
Authentication and Authorization Layer (AAI)
GUI Client
API
METADATA
catalogue
Accounting System
System managing software
Workflow Engine
Resource
connector
ICS-D Connector
TCS Connector

35. Connectors
Software Interfaces to RESOURCES ( TCS or ICS-D)
EPOS ICS
Resource
connector
ICS-D Connector
TCS Connector
Service API
Service API
Service API
Service API
……..
Service (ICS-D)
Service (ICS-D)
Service (TCS)
Service (TCS)

36. Thematic Core Services - TCS -

37. Time for QUIZ Which are the main ingredients of TCS architecture?
New building and new hardware
New furniture for my office
Money
System managing the data; metadata; web services
Schema describing the data management plan

38. TCS generic architecture
D) System managing the data; metadata; web services
National Layer Research Infrastructures: these can vary in different countries within a specific scientific community. They represent the existing DDSS.
TCS system: this represents the e-Infrastructure for a specific scientific community. It may include the software used to federate National RIs, or the software to present results on the web (web portal).
Metadata Catalogue: this is usually a database where each data object (e.g. file, in case of non-streamed data) is referenced and described by the metadata. It can be used to drive the Web Services

39. TCS Checklist DDSS / Assets Metadata describing them (standard)
Web services (API) exposing them
ICS-TCS common work on metadata
data policies implemented (AAAI)

40. TCS Metadata principles
Metadata used for
Data / Dataproducts, Software, Services, Resources, Users
Approved (e.g. OSI,OGC, INSPIRE…) or de facto standard
Intra community common strategy
Metadata already used by other communities?
Common developments?
community data (e.g. seismic waveforms, GPS time-series, geological maps),
software (e.g. an analysis or visualisation application),
services (e.g. use of specialist equipment),
resources (e.g. computers, instrumentation, detectors) and
users (with their roles, responsibilities and authorities used for AAI).

41. TCS Web services / APIs principles
Each type of Data delivered (and its associated metadata) should be accessible via web services and/or APIs
Based on approved (e.g. OSI) or de facto standards
Main purposes
Discovery (metadata)
Data access and retrieval (e.g. download)
Data analysis / visualization / processing

42. ICS-TCS common work Metadata
Definition of community, standard-based metadata
APIs
Definition of APIs for data discovery and for data access/retrieval
Match & Map:
Match/Mapping of community metadata within the Metadata Catalogue

43. Metadata match and map
Process of mapping Metadata elements to the canonical CERIF
ICS –TCS common work in selecting discovery and contextual metadata element (e.g. lat, lon, author,etc)

44. Metadata & interoperability
Do TCSs need to copy their metadata into the ICS metadata catalogue?
Metadata dump: Metadata is copied. Requires polling, many entries in ICS catalogue, TCS low “burden”
Metadata Runtime access: MD accessed by querying web services. No polling, avoids errors, needs reliable TCS webservices
Match / Mapping is required for both options

45. TCS-ICS Open issues

46. Computational Earth Sciences (CES)
Not reinvent the wheel principle: re-use existing software / available resources
TCS contribution: mapped in the assets/requirement collection phase
ICS will set up a bid procedure for CES

47. Identifiers (PID / UID):
encourage their adoption by TCS to
Unique and reliable reference to Digital Object
Reproducibility of data & citation
Interlinked with other initiatives (EUDAT, RDA)
Ongoing discussion: ICS-C will provide further recommendations

48. AAAI

49. “the ITs”
We support you!

50. ICS STAFF - WP6&7 Needed urgently BGS, KNMI, INGV, UiB, GFZ WP Chairs:
Keith Kuvvet Matt
Institutions:
BGS, KNMI, INGV, UiB, GFZ
Task forces:
Architecture Graham
PID / UID Martin
Workflow James
AAAI Mariusz
Documentation
IT TCS CONTACTS
Needed urgently

51. Thank You Epos Social www.epos-eu.org WebSite Newsletter
R.I.D.E.
Epos Social

52. Common questions
Do ICS store the data? No but for technical purposes. Data is stored & curated by users.
Do ICS assign PIDs? In principle no. Ongoing discussion
I feel lost: where should I start from? Define metadata for your data