DI4R, 30th September 2016, Krakow Collection and Analysis of Ocean Big Data: Building the EMSODEV Data Management Platform using EGI Federated Cloud DI4R, 30th September 2016, Krakow Pasquale Andriani pasquale.andriani@eng.it

..about EMSODEV

EMSODEV scenario DATA MANAGEMENT PLATFORM Data ingestion Data access Real-time Asynch. Data ingestion Ingestion speed Data access DMP tools

DMP Design 1/2 ENVRI Reference Model v2.0 Data acquisition Data curation Data publishing Data processing Data use Computational Viewpoint (CV) has been used to identify a standard set of components (CV Objects) and interfaces that inspired the design of the EMSODEV DMP architecture in different phases

EMSODEV DATA MANAGEMENT PLATFORM DMP Design 2/2 EMSODEV DATA MANAGEMENT PLATFORM <<external resource>> EMSO Regional Nodes Data Files <<experimental lab>> EMSODEV API <<security service>> Authentication &Authorization Tool <<virtual lab>> DMP Tools data use <<experimental lab>> Data Analysis Tool <<instrument controller>> Sensor Observation Service data acquisition <<data transfer service>> Transfer Flow Orchestrator <<raw data collector>> Push Transfer Flow Pull Transfer Flow <<data store controller>> NoSQL DBs Streaming Store Controller Distributed File System Time Series DB <<catalogue service>> Metadata and Service Repository <<data exporter>> Dataset Exporter <<data importer>> Processing Results Importer Regional Node Importer <<data stager>> Stager Engine data curation <<process controller>> Batch Processor Engine Streaming Processor Engine <<coordination service>> Analysis Manager data processing Mapping of ENVRI CV Objects to EMSODEV DMP architectural components Instrument Controller in the Data Acquistion <<data broker>> Broker Engine data publishing

DMP infra. on EGI FedCloud EMSODEV DMP current prototype Test VO: fedcloud.egi.eu Cloud Compute: 8 VMs (8 CPUs + 16GB RAM + 40GB HD) EMSODEV DMP (requested SLA request in early August 2016) Production VO: vo.emsodev.eu Cloud Compute: ~10 VMs (8 CPUs + 16GB RAM + 40GB HD) File Storage: 5 TB

DMP Operation – Apache Ambari Dashboard for provisioning, managing, monitoring and securing the EMSODEV cluster hosting the EMSODEV DMP.

Data Acquisition and Curation At this stage, two raw data collectors exist: A Pull Transfer Flow: data is retrieved via API exposed by an OGC SOS server available at the OBSEA observatory located in Vilanova and managed by Universitat Politecnica De Catalunya. SOS server API GetCapabilities EMSODEV DATA MANAGEMENT PLATFORM GetObservation OBSEA data DescribeSensor A Push Transfer Flow: data is sent to a DMP service which “listens” to near-real time updates on XML files describing sensors data and measurements.

Data Publishing and Use Real-time dashboard solution for time-series data analysis After the storing phase, data is visualized by using a real-time dashboard, in particular we are testing elasticsearch with Grafana. The choice of this two tools has been made because elasticsearch allow Real-time data search.. . . . . And with Grafana is possible to have real-time summary and charting and both are Open source and under Apache 2 license that is one of the common licenze type in research project. Real-time data search Real-time advanced analytics Schema-free Real-time summary and charting Apache2 Open Source License Distributed, scalable, and highly available

DMP preliminary REST API Built and managed through: Swagger Editor Swagger UI Swagger CodeGen

Advantages of using EGI FedCloud A ready-to-use IaaS where to deploy on-demand IT services Easy VM and security management via OpenStack Horizon Scalable according to community needs (within the boundaries established through SLA) Secure VM access via a mechanism (VOMS credentials) based on proxy credentials issued and verified by EGI Fast and reliable support (ggus.eu trouble-ticketing and by mail)

Questions Pasquale Andriani Engineering Ingegneria Informatica SpA Italy pasquale.andriani@eng.it