INTAROS WP5 Data integration and management iAOS Service integration tools and process (task 5.5) Hervé Caumont, Terradue INTAROS Project Kickoff, January 12th 2017, Bergen

Introduction the iAOS concept

INTAROS WP5 main objectives: ➽ Integrate multidisciplinary and distributed data repositories into a scalable and resilient Pan-Arctic observing system, iAOS ➽ Offer seamless access to observations and derived parameters, provide a set of tools for data analysis, transformation and visualization of spatio-temporal datasets Collect Ingest Store Search Analyse Visualise ➽ build on present observing systems developed over several years and operated with funding from countries and international agencies to ensure that the iAOS is maintained as a platform for a SAOS ➽ leverage on existing data repositories for storage and curation of new data collected during INTAROS, and provide seamless access to distributed data repositories through a common ICT platform with embedded processing and analysis tools How to achieve the objectives...

INTAROS WP5 specific objectives: Deploy underlying infrastructure for the Integrated Arctic Observing System (iAOS) and operate associated tools for IT management and user support Provide a framework to ease the discovery and retrieval of dataset collections from existing spatial data infrastructures Develop new geo-statistical methods for interpolation of spatiotemporal datasets, and provide a set of tools for data analysis, transformation and visualization Process new observations from WP2-4, and store the generated datasets in an IAOS enabled repository Search icon: https://www.iconfinder.com/icons/314689/search_icon#size=128 Map layer: https://www.iconfinder.com/icons/64437/layers_map_icon#size=32

iAOS - Conceptual architecture

Service Implementation INTAROS WP5 service integration approach: Service Design Service Implementation Service Testing Service Deployment Service Validation Task 5.5 Integration of new processing services: exploit the data processing tools and geo-statistical algorithms in the Cloud Sandboxes & Operations environments support the full lifecycle of the integration of new processing services offer an environment where scientists have simultaneous access to data , tools and processing resources allow to implement services within shared Platform as a Service (PaaS) Selected data processing services will be developed and made available to demonstrate the iAOS platform Task 5.6 iAOS portal development: outreach to stakeholders in the INTAROS WP6

Service integration the iAOS tools and process

iAOS technical overview...

...and our focus (where Service integration occurs)

iAOS - Service integration Purpose of Terradue Cloud Platform A workplace & APIs to connect Data Providers, Developers, Integrators and Producers Key actors: Data Providers Developers Integrators Producers Toolboxes Apps Developers Integrators Services Appliances Data providers Producers

iAOS - Service integration in 4 steps Applications lifecycle on Terradue Cloud Platform Process Cloud Sandbox Environment Design The data samples Python, R, Dev. Toolboxes, ... Integrate Test Validate The laboratory bench Service #1 Cloud Operations Environment The factory chain Service #2 Deploy Scale The recipes … Process Service #N The goods shipment

Service integration (1/4) Application design & software libraries management Processing libs compiled from any language supported on CentOS 6.5 C/C++, Java, R, Python (other upon verification) Runtimes: Matlab and IDL Referenced on the Platform Repos (Artifactory, GitHub, Epel…) and accessed via standard software packages management tools yum, conda, npm, mvn... Wrapped in an application archetype for tests with representative data chunks (samples) “MapReduce streaming” simple workflow (DAG) archetype provides with natively scalable services

Service integration (2/4) Application tests and validation on the Cloud Sandbox Service developers use the Cloud Sandbox tools to implement, test and validate a scalable workflow A complete laboratory bench for service developers Cloud native & cost effective work environment The Cloud Sandbox software framework includes tools to proof test data discovery, parallel processing and access to results Cloud Sandbox dashboard job wrapper functions (copy, getParam, publish,...) data management (OGC ® OpenSearch extensions) service invocation (OGC ® Web Processing Service)

Service integration (3/4) Application deployment and scaling on Cloud Operations environment After validation, the developer publishes the Service for integration on a selected Cloud Operations infrastructure (e.g. EGI) A ‘Processor as a Service’ accesses complete pools of datasets allows application intensive computing stores processor outputs, e.g. back to the lab

Service integration (4/4) Production & operations management Scaler productions: automated, data-driven production workflows Processing Services are operated on a selected Cloud provider Operators manage the service via their user account on Terradue Cloud Platform Resources can scale up/down The development environment remains active for maintenance activities on the Service Service End Users can access the service’s products from ad-hoc community portals Includes support for standard protocols like OGC Web Services

iAOS services integration for data processing

WP5 iAOS services integration goals sea-ice geo-statistics Sea-ice data generation Time series of satellite EO-derived parameters (e.g. ice concentration, ice thickness) for selected regions Support to sea-ice information combined with: weather and wave forecasts passive acoustic data Dynamic ice and ocean processes Support to Cal-Val analysis of Sentinel- derived sea-ice thickness Arctic Risk Map WP5 iAOS services integration goals

(Source: M. Babiker, NERSC) Processing Service 1: Time series of sea ice parameters NERSC Sentinel-1 SAR provides frequent coverage of Arctic regions Offering high-resolution imagery for detailed sea ice mapping and statistics Stakeholders: Science Operators Shipping Fisheries Tourism Authorities S-1A SAR HH Sea Ice Type Open water & thin ice Young ice Grey ice First year ice Land (Source: M. Babiker, NERSC)

Acoustic data from D mooring Processing Service 2: Acoustic impact assessment NERSC Combining acoustic data from FP7 ACOBAR and INTAROS with data from ship traffic (AIS), seismic surveys (airguns) Acoustic data from D mooring D Characterization of acoustic data Noise in the ocean is an indicator in the Marine Strategy Framework Directive

Cloud services front ends iAOS services integration - examples of user front ends

Cloud Services http://www.terradue.com

Cloud Services Examples of user front ends

Cloud Services Examples of user front ends

Cloud Services Examples of user front ends

iAOS - Service integration benefits for Platform users Innovation & automation Setup Workflow Validate App Develop Jobs Deploy Service Design Service Monitor Service Cloud Sandbox Environment Cloud Operations Environment

herve.caumont@terradue.com Platform Operations, Terradue Questions ? herve.caumont@terradue.com Platform Operations, Terradue

Cesare Rossi, Hervé Caumont Demo Session Cesare Rossi, Hervé Caumont Operations Support team at Terradue

Demo Session Overview Session 1 Working with Cloud resources on Terradue Cloud Platform How are users able to self-register, get their access credentials to Cloud resources, create a work environment with state of the art tools (e.g. Git and GitHub, Maven, Oozie workflows) and work collaboratively for the development of powerful Apps ? The session will showcase App creation for EO data processing on the Cloud. Session 2 Working with Data resources on Terradue Cloud Platform What are the Platform connectors to open data sources, along with integrated tools for data staging within Cloud Apps, that deliver a unique open data access experience to developers on the Platform ? The session will showcase Cloud service configuration and deployment to enable Apps with live connectivity to open data.

Visit https://gist.github.com/HerveCaumont Get the instructions Visit https://gist.github.com/HerveCaumont

Choose your language

The Sen2Cor processing tool Sen2Cor is a processor for Sentinel-2 Level 2A product generation and formatting; it performs the atmospheric, terrain and cirrus correction of Top-Of- Atmosphere Level 1C input data. Sen2Cor creates Bottom-Of-Atmosphere, optionally terrain and cirrus corrected reflectance images; additional, Aerosol Optical Thickness, Water Vapor, Scene Classification Maps and Quality Indicators for cloud and snow probabilities. http://step.esa.int/main/third-party-plugins-2/sen2cor/ https://anaconda.org/Terradue/sen2cor

Visit https://gist.github.com/HerveCaumont Get the instructions (Session 2) Visit https://gist.github.com/HerveCaumont