1. INTAROS WP5 Data integration and management
iAOS Service integration tools and process
(task 5.5)
Hervé Caumont, Terradue
INTAROS Project Kickoff, January 12th 2017, Bergen

2. Introduction
the iAOS concept

3. INTAROS WP5 main objectives:
➽ Integrate multidisciplinary and distributed data repositories into a scalable and resilient Pan-Arctic observing system, iAOS
➽ Offer seamless access to observations and derived parameters, provide a set of tools for data analysis, transformation and visualization of spatio-temporal datasets
Collect
Ingest
Store
Search
Analyse
Visualise
➽ build on present observing systems developed over several years and operated with funding from countries and international agencies to ensure that the iAOS is maintained as a platform for a SAOS
➽ leverage on existing data repositories for storage and curation of new data collected during INTAROS, and provide seamless access to distributed data repositories through a common ICT platform with embedded processing and analysis tools
How to achieve the objectives...

4. INTAROS WP5 specific objectives:
Deploy underlying infrastructure for the Integrated Arctic Observing System (iAOS) and operate associated tools for IT management and user support
Provide a framework to ease the discovery and retrieval of dataset collections from existing spatial data infrastructures
Develop new geo-statistical methods for interpolation of spatiotemporal datasets, and provide a set of tools for data analysis, transformation and visualization
Process new observations from WP2-4, and store the
generated datasets in an IAOS enabled repository
Search icon:
Map layer:

5. iAOS - Conceptual architecture

6. Service Implementation
INTAROS WP5 service integration approach:
Service Design
Service Implementation
Service Testing
Service Deployment
Service Validation
Task 5.5 Integration of new processing services:
exploit the data processing tools and geo-statistical algorithms in the Cloud Sandboxes & Operations environments
support the full lifecycle of the integration of new processing services
offer an environment where scientists have simultaneous access to data , tools and processing resources
allow to implement services within shared Platform as a Service (PaaS)
Selected data processing services will be developed and made available to demonstrate the iAOS platform
Task 5.6 iAOS portal development:
outreach to stakeholders in the INTAROS WP6

7. Service integration
the iAOS tools and process

8. iAOS technical overview...

9. ...and our focus
(where Service integration occurs)

10. iAOS - Service integration
Purpose of Terradue Cloud Platform
A workplace & APIs to connect Data Providers, Developers, Integrators and Producers
Key actors:
Data Providers
Developers
Integrators
Producers
Toolboxes
Apps
Developers
Integrators
Services
Appliances
Data providers
Producers

11. iAOS - Service integration in 4 steps
Applications lifecycle on Terradue Cloud Platform
Process
Cloud Sandbox Environment
Design
The
data samples
Python, R, Dev. Toolboxes, ...
Integrate
Test
Validate
The laboratory bench
Service #1
Cloud Operations Environment
The factory chain
Service #2
Deploy
Scale
The recipes …
Process
Service #N
The goods shipment

12. Service integration (1/4)
Application design & software libraries management
Processing libs compiled from any language supported on CentOS 6.5
C/C++, Java, R, Python (other upon verification)
Runtimes: Matlab and IDL
Referenced on the Platform Repos (Artifactory, GitHub, Epel…) and accessed via standard software packages management tools
yum, conda, npm, mvn...
Wrapped in an application archetype for tests with representative data chunks (samples)
“MapReduce streaming” simple workflow (DAG) archetype provides with natively scalable services

13. Service integration (2/4)
Application tests and validation on the Cloud Sandbox
Service developers use the Cloud Sandbox tools to implement, test and validate a scalable workflow
A complete laboratory bench for service developers
Cloud native & cost effective work environment
The Cloud Sandbox software framework includes tools to proof test data discovery, parallel processing and access to results
Cloud Sandbox dashboard
job wrapper functions (copy, getParam, publish,...)
data management (OGC ® OpenSearch extensions)
service invocation (OGC ® Web Processing Service)

14. Service integration (3/4)
Application deployment and scaling on Cloud Operations environment
After validation, the developer publishes the Service for integration on a selected Cloud Operations infrastructure (e.g. EGI)
A ‘Processor as a Service’
accesses complete pools of datasets
allows application intensive computing
stores processor outputs, e.g. back to the lab

15. Service integration (4/4)
Production & operations management
Scaler productions: automated, data-driven production workflows
Processing Services are operated on a selected Cloud provider
Operators manage the service via their user account on Terradue Cloud Platform
Resources can scale up/down
The development environment remains active for maintenance activities on the Service
Service End Users can access the service’s products from ad-hoc community portals
Includes support for standard protocols like OGC Web Services

16. iAOS services integration
for data processing

17. WP5 iAOS services integration goals
sea-ice
geo-statistics
Sea-ice data generation
Time series of satellite EO-derived parameters (e.g. ice concentration, ice thickness) for selected regions
Support to sea-ice information combined with:
weather and wave forecasts
passive acoustic data
Dynamic ice and ocean processes
Support to Cal-Val analysis of Sentinel- derived sea-ice thickness
Arctic Risk Map
WP5 iAOS services integration goals

18. (Source: M. Babiker, NERSC)
Processing Service 1: Time series of sea ice parameters
NERSC
Sentinel-1 SAR provides frequent coverage of Arctic regions
Offering high-resolution imagery for detailed sea ice mapping and statistics
Stakeholders:
Science
Operators
Shipping
Fisheries
Tourism
Authorities
S-1A SAR HH
Sea Ice Type
Open water & thin ice
Young ice
Grey ice
First year ice
Land
(Source: M. Babiker, NERSC)

19. Acoustic data from D mooring
Processing Service 2: Acoustic impact assessment
NERSC
Combining acoustic data from FP7 ACOBAR and INTAROS with data from ship traffic (AIS), seismic surveys (airguns)
Acoustic data from D mooring D
Characterization of acoustic data
Noise in the ocean is an indicator in the Marine Strategy Framework Directive

20. Cloud services front ends
iAOS services integration - examples of user front ends

21. Cloud Services

22. Cloud Services
Examples of user front ends

23. Cloud Services
Examples of user front ends

24. Cloud Services
Examples of user front ends

25. iAOS - Service integration benefits for Platform users
Innovation & automation
Setup Workflow
Validate App
Develop Jobs
Deploy Service
Design Service
Monitor Service
Cloud Sandbox Environment
Cloud Operations Environment

26. herve.caumont@terradue.com Platform Operations, Terradue
Questions ?
Platform Operations, Terradue

27. Cesare Rossi, Hervé Caumont
Demo Session
Cesare Rossi, Hervé Caumont
Operations Support team at Terradue

28. Demo Session Overview Session 1
Working with Cloud resources on Terradue Cloud Platform
How are users able to self-register, get their access credentials to Cloud resources, create a work environment with state of the art tools (e.g. Git and GitHub, Maven, Oozie workflows) and work collaboratively for the development of powerful Apps ?
The session will showcase App creation for EO data processing on the Cloud.
Session 2
Working with Data resources on Terradue Cloud Platform
What are the Platform connectors to open data sources, along with integrated tools for data staging within Cloud Apps, that deliver a unique open data access experience to developers on the Platform ?
The session will showcase Cloud service configuration and deployment to enable Apps with live connectivity to open data.

29. Visit https://gist.github.com/HerveCaumont
Get the instructions
Visit

30. Choose your language

31. The Sen2Cor processing tool
Sen2Cor is a processor for Sentinel-2 Level 2A product generation and formatting; it performs the atmospheric, terrain and cirrus correction of Top-Of- Atmosphere Level 1C input data.
Sen2Cor creates Bottom-Of-Atmosphere, optionally terrain and cirrus corrected reflectance images; additional, Aerosol Optical Thickness, Water Vapor, Scene Classification Maps and Quality Indicators for cloud and snow probabilities.

32. Visit https://gist.github.com/HerveCaumont
Get the instructions (Session 2)
Visit