1. Ideas for an ICOS Competence Centre Implementation of an on-demand computation service
Ute Karstens, André Bjärby, Oleg Mirzov, Roger Groth, Mitch Selander, Maggie Hellström, Alex Vermeulen
ICOS Carbon Lund University
Diego Scardaci, Matthew Viljoen
EGI Foundation
Peter Gille, Michaela Barth
EUDAT and PDC Center for High Performance Computing, KTH Royal Institute of Technology, Stockholm

2. Integrated Carbon Observation System
“A pan-European research infrastructure for quantifying and understanding the greenhouse gas balance of the European continent”
Collect high-quality observational data relevant to the greenhouse gas budget of Europe
Make the ICOS data freely available to all interested parties
Promote the use of the ICOS data for further scientific study
Support modelling activities of the greenhouse gas fluxes in time and space
Support verification of the effectiveness of policies aiming to reduce greenhouse gas emissions

3. Footprint tool for atmospheric sites
Web-based service at ICOS Carbon Portal
On-demand computation and visualization of footprints and GHG concentrations at atmospheric measurement stations
Based on the Lagrangian atmospheric transport model STILT
Use case for testing interoperability between EGI and EUDAT services in WP7/Task 7.2 of EUDAT2020
Application examples:
Analysis of the sensitivity of GHG concentration signals at potential and existing ICOS atmospheric measurement stations to GHG emissions and fluxes
Evaluation of measurement strategies
Network design studies

4. STILT atmospheric transport model calculations
Atmospheric observations
Emissions
Meteorological driver fields
≈ 1 GB
≈ TB
≈ 2-3 TB
≈ 1-2 TB
per year
Station Footprints
GHG concentrations
Federated Cloud
STILT
Lagrangian transport model
≈ 300 CPUs per footprint
=> 750 CPUh/station/year
ICOS
Carbon Portal
Atmospheric observations
Prior fluxes
Emissions
Meteorological driver fields
EUDAT B2SAFE
≈ 1 GB
≈ TB
≈ 2-3 TB
≈ 1-2 TB
per year
Station Footprints
GHG concentrations
EGI Federated Cloud
STILT
Lagrangian transport model
≈ 670 CPUs per footprint
=> 1700 CPUh per station per year
ICOS
Carbon Portal
EUDAT B2SAFE
≈ 1-2 TB
per year
Station Footprints
GHG concentrations
Atmospheric observations
Prior fluxes
Emissions
Meteorological driver fields
EUDAT B2SAFE
≈ 1 GB
≈ TB
≈ 2-3 TB
EUDAT B2SAFE
≈ 1-2 TB
per year
Station Footprints
GHG concentrations
Atmospheric observations
Prior fluxes
Emissions
Meteorological driver fields
EUDAT B2SAFE
≈ 1 GB
≈ TB
≈ 2-3 TB
EUDAT B2SAFE
≈ 1-2 TB
per year
Station Footprints
GHG concentrations
EGI Federated Cloud
STILT
Lagrangian transport model
≈ 670 CPUs per footprint
=> 1700 CPUh per station per year
ICOS
Carbon Portal
Atmospheric observations
Prior fluxes
Emissions
Meteorological driver fields
EUDAT B2SAFE
≈ 1 GB
≈ TB
≈ 2-3 TB
EUDAT B2SAFE
≈ 1-2 TB
per year
Station Footprints
GHG concentrations
EGI Federated Cloud
STILT
Lagrangian transport model
≈ 670 CPUs per footprint
=> 1700 CPUh per station per year
ICOS
Carbon Portal
Atmospheric observations
Prior fluxes
Emissions
Meteorological driver fields
EUDAT B2SAFE
≈ 1 GB
≈ TB
≈ 2-3 TB
EGI Federated Cloud
STILT
Lagrangian transport model
≈ 670 CPUs per footprint
=> 1700 CPUh per station per year
ICOS
Carbon Portal
EGI Federated Cloud
STILT
Lagrangian transport model
≈ 670 CPUs per footprint
=> 1700 CPUh per station per year
ICOS
Carbon Portal

5. Footprint tool workflow
ICOS CP
account
User 1 VM
Web service VM
Worker
AAI
AAI
User 2
Controller
Model VM
NFS
Model Output
Particle Location
Footprints
GHG conc.
datahub.egi.eu
OneData
PDC/KTH VM
ICOS Data
Model Input
Meteo
Model Output
Model Input

6. Footprint tool workflow
ICOS CP
account
User 1 VM
Web service VM
Worker
AAI VM
Worker
AAI
User 2
Controller
Model
Model VM
NFS
Model Output
Particle Location
Footprints
GHG conc.
datahub.egi.eu
OneData
PDC/KTH VM
ICOS Data
Model Input
Meteo
Model Output
Model Input

7. … Footprint tool workflow User 1 User 2 User 3 User 4 NFSVM
Model Input
Meteo
datahub.egi.eu
OneData
ICOS CP
account
User 1
User 2
AAI
Web service
Controller
ICOS Data
Model Output
PDC/KTH
Worker
Model
NFS
Particle Location
Footprints
GHG conc.
User 3
User 4 …
on demand

8. Components of the workflow
Generic simulation
Linux VMs in EGI Federated Cloud (fedcloud.egi.eu VO for testing)
Scala for backend development, Akka Cluster for orchestration
Docker-based version of model (or data processing tool)
Storage
B2SAFE long-term storage for ICOS measurement data and products
Network File System for intermediate storage of model input/output
EGI DataHub for storage of data from multiple providers (in future)
STILT-specific components
Split model runs into small jobs to allow distribution over many cores/VMs
Handling of large numbers of small files (model output re-used as input)
Prototype: 4 VMs of medium size (8 CPUs, GB RAM) + 4 TB storage
General framework will be applied to other types of model simulations and data processing tasks

9. Proposed ICOS + eLTER Competence Centre
Integrate ICOS Carbon Portal and eLTER* data management system with European Open Science Cloud services
Many environmental RIs (like ICOS and eLTER) share a lot of common issues and challenges (data flow, storage, workflows, …)
Establish operational data processing services, routinely & seamlessly using “EOSC” services
Active collaboration between RIs and e-Service providers to develop and implement solutions for specific operational data managing and processing needs
Ongoing collaborations provided a first opportunity to connect both sides, RIs and e-Service Providers. We should build on this and continue to learn from each other so we can “speak the same language”
*Long-term Ecosystem Research in Europe

10. Proposed ICOS + eLTER Competence Centre
H2020 EINFRA-12 proposal “EOSC-Hub”
Based on experiences from WP7/Task 7.2 of EUDAT2020 and WP9 of the ENVRIplus cluster project
Prototype platform (portal and/or VRE) that allows RI users to:
Deploy RI data from long-term e-infrastructure storage based on the use of PIDs and ontology-driven metadata catalogues
Stage and couple ICOS RI and eLTER data with HTC/Cloud computing resources
Implement operational workflows using virtualized compute services to process ICOS RI and eLTER datasets
Automatically orchestrate VMs in the EOSC-Hub infrastructure based on data location and distribution

11. Thank you ! More information about ICOS: www.icos-ri.eu
and the Carbon Portal:
Contact: