Ideas for an ICOS Competence Centre Implementation of an on-demand computation service Ute Karstens, André Bjärby, Oleg Mirzov, Roger Groth, Mitch Selander, Maggie Hellström, Alex Vermeulen ICOS Carbon Portal @ Lund University Diego Scardaci, Matthew Viljoen EGI Foundation Peter Gille, Michaela Barth EUDAT and PDC Center for High Performance Computing, KTH Royal Institute of Technology, Stockholm

Integrated Carbon Observation System “A pan-European research infrastructure for quantifying and understanding the greenhouse gas balance of the European continent” Collect high-quality observational data relevant to the greenhouse gas budget of Europe Make the ICOS data freely available to all interested parties Promote the use of the ICOS data for further scientific study Support modelling activities of the greenhouse gas fluxes in time and space Support verification of the effectiveness of policies aiming to reduce greenhouse gas emissions

Footprint tool for atmospheric sites Web-based service at ICOS Carbon Portal On-demand computation and visualization of footprints and GHG concentrations at atmospheric measurement stations Based on the Lagrangian atmospheric transport model STILT Use case for testing interoperability between EGI and EUDAT services in WP7/Task 7.2 of EUDAT2020 Application examples: Analysis of the sensitivity of GHG concentration signals at potential and existing ICOS atmospheric measurement stations to GHG emissions and fluxes Evaluation of measurement strategies Network design studies

STILT atmospheric transport model calculations Atmospheric observations Emissions Meteorological driver fields ≈ 1 GB ≈ 0.5-1 TB ≈ 2-3 TB ≈ 1-2 TB per year Station Footprints GHG concentrations Federated Cloud STILT Lagrangian transport model ≈ 300 CPUs per footprint => 750 CPUh/station/year ICOS Carbon Portal Atmospheric observations Prior fluxes Emissions Meteorological driver fields EUDAT B2SAFE ≈ 1 GB ≈ 0.5-1 TB ≈ 2-3 TB ≈ 1-2 TB per year Station Footprints GHG concentrations EGI Federated Cloud STILT Lagrangian transport model ≈ 670 CPUs per footprint => 1700 CPUh per station per year ICOS Carbon Portal EUDAT B2SAFE ≈ 1-2 TB per year Station Footprints GHG concentrations Atmospheric observations Prior fluxes Emissions Meteorological driver fields EUDAT B2SAFE ≈ 1 GB ≈ 0.5-1 TB ≈ 2-3 TB EUDAT B2SAFE ≈ 1-2 TB per year Station Footprints GHG concentrations Atmospheric observations Prior fluxes Emissions Meteorological driver fields EUDAT B2SAFE ≈ 1 GB ≈ 0.5-1 TB ≈ 2-3 TB EUDAT B2SAFE ≈ 1-2 TB per year Station Footprints GHG concentrations EGI Federated Cloud STILT Lagrangian transport model ≈ 670 CPUs per footprint => 1700 CPUh per station per year ICOS Carbon Portal Atmospheric observations Prior fluxes Emissions Meteorological driver fields EUDAT B2SAFE ≈ 1 GB ≈ 0.5-1 TB ≈ 2-3 TB EUDAT B2SAFE ≈ 1-2 TB per year Station Footprints GHG concentrations EGI Federated Cloud STILT Lagrangian transport model ≈ 670 CPUs per footprint => 1700 CPUh per station per year ICOS Carbon Portal Atmospheric observations Prior fluxes Emissions Meteorological driver fields EUDAT B2SAFE ≈ 1 GB ≈ 0.5-1 TB ≈ 2-3 TB EGI Federated Cloud STILT Lagrangian transport model ≈ 670 CPUs per footprint => 1700 CPUh per station per year ICOS Carbon Portal EGI Federated Cloud STILT Lagrangian transport model ≈ 670 CPUs per footprint => 1700 CPUh per station per year ICOS Carbon Portal

Footprint tool workflow ICOS CP account User 1 VM Web service VM Worker AAI AAI User 2 Controller Model VM NFS Model Output Particle Location Footprints GHG conc. datahub.egi.eu OneData PDC/KTH VM ICOS Data Model Input Meteo Model Output Model Input

Footprint tool workflow ICOS CP account User 1 VM Web service VM Worker AAI VM Worker AAI User 2 Controller Model Model VM NFS Model Output Particle Location Footprints GHG conc. datahub.egi.eu OneData PDC/KTH VM ICOS Data Model Input Meteo Model Output Model Input

… Footprint tool workflow User 1 User 2 User 3 User 4 NFS VM Model Input Meteo datahub.egi.eu OneData ICOS CP account User 1 User 2 AAI Web service Controller ICOS Data Model Output PDC/KTH Worker Model NFS Particle Location Footprints GHG conc. User 3 User 4 … on demand

Components of the workflow Generic simulation Linux VMs in EGI Federated Cloud (fedcloud.egi.eu VO for testing) Scala for backend development, Akka Cluster for orchestration Docker-based version of model (or data processing tool) Storage B2SAFE long-term storage for ICOS measurement data and products Network File System for intermediate storage of model input/output EGI DataHub for storage of data from multiple providers (in future) STILT-specific components Split model runs into small jobs to allow distribution over many cores/VMs Handling of large numbers of small files (model output re-used as input) Prototype: 4 VMs of medium size (8 CPUs, 16-32 GB RAM) + 4 TB storage General framework will be applied to other types of model simulations and data processing tasks

Proposed ICOS + eLTER Competence Centre Integrate ICOS Carbon Portal and eLTER* data management system with European Open Science Cloud services Many environmental RIs (like ICOS and eLTER) share a lot of common issues and challenges (data flow, storage, workflows, …) Establish operational data processing services, routinely & seamlessly using “EOSC” services Active collaboration between RIs and e-Service providers to develop and implement solutions for specific operational data managing and processing needs Ongoing collaborations provided a first opportunity to connect both sides, RIs and e-Service Providers. We should build on this and continue to learn from each other so we can “speak the same language” *Long-term Ecosystem Research in Europe

Proposed ICOS + eLTER Competence Centre H2020 EINFRA-12 proposal “EOSC-Hub” Based on experiences from WP7/Task 7.2 of EUDAT2020 and WP9 of the ENVRIplus cluster project Prototype platform (portal and/or VRE) that allows RI users to: Deploy RI data from long-term e-infrastructure storage based on the use of PIDs and ontology-driven metadata catalogues Stage and couple ICOS RI and eLTER data with HTC/Cloud computing resources Implement operational workflows using virtualized compute services to process ICOS RI and eLTER datasets Automatically orchestrate VMs in the EOSC-Hub infrastructure based on data location and distribution

Thank you ! More information about ICOS: www.icos-ri.eu and the Carbon Portal: www.icos-cp.eu Contact: ute.karstens@nateko.lu.se