1. Data Flow & Data Services for MOSAiC
Challenges from a large scale program
Arndt
Steinhage
MOSAiC Implementation Workshop 13-16/11/ Stephan Frickenhaus, Hans Pfeiffenberger
AWI Computing and Data Centre

2. Vision for MOSAiC Data Management
Interdisciplinary Data Collection, Collective Data publications
Data Policy
Co-ordinated Data Flows
Data Protocol, Data Management Plan
Underway Data Science support/ Service, on-shore Data Scientist
Taking care of the MOSAiC Data Collection

3. Challenges from the data point of view
to harmonize Science and Infrastructure/ Logistics/ Services in interdisciplinary work
Need for a common data protocol
Need a committment from the groups to help organize this

4. The Data Protocol
MOSAiC [2019/20] is an observatory producing meaningful scientific data
Successful data management needs scientists‘ 100% committment
The MOSAiC Data Legacy [2021] is the primary and lasting output, and the basis for science
Prerequisites for high impact publications:
Common meta-data and procedure standards
Quality managed and published data
Consortium agreements on publication strategy
Open by Default; group-specific embargo timing tbd

5. Chapters of the Data Protocol
Objective
Definitions
raw data, primary data, data products, …
Data Policy, incl. fair publication rules
Data Standards
Data Management
roles and responsibilities
common data pool
Sample Management
Data Archival
Amendmends and resolution of conflicts

6. Data Protocol Group
AWI Roland Neuber (Atmos) Thomas Krumpen (Ice/Snow) Allison Fong (Eco, Bio-Sampling) Ellen Damm (BGC) Ben Rabe (Ocean) Andreas Herber (airbourne) Amelie Driemel (PANGAEA) Peter Gerchow, Angela Schäfer, Hans Pfeiffenberger, Ingo Schewe (Logistics, Data Flow/ Data Services/ Data Science)
Further members tbd during break out!
Gunnar Spreen
(remote sensing) …

7. Expected Contributions from Groups
Initial work
Expected Contributions from Groups
One Person per Group for Data Protocol
Available for regular Telcos
Device lists (until next spring meeting)
Responsible person
Sensor type, transport through POLARSTERN Satcom | access through home institution, commercial processor, …
Data volumes, data frequencies, near real time needs, data processing tools, shore-to-ship needs?
Embargo periods
… (input from breakouts)

8. Multivariate Visualization as an App
Data science services
Managed Data Base on board for core variables as multivariate time-series
Multivariate Visualization as an App
Show full time series (PCA, MDS, …)
Allow for clustering
Enable focussing on sub-sets/ events
Export selected data
Allow for comparing events

9. Offer to publish MOSAiC data through an
ESSD Special Issue
Chief Eds offer support structuring issue
At least one guest editor from MOSAiC,
At least one guest editor from outside

10. O2A – Observations to Archive
Data Flow Framework
Arndt
Steinhage

11. Current Use Case: FRAM Ice tethered platforms
Frontiers in Arctic Marine Monitoring
Seit 2014
25 M Euro
Ice tethered: distributed buoys and networks
Water columns: moorings, winched profiling, sampler, autonoumous under water vehicle (AUV)
Deeper water column: particle camers, light frame on sight key species investigations (LOKI), acoustic recorders
Ocean floor: ocean floor observing system (OFOS), benthic lander system, autonomous crawler (Tramper)
radiation, snow height, depth,
ice thickness, temperature,
salinity, oxygen, chlorophyll a …
Medieningenieure Bremen / Sabine Lüdeling

12. Current Use Case: FRAM Water column fluorescence, nutrients, salinity,
Ice tethered: distributed buoys and networks
Water columns: moorings, winched profiling, sampler, autonoumous under water vehicle (AUV)
Deeper water column: particle camers, light frame on sight key species investigations (LOKI), acoustic recorders
Ocean floor: ocean floor observing system (OFOS), benthic lander system, autonomous crawler (Tramper)
fluorescence, nutrients, salinity,
temperature, conductivity, depth,
acoustic doppler current profiler,
water and phytoplankton samples, …
Medieningenieure Bremen / Sabine Lüdeling

13. Next Use Case: MOSAiC?
Multidisciplinary drifting Observatory for the Study of Arctic Climate
> 60 M Euro

14. Data Flow Framework

15. Data Flow Framework

16. Objectives Generic infrastructure for data flows
Sustainability and up-to-date services
Interoperability and standards
e.g. Open Geospatial Consortium
Seamless integration with existing infrastructure
Web GIS
Web Portals
Data Archive

17. Challenges Heterogeneity of scientific needs and workflows
Number of different instruments, data sources and formats
Integration with existing solutions, e.g. for the data flow, but also administrative information
limited additional Effort acceptable
multitude of Standards

18. Sensor Description
Platform and device descriptions for provenance information and reduced data integration effort
Versioning and citability
Interoperability and standards
~1200 descriptions available and counting

19. Dashboard User-customizable, flexible dashboards for data monitoring
Automatic data streaming of near-real time and delayed-mode data
Based on sensor descriptions and configurations
Since 2011
Fast growing number of values and sensors
350 M measurements
460 sensors

20. Dashboard Since 2011 Fast growing number of values and sensors
350 M measurements
460 sensors

21. Maps and Portals

22. Data Flow Framework

23. Current work for FRAM
Developing a science community workspace for data sharing and data analyses
State-of-the-art storage, replicated between Bremerhaven and Potsdam
User-friendly “one-click” compute solutions with virtual machines and containers
Hadoop big data analysis based on Hortonworks data flow and data platform
Raster data management and analysis with rasdaman

24. Use Case: MOSAiC
Multidisciplinary drifting Observatory for the Study of Arctic Climate
> 60 M Euro

25. O2A: MOSAiC only 2 x 100MB/day ?
Polarstern Satellite Link for Data Monitoring and Remote Service
Polarstern Data Storage
MOSAiC Raw Data
only 2 x 100MB/day ?
Ship-to-shore Data Transfer
Onboard Data Transfer
“direct” satellite links to partner sites

26. Comprehensive onshore
O2A: MOSAiC
ftp from 3rd party’s sites
raw data
Aircraft data
ftp from partner’s sites
ftp from EO (sat.) providers
Polarstern Data
MOSAiC
Comprehensive onshore
Data Collection
Primary data

27. With contributions from:
Roland Koppe
Peter Gerchow
Angela Schäfer
Ana Macario