Reiner Schlitzer & Stephan Heckendorff Alfred Wegener Institute for Polar and Marine Research Multi-file Data Aggregation with Ocean Data View.

Slides:

Advertisements

Similar presentations

(Multi-model crop yield estimates)

Advertisements

WP1 - Data collection and metadata compilation in sea regions Sissy Iona (HCMR/HNODC) EMODNET Chemistry th Steering Committee, 2-3 December 2014,

ECOOP Data Management System (T2.2/WP2) Declan Dunne 13 th February 2008, Athens.

Training course – Ostende – 2-6 July 2012 Training Workshop – Ostende – 1-4 March 2010 Practical work on NEMO Converting files M. Fichaut.

Reiner Schlitzer Alfred Wegener Institute for Polar and Marine Research ODV Gridding Methods.

Variable section sweep  The Variable section sweep option is used to create a sweep feature in which the section varies according to the trajectory.

Using American FactFinder John DeWitt Project Manager Social Science Data Analysis Network Lisa Neidert Data Services Population Studies Center.

Reiner Schlitzer Alfred Wegener Institute for Polar and Marine Research Ocean Data View - Available Data Collections and Data Model.

Reiner Schlitzer Alfred Wegener Institute for Polar and Marine Research Ocean Data View 4 - Data Quality Control.

WP2: Data products generation in Mediterranean Sea EMODNET Chemistry Data Products, Experts Workshop Split (Croatia), June 2014 Sissy Iona, HCMR-HNODC.

Status of upgrading CDI service (user interface, harvesting via GeoNetwork, CDI interoperability options following SeaDataNet D8.7) By Dick M.A. Schaap.

Screens appear here to display the: Data Tables Plots Bibliographic Info Add New Data Form, Molecular Structure-Drawing Form ‘Tree’ for navigation between.

OBSERVATIONS & PRÉVISIONS CÔTIÈRES 3 rd SeaDataNet training course – Ostende – June 2008 NEMO reformatting tool v1 M. Fichaut.

7 th TTG meeting – Trieste – 3-4 March, 2015 SeaDataNet tools, new developments IFREMER : M. Fichaut, A. Briand, M. Larour, T. Loubrieu, V. Tosello ALTRAN.

Reiner Schlitzer Alfred Wegener Institute for Polar and Marine Research Ocean Data View.

Gas transfer coefficient : K = F / (pCO 2 ice - pCO 2 air ) = 2,5 mol m -2 d -1 atm -1 The study of air-ice CO 2 exchange emphasize the importance of gas.

DEFENSE ACQUISITION MANAGEMENT INFORMATION RETRIEVAL DAMIR Funding Formats Klif Hodgkin CACI DAMIR Team Presenter.

Object-Oriented Programming (OOP). Implementing an OOD in Java Each class is stored in a separate file. All files must be stored in the same package.

Exploring PMEL EPIC Profile Data With Java OceanAtlas (JOA). John R. Osborne

EMODnet Chemistry 2 Service Contract MARE/2012/10 S Progress of the CDI service By Dick M.A. Schaap – Technical Coordinator Istanbul – Turkey,

Profile Solutions Ltd. In Partnership with it’s Clients SIMCAP® Capacity Simulation and Business Modelling.

International Data Exchange Workshop, Kiel, PANGAEA Publishing Network for Geoscientific & Environmental Data.

Reiner Schlitzer & Stephan Heckendorff Alfred Wegener Institute for Polar and Marine Research Ocean Data View – New Developments.

ODV for Data Products Reiner Schlitzer, Alfred Wegener Institute, Bremerhaven, Germany.

Sara E. Richardson Calit2 Summer Undergraduate Research Scholarship Program Advisor: Jurgen Schulze Ivl.calit2.net/wiki CAMERA is.

OBSERVATIONS & PRÉVISIONS CÔTIÈRES SeaDataNet annual meeting, Madrid, March 2009 How to prepare data for integration in SeaDataNet.

ASEE Profiles and Salary Surveys: An Overview

WP15: Data management 1. Observations: Field Work (sediment cores & water column; sections, time series stations, moorings, etc.) Mesocosm Experiments.

Distributed Data Servers and Web Interface in the Climate Data Portal Willa H. Zhu Joint Institute for the Study of Ocean and Atmosphere University of.

Reiner Schlitzer Alfred Wegener Institute for Polar and Marine Research ODV – New Developments.

2 nd training course – Ostende – May, 2014 What’s new in NEMO 1.5?

Data management dedicated for assessing and modeling of the Mediterranean and Black Seas ecosystem changes (SESAME integrated project). Isaac Gertman,

Ocean Data View Course Objective: Import, quality control and visualize own or external datasets. What you learn: Large publicly available datasets exist.

The SeaDataNet data products regional temperature and salinity historical data collections S. Simoncelli 1, C. Coatanoan 2, O. Bäck 3, H. Sagen 4, S.

Reiner Schlitzer Alfred Wegener Institute for Polar and Marine Research Data Quality Control and Visualization with Ocean Data View 4.

Reiner Schlitzer Alfred Wegener Institute for Polar and Marine Research Ocean Data View Achievements and Future Developments.

EDMODnet Chemistry, Baltic Sea - Oxygen, Chlorophyll and contaminants - General issues Magnus Wenzer, Lotta Fyrberg, Örjan Bäck Swedish Meteorological.

Reiner Schlitzer Alfred Wegener Institute for Polar and Marine Research Ocean Data View and itsRole in SeaDatanet and its Role in SeaDatanet.

MIKADO – Generation of ISO – SeaDataNet metadata files

SeaDataNet tools NEMO, OCTOPUS, MIKADO

NEMO – Reformating tool

Practical work on NEMO Converting files

Data aggregation and products generation in the Mediterranean Sea

SeaDataNet tools NEMO, MIKADO, VALIDATOR

Using Ocean Data View for EMODnet Chemistry Reiner Schlitzer

SCAR 2016 Standing Committee on Antarctic Geographic Information

OCTOPUS – SeaDataNet Format conversion tool

WP3: Data aggregation and products generation NE Atlantic Sea

Status of ODV developments

WP2 Products in Mediterranean Sea

ISRAMAR Work Flow for SeaDataNet

Avaya Professional Services (APS) Tools ProVision

Dick M.A. Schaap – Technical Coordinator SeaDataNet Training Workshop

Tools for generation of SDN NetCDF (CFPOINT) files

Ocean Data View Reiner Schlitzer

Ocean Data View 4 - Data Quality Control Reiner Schlitzer

Heat Transport by the Atmosphere and ocean

Ocean Data View 4 - Data Quality Control Reiner Schlitzer

Data aggregation and products generation

Modeling Data Set Versioning Operations

NGMS - Software workshop pre-release PR08 Archiving

Data aggregation and products generation in the Mediterranean Sea

ODV Gridding Methods Reiner Schlitzer

Practical work on NetCDF - CFPOINT

Problem Statement and Significance

Ocean Data View Spreadsheet Format Reiner Schlitzer

Ocean Data View Reiner Schlitzer

ODV beta Multiple views, new views bar

Generation of data products in the Mediterranean Sea

Presentation transcript:

Reiner Schlitzer & Stephan Heckendorff Alfred Wegener Institute for Polar and Marine Research Multi-file Data Aggregation with Ocean Data View

SDN Data Retrieval… may yield a large number of zip files may yield a large number of zip files each zip file may contain a potentially large number of individual data files each zip file may contain a potentially large number of individual data files the type of data and set of variables may differ from file to file the type of data and set of variables may differ from file to file Challenge: While all individual data files are in ODV spreadsheet format the large number of files and their diversity make normal (single file) ODV spreadsheet import impractical.

Solution: SDN Spreadsheet importer consisting of file analysis and aggregation.

File Analysis… For every file determine the type of data (profile, time-series, trajectory). For every file determine the type of data (profile, time-series, trajectory). Uses the //SDN_parameter_mapping sectionUses the //SDN_parameter_mapping section //SDN_parameter_mapping // SDN:LOCAL:PRESSURE SDN:P011::PRESPS01 SDN:P061::UPDB // SDN:LOCAL:T90 SDN:P011::TEMPS901 SDN:P061::UPAA // SDN:LOCAL:Salinity SDN:P011::PSALPR02 SDN:P061::UUUU // SDN:LOCAL:fluorescence SDN:P011::CPHLPM01 SDN:P061::UGPL // SDN:LOCAL:Trans_red_25cm SDN:P011::POPTDR01 SDN:P061::UPCT // SDN:LOCAL:POTM SDN:P011::POTMCV01 SDN:P061::UPAA // SDN:LOCAL:Density SDN:P011::SIGTPR01 SDN:P061::UKMC

Aggregation… ODV collections are created for every data type found. ODV collections are created for every data type found. Data of a given file are imported into the corresponding ODV collection. Data of a given file are imported into the corresponding ODV collection. Process is fully automatic. Process is fully automatic. Results in one or more ODV collections containing all data of a given type. Results in one or more ODV collections containing all data of a given type.

Availability… This feature is available in ODV 4.3.3, released March 25, This feature is available in ODV 4.3.3, released March 25, WARNING: Many SDN files do not fully comply with SDN spreadsheet format. WARNING: Many SDN files do not fully comply with SDN spreadsheet format. List of problems has been submitted. List of problems has been submitted.