EUFAR - EUropean Facility for Airborne Research Workshop on Research Metadata in Context – Sept 7/8, Nijmegen, Netherlands Metadata Frameworks in EUFAR.

Slides:



Advertisements
Similar presentations
THE DONOR PROJECT Titia van der Werf-Davelaar. Project Financed by: Innovation of Scientific Information Provision (IWI) Duration: –phase 1: 1 may 1998.
Advertisements

Future Directions and Initiatives in the Use of Remote Sensing for Water Quality.
SWOT and User Needs Workshop, DLR Oberpfaffenhofen, 5-6 July 2006 HYRESSA - HYperspectral REmote Sensing in Europe specific Support Actions 1 HYRESSA HYperspectral.
WMO Core Profile of the ISO Metadata Standard Steve Foreman Chair IPET-Metadata Implementation.
BADC Workshop 2: BADC Services to Data Suppliers Royal Met. Soc. Conference – 14 September 2005 Ag Stephens et al.
MEDIN Standards M. Charlesworth and the MEDIN Standards Working Group.
FP7-Infra : Design studies for European Research Infrastrutures 1st October 2011 – 31st December 2014 Duration 39 months – Periods : 2 (month.
NOAA Metadata Update Ted Habermann. NOAA EDMC Documentation Directive This Procedural Directive establishes 1) a metadata content standard (International.
ISO Standards: Status, Tools, Implementations, and Training Standards/David Danko.
1 CF Unleashed: Introduction to Cf/Radial Joe VanAndel National Center for Atmospheric Research 2013/1/8 The National Center for Atmospheric.
Meteorological Observatory Lindenberg – Richard Assmann Observatory The GCOS Reference Upper Air Network.
AIXM 5.1 Seminar 12 – 13 December 2011
Data Formats: Using Self-describing Data Formats Curt Tilmes NASA Version 1.0 February 2013 Section: Local Data Management Copyright 2013 Curt Tilmes.
EXPECTATIONS OF TURKISH ENVIRONMENTAL SECTOR FROM INSPIRE Ministry of Environment and Forestry June, 2010 Özlem ESENGİN Ahmet ÇİVİ Tuncay DEMİR.
AON Data Questionnaire Results 21 Respondents Last Updated 27 March 2007 First AON PI Meeting Scot Loehrer, Jim Moore.
, Implementing GIS for Expanded Data Accessibility and Discoverability ASDC Introduction The Atmospheric Science Data Center (ASDC) at NASA Langley Research.
Database System Concepts and Architecture
North American Profile: Partnership across borders. Sharon Shin, Metadata Coordinator, Federal Geographic Data Committee Raphael Sussman; Manager, Lands.
Mapping between SOS standard specifications and INSPIRE legislation. Relationship between SOS and D2.9 Matthes Rieke, Dr. Albert Remke (m.rieke,
The elements of the MEDIN discovery standard Becky Seeley, DASSH & MEDIN helpdesk.
IntroductionToSensorML Alexandre Robin – October 2006.
Metadata and Geographical Information Systems Adrian Moss KINDS project, Manchester Metropolitan University, UK
The netCDF-4 data model and format Russ Rew, UCAR Unidata NetCDF Workshop 25 October 2012.
1 Interoperability of Spatial Data Sets and Services Data quality and Metadata: what is needed, what is feasible, next steps Interoperability of Spatial.
Metadata Implementation Ted Habermann NOAA National Geophysical Data Center Documentation: It’s not just discovery... 50% change in global average Why?
Introduction to EUFAR EUropean Fleet for Airborne Research
Documentation from NcML to ISO Ted Habermann, NOAA NESDIS NGDC.
Adoption of RDA-DFT Terminology and Data Model to the Description and Structuring of Atmospheric Data Aaron Addison, Rudolf Husar, Cynthia Hudson-Vitale.
1 Computing Challenges for the Square Kilometre Array Mathai Joseph & Harrick Vin Tata Research Development & Design Centre Pune, India CHEP Mumbai 16.
Using the Global Change Master Directory (GCMD) to Promote and Discover ESIP Data, Services, and Climate Visualizations Presented by GCMD Staff January.
MEDIN Work Plan for By March 2011 MEDIN will be 3 years into the original 5 year development plan started in Would normally ask for continued.
Metadata – use data discovery e.g. a library catalog data assessment determine the fitness-for-purpose of a data set data retrieval e.g., format.
The Geographic Information System of the European Commission (GISCO) By Albrecht Wirthmann, GISCO, Eurostat ESPON.
Comparison of CEN, FGDC and ISO standards for metadata Ing. Jan Ruzicka Institute of Economics and Control Systems VŠB – Technical university Ostrava 17.
Documenting UAF Data Ted Habermann NOAA/NESDIS/National Geophysical Data Center.
Creating Good Documentation NOAA National Geophysical Data Center
Rob Walker The INSPIRE metadata regulations and quality issues – a user view Rob Walker Association for Geographic Information, London.
Exporting WaterML from the Earth System Modeling Framework Xinqi Wang Louisiana State University NCAR SIParCS Program August 4, 2009.
The HDF Group Introduction to netCDF-4 Elena Pourmal The HDF Group 110/17/2015.
Eurostat 4. SDMX: Main objects for data exchange 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October.
Fire Emissions Network Sept. 4, 2002 A white paper for the development of a NSF Digital Government Program proposal Stefan Falke Washington University.
An introduction to the MEDIN Discovery Metadata Standard.
The Proliferation of Metadata Standards and the Evolution of NASA’s Global Change Master Directory (GCMD) Standard for Uses in Earth Science Data Discovery.
A look to the past for the future- The North American Profile Sharon Shin Metadata Coordinator Federal Geographic Data Committee.
An introduction to the MEDIN Discovery Metadata Standard.
Global Change Master Directory (GCMD) Mission “To assist the scientific community in the discovery of Earth science data, related services, and ancillary.
CF 2.0 Coming Soon? (Climate and Forecast Conventions for netCDF) Ethan Davis ESO Developing Standards - ESIP Summer Mtg 14 July 2015.
Cloud-based e-science drivers for ESAs Sentinel Collaborative Ground Segment Kostas Koumandaros Greek Research & Technology Network Open Science retreat.
An Introduction to the MEDIN Discovery Metadata Standard MEDIN Workshop BGS, Edinburgh, June 2015.
An Introduction to the MEDIN Discovery Metadata Standard MEDIN Workshop NOC, Liverpool, Sept 2015.
QA4EO in 10 Minutes! A presentation to the 10 th GHRSST Science Team Meeting.
Geospatial metadata Prof. Wenwen Li School of Geographical Sciences and Urban Planning 5644 Coor Hall
1 © 2007 Humboldt Consortium Fraunhoferstraße Darmstadt HUMBOLDT Surveys: The Handbook of Standards and the User Classification.
Making FAAM Flights Discoverable
GSICS Collaboration Servers a Vehicle for International Collaboration Status 2011 Peter Miu EUMETSAT.
Tomas Kliment Junior Researcher Italian National Research Council
Data Management: Documentation & Metadata
MIWP : Deliverables for approval
DRIVER Digital Repository Infrastructure Vision for European Research
Helping a friend out Guidelines for better software
2. An overview of SDMX (What is SDMX? Part I)
Relation between WIS and WIGOS metadata
Session 2: Metadata and Catalogues
9 April 2018 Phil Brown, Met Office, UK
Geographical Information System
School of Information Studies, Syracuse University, Syracuse, NY, USA
Hans Dufourmont Eurostat Unit E4 – Structural Funds
Proposal of a Geographic Metadata Profile for WISE
Hans Dufourmont Eurostat Unit E4 – Structural Funds
Chapter 5.5 Metadata John Cima.
Presentation transcript:

EUFAR - EUropean Facility for Airborne Research Workshop on Research Metadata in Context – Sept 7/8, Nijmegen, Netherlands Metadata Frameworks in EUFAR Matt Freer, EUFAR

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary Overview EUFAR Introduction Airborne Science Metadata Overview EUFAR Implementation Summary

EUFAR - EUropean Facility for Airborne Research EUFAR Integrating Activity of the EC FP7 Budget 8 M€ Duration 4 years ( ) 33 Partners 7 instruments and 22 aircraft open to Trans-national Access

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR Introduction Jülich (DE) ‏ ULEI (DE) ‏ STFC (UK) ‏ USZ (HU) ‏ UCAM-DCHEM (UK) ‏ UWAR (PL) ‏ COSINE (NL) ‏ IRSN (FR) ‏ COMAT (FR) ‏ VKI (BE) ‏ UZH (CH) ‏ WU (NL) ‏ USBE (CZ) ‏ TAU (IL) ‏ GFZ (DE) ‏ PML (UK) ‏ PTB (DE) ‏ ONERA (FR) ‏ Météo-France (FR) Coordinator MetOffice (UK) DLR (DE) ‏ NLR (NL) ‏ Enviscope (DE) ‏ CNRS (FR) ‏ NERC (UK) ‏ INTA (ES) ‏ FUB (DE) ‏ KIT (DE) ‏ AWI (DE) ‏ CNR (IT) ‏ UNIMAN (UK) ‏ VITO (BE) ‏ 14 aircraft or instruments operators and 18 experts in airborne measurements

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR Introduction EUFAR objectives: Provide scientists with access at equal terms to the most complete range of research infrastructures Develop trans-national access to national infrastructures Reduce redundancy and fill the gaps Promote the use of research infrastructures, particularly among young scientists from countries where such infrastructures are lacking Strengthen expertise through exchange of knowledge, development of standards and protocols, construction of databases and joint instrumental research activities

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary Airborne Metadata Overview Aircraft in EUFAR measure large range of atmospheric data: Aircraft parameters –Altitude/Lat/Long –Roll/Pitch/Yaw State parameters –Temperature –Pressure –Humidity Chemistry –CO2 –O3 –NOx Microphysics –Water content –Ice crystal habits –Particle concentration –Particle size distributions Radiation –Long wave –Short wave Winds Hyperspectral imaging –Land use –Reflectivity

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary Airborne Metadata Overview Challenges: Heterogeneity: Instruments Measurements Data systems Processing techniques Data streams Collaborations on projects, research and data production between varied countries and institutions (Europe, USA, etc) Many different levels of data (raw, intermediate, processed, final, etc) ‏ All of these necessitate clearly defined, robust protocols for describing data

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary Airborne Metadata Overview External metadata Gives users broad overview of data Location Project Responsible party Quality Aids in data classification, search, retrieval, archival Embedded metadata Makes files self-describing Can define metadata globally or per-variable Reduces errors in usage Helps automate data processing tasks Stores logistical information (calibration, maintenance and other historical sensor information) ‏

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR Metadata Conventions Draft version of proposed conventions has been created and circulated to members for comment At the International Conference on Airborne Research in the Environment (ICARE) in Toulouse, Oct 2010, version 1.0 of these standards will be finalized EUFAR proposes: External INSPIRE Embedded NetCDF CF Conventions IWGADTS

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary External Metadata – INSPIRE Directive Infrastructure for spatial information in Europe (INSPIRE) ‏ Directive entered into force May 2007 by the European Commission Ensures compatibility and usability of spatial data infrastructures with the adoption of common implementing rules (e.g. Metadata) ‏ Implementation rules available at INSPIRE Principles Data should be collected only once and kept where it can be maintained most effectively Spatial information should be combined into seamless product Data should be scalable to multiple levels – detailed enough for investigation, general enough for strategic purposes Geographic information should be readily available at all levels Available data, metadata and usage conditions should be clear and easy to find

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary INSPIRE Metadata elements INSPIRE Metadata regulation mainly based on ISO Includes Information on Data set or series Classification (scientific field) ‏ Geographic location Temporal reference Quality and validity Conformity Constraints related to access and use The responsible organisation Descriptions of different image and instrument processing steps List has been expanded within EUFAR by additional quality indicators

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary INSPIRE Metadata rules implementation into EUFAR (1) ‏ Identification 1.Resource title: EUFAR project acronym 2.Resource abstract: EUFAR abstract 3.Resource type: dataset or series 4.Resource locator: 5.Unique resource identifier: EUFAR file name specification 6.Coupled resource: not applicable 7.Resource language: eng Additional: List of related available files Platform: aircraft all sign Instrument: list of instruments incl. basic sensor characteristics (e.g. spectral coverage) ‏

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary INSPIRE Metadata rules implementation into EUFAR (2) ‏ 1.Classification Topic category: EUFAR main scientific field Spatial data service type: not applicable 2.Keyword Keyword value: EUFAR scientific theme Originating controlled vocabulary: GEMET - INSPIRE themes, version Geographic Location Geographic bounding box Additional: Area information: Country, Province/Region, Local name 4.Temporal Reference Temporal extent: date of data acquisition Date of publication: date of entry into EUFAR-DB Date of last revision: date of last revision Date of creation: date of data processing

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary INSPIRE Metadata rules implementation into EUFAR (3) ‏ Quality and Validity 1.Lineage 2.Spatial resolution Additional: Processing information (e.g. processing level) ‏ Calibration information (e.g. date of spectral calibration) ‏ Acquisition information (e.g. overall heading) ‏ Data quality layers for hyperspectral imagery Sensor calibration and system correction Image data artefacts and processing errors GPS / IMU related errors, geometric correction Atmospheric correction and atmospheric conditions quality control flags hyperspectral image attitude and position black body information, …

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary INSPIRE Metadata rules implementation into EUFAR (4) ‏ 1.Conformity Specification: INSPIRE implementing rules Degree: degree of conformity (true or false) ‏ 2.Constraints related to access and use Conditions applying to access and use: EUFAR consortium agreement Limitations on public access: EUFAR consortium agreement 3.Organisation responsible for the establishment, management, maintenance and distribution Responsible party: Responsible party role: pointOfContact 4. Metadata on metadata Metadata point of contact: e.g. DLR, PAF Metadata date: date when the metadata record was created or updated Metadata language: eng

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary Embedded metadata Tangent: EUFAR recommended file format NetCDF (network Common Data Form) ‏ Wide use in atmospheric science community APIs for most common programming languages (C, C++, Fortran, Java, Python, Matlab, IDL) NetCDF is: Binary Portable Scalable: –Multi-dimensional array oriented –Subsets of large datasets easily accessible Appendable Self describing Data model: Dimensions Variables Attributes

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary Embedded metadata Tangent: NetCDF (network Common Data Form) ‏ NetCDF is: Binary Portable, i.e. machine independent Scalable, I.e. multi-dimensional array oriented Appendable Self describing Wide use in atmospheric science community Data model: Dimensions Variables Attributes

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR NetCDF Metadata Conventions EUFAR conventions for NetCDF extend existing, common standards: 1.NetCDF Climate and Forecast (CF) Conventions Broad conventions, applicable across atmospheric and earth science 1.Interagency Working Group for Airborne Data and Telemetry Systems (IWGADTS) Conventions US-based group Tailored for aircraft

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR NetCDF Metadata Conventions Specifies file structure Names for time dimensions Time for 1Hz signal Time25 for 25Hz signal etc Order of dimensions Definition of time coordinate – follow CF conventions Defines standard file attributes Conventions Title Institution Source History References Comment

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR NetCDF Metadata Conventions Defines standard variable attributes units Long name Standard name Valid range Sampled rate Category Calibration coefficients Dependencies Processor Provides standard vocabularies based on CF Conventions standard names units

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR NetCDF Metadata Conventions EUFAR NetCDF Metadata Conventions do not: Define or standardize variable names Why? Unique and varied instruments and measurements between aircraft and institutions Can be multiple measurements of the same quantity Example: NCAR C130 Pressure measurements

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary NetCDF metadata rules for EUFAR Example: NCAR C130 Pressure measurements PSFD PSFDC PSFRD PSFRDC PCAB Solution: Data Discovery Define global attribute ‘ air_pressure ’ The value of this attribute indicates recommended field Example: reference:air_pressure = PSFDC Data discovery helps users and software choose correct measurement All pressure: Which one to use? }

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary NetCDF metadata rules for EUFAR EUFAR standard data discovery fields Highly recommended Latitude Longitude Altitude Recommended Wind speed Upward air velocity Air temperature Platform speed Dew point temperature Air pressure Platform roll angle Platform pitch angle Platform orientation Ground speed E/W Ground speed N/S

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR conventions: Proposal has been circulated Version 1.0 being released in Oct 2010 External - INSPIRE General description Archival Search Embedded - NetCDF Data usage Processing Accuracy