EUFAR - EUropean Facility for Airborne Research Workshop on Research Metadata in Context – Sept 7/8, Nijmegen, Netherlands Metadata Frameworks in EUFAR Matt Freer, EUFAR

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary Overview EUFAR Introduction Airborne Science Metadata Overview EUFAR Implementation Summary

EUFAR - EUropean Facility for Airborne Research EUFAR Integrating Activity of the EC FP7 Budget 8 M€ Duration 4 years ( ) 33 Partners 7 instruments and 22 aircraft open to Trans-national Access

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR Introduction Jülich (DE) ‏ ULEI (DE) ‏ STFC (UK) ‏ USZ (HU) ‏ UCAM-DCHEM (UK) ‏ UWAR (PL) ‏ COSINE (NL) ‏ IRSN (FR) ‏ COMAT (FR) ‏ VKI (BE) ‏ UZH (CH) ‏ WU (NL) ‏ USBE (CZ) ‏ TAU (IL) ‏ GFZ (DE) ‏ PML (UK) ‏ PTB (DE) ‏ ONERA (FR) ‏ Météo-France (FR) Coordinator MetOffice (UK) DLR (DE) ‏ NLR (NL) ‏ Enviscope (DE) ‏ CNRS (FR) ‏ NERC (UK) ‏ INTA (ES) ‏ FUB (DE) ‏ KIT (DE) ‏ AWI (DE) ‏ CNR (IT) ‏ UNIMAN (UK) ‏ VITO (BE) ‏ 14 aircraft or instruments operators and 18 experts in airborne measurements

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR Introduction EUFAR objectives: Provide scientists with access at equal terms to the most complete range of research infrastructures Develop trans-national access to national infrastructures Reduce redundancy and fill the gaps Promote the use of research infrastructures, particularly among young scientists from countries where such infrastructures are lacking Strengthen expertise through exchange of knowledge, development of standards and protocols, construction of databases and joint instrumental research activities

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary Airborne Metadata Overview Aircraft in EUFAR measure large range of atmospheric data: Aircraft parameters –Altitude/Lat/Long –Roll/Pitch/Yaw State parameters –Temperature –Pressure –Humidity Chemistry –CO2 –O3 –NOx Microphysics –Water content –Ice crystal habits –Particle concentration –Particle size distributions Radiation –Long wave –Short wave Winds Hyperspectral imaging –Land use –Reflectivity

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary Airborne Metadata Overview Challenges: Heterogeneity: Instruments Measurements Data systems Processing techniques Data streams Collaborations on projects, research and data production between varied countries and institutions (Europe, USA, etc) Many different levels of data (raw, intermediate, processed, final, etc) ‏ All of these necessitate clearly defined, robust protocols for describing data

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary Airborne Metadata Overview External metadata Gives users broad overview of data Location Project Responsible party Quality Aids in data classification, search, retrieval, archival Embedded metadata Makes files self-describing Can define metadata globally or per-variable Reduces errors in usage Helps automate data processing tasks Stores logistical information (calibration, maintenance and other historical sensor information) ‏

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR Metadata Conventions Draft version of proposed conventions has been created and circulated to members for comment At the International Conference on Airborne Research in the Environment (ICARE) in Toulouse, Oct 2010, version 1.0 of these standards will be finalized EUFAR proposes: External INSPIRE Embedded NetCDF CF Conventions IWGADTS

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary External Metadata – INSPIRE Directive Infrastructure for spatial information in Europe (INSPIRE) ‏ Directive entered into force May 2007 by the European Commission Ensures compatibility and usability of spatial data infrastructures with the adoption of common implementing rules (e.g. Metadata) ‏ Implementation rules available at INSPIRE Principles Data should be collected only once and kept where it can be maintained most effectively Spatial information should be combined into seamless product Data should be scalable to multiple levels – detailed enough for investigation, general enough for strategic purposes Geographic information should be readily available at all levels Available data, metadata and usage conditions should be clear and easy to find

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary INSPIRE Metadata elements INSPIRE Metadata regulation mainly based on ISO Includes Information on Data set or series Classification (scientific field) ‏ Geographic location Temporal reference Quality and validity Conformity Constraints related to access and use The responsible organisation Descriptions of different image and instrument processing steps List has been expanded within EUFAR by additional quality indicators

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary INSPIRE Metadata rules implementation into EUFAR (1) ‏ Identification 1.Resource title: EUFAR project acronym 2.Resource abstract: EUFAR abstract 3.Resource type: dataset or series 4.Resource locator: 5.Unique resource identifier: EUFAR file name specification 6.Coupled resource: not applicable 7.Resource language: eng Additional: List of related available files Platform: aircraft all sign Instrument: list of instruments incl. basic sensor characteristics (e.g. spectral coverage) ‏

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary INSPIRE Metadata rules implementation into EUFAR (2) ‏ 1.Classification Topic category: EUFAR main scientific field Spatial data service type: not applicable 2.Keyword Keyword value: EUFAR scientific theme Originating controlled vocabulary: GEMET - INSPIRE themes, version Geographic Location Geographic bounding box Additional: Area information: Country, Province/Region, Local name 4.Temporal Reference Temporal extent: date of data acquisition Date of publication: date of entry into EUFAR-DB Date of last revision: date of last revision Date of creation: date of data processing

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary INSPIRE Metadata rules implementation into EUFAR (3) ‏ Quality and Validity 1.Lineage 2.Spatial resolution Additional: Processing information (e.g. processing level) ‏ Calibration information (e.g. date of spectral calibration) ‏ Acquisition information (e.g. overall heading) ‏ Data quality layers for hyperspectral imagery Sensor calibration and system correction Image data artefacts and processing errors GPS / IMU related errors, geometric correction Atmospheric correction and atmospheric conditions quality control flags hyperspectral image attitude and position black body information, …

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary INSPIRE Metadata rules implementation into EUFAR (4) ‏ 1.Conformity Specification: INSPIRE implementing rules Degree: degree of conformity (true or false) ‏ 2.Constraints related to access and use Conditions applying to access and use: EUFAR consortium agreement Limitations on public access: EUFAR consortium agreement 3.Organisation responsible for the establishment, management, maintenance and distribution Responsible party: Responsible party role: pointOfContact 4. Metadata on metadata Metadata point of contact: e.g. DLR, PAF Metadata date: date when the metadata record was created or updated Metadata language: eng

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary Embedded metadata Tangent: EUFAR recommended file format NetCDF (network Common Data Form) ‏ Wide use in atmospheric science community APIs for most common programming languages (C, C++, Fortran, Java, Python, Matlab, IDL) NetCDF is: Binary Portable Scalable: –Multi-dimensional array oriented –Subsets of large datasets easily accessible Appendable Self describing Data model: Dimensions Variables Attributes

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary Embedded metadata Tangent: NetCDF (network Common Data Form) ‏ NetCDF is: Binary Portable, i.e. machine independent Scalable, I.e. multi-dimensional array oriented Appendable Self describing Wide use in atmospheric science community Data model: Dimensions Variables Attributes

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR NetCDF Metadata Conventions EUFAR conventions for NetCDF extend existing, common standards: 1.NetCDF Climate and Forecast (CF) Conventions Broad conventions, applicable across atmospheric and earth science 1.Interagency Working Group for Airborne Data and Telemetry Systems (IWGADTS) Conventions US-based group Tailored for aircraft

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR NetCDF Metadata Conventions Specifies file structure Names for time dimensions Time for 1Hz signal Time25 for 25Hz signal etc Order of dimensions Definition of time coordinate – follow CF conventions Defines standard file attributes Conventions Title Institution Source History References Comment

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR NetCDF Metadata Conventions Defines standard variable attributes units Long name Standard name Valid range Sampled rate Category Calibration coefficients Dependencies Processor Provides standard vocabularies based on CF Conventions standard names units

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR NetCDF Metadata Conventions EUFAR NetCDF Metadata Conventions do not: Define or standardize variable names Why? Unique and varied instruments and measurements between aircraft and institutions Can be multiple measurements of the same quantity Example: NCAR C130 Pressure measurements

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary NetCDF metadata rules for EUFAR Example: NCAR C130 Pressure measurements PSFD PSFDC PSFRD PSFRDC PCAB Solution: Data Discovery Define global attribute ‘ air_pressure ’ The value of this attribute indicates recommended field Example: reference:air_pressure = PSFDC Data discovery helps users and software choose correct measurement All pressure: Which one to use? }

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary NetCDF metadata rules for EUFAR EUFAR standard data discovery fields Highly recommended Latitude Longitude Altitude Recommended Wind speed Upward air velocity Air temperature Platform speed Dew point temperature Air pressure Platform roll angle Platform pitch angle Platform orientation Ground speed E/W Ground speed N/S

Overview EUFAR Intro. Airborne Metadata Challenges Schemes EUFAR Conv. External Embedded Summary EUFAR conventions: Proposal has been circulated Version 1.0 being released in Oct 2010 External - INSPIRE General description Archival Search Embedded - NetCDF Data usage Processing Accuracy