HDF and HDF-EOS Workshop November 30, 2006 The Homogenization and Reporting of Groundbased Atmospheric Datasets for the Validation of Earth Observing Satellite Instruments - the Aura and Envisat validation approach using HDF B. R. Bojkov, NASA-GSFC (USA) R. M. Koopman, ESA-ESRIN (I) M. De Maziere, BIRA-IASB (B) I. Boyd, NIWA (NZ)

HDF and HDF-EOS Workshop November 30, 2006 Outline Context (and some history) Implementation for Envisat and Aura Future activities

HDF and HDF-EOS Workshop November 30, 2006 Why did we do this for atmospheric sciences? (from the perspective of validation)

HDF and HDF-EOS Workshop November 30, 2006 Ensuing from European Arctic campaigns in the 1990’s ~35 stations reporting » operational NRT and scientific All stations uses NASA-Ames » “simple” ASCII More than 18 file variations » nomenclature and formulations Clear need of homogenization Near-real-time ozonesondes

HDF and HDF-EOS Workshop November 30, 2006 The problem is accentuated: Different measurement networks » LIDAR, MWR, FTIRs, UV-Vis instruments, etc. Different platforms » Groundbased, aircraft, balloon, ship, in-situ, etc. Different agencies » “Dialog” across timezones, reporting facilities, etc.

HDF and HDF-EOS Workshop November 30, 2006 So, in 1996/1997… Near impossible to use effectively for research or tools/RDB/web implementations Need to resolve this “file reporting ” problem by investigating/understanding the state of affairs in the field First objective: satellite validation of atmospheric chemistry instruments » up and coming ESA and NASA missions

HDF and HDF-EOS Workshop November 30, 2006 COSE COSE - Compilation of atmospheric Observations in support. of Satellite measurements over Europe European Commission (EC) funded project Consortium of 25 groundbased and satellite teams Timeline:

HDF and HDF-EOS Workshop November 30, 2006 COSE (2) Approach: » Investigated existing file formats, usage, community needs in consultation with the different stake-holders Recommendations: » Defined fixed file formulation: HDF4 (SDS) » Rigid metadata, including strict guidelines for attributes requirements, variable naming, etc. » Make files truly homogeneous and self explanatory » Final document by Bojkov et al., 2002 (available through AVDC)

HDF and HDF-EOS Workshop November 30, 2006 Basic HDF4 SDS file layout Global attributes: 31 attributes » Data source / contact information » Dataset contents / location / … » File generation date / version / caveats / … Attributes for each SDS: 19 attributes » Variable description / notes / caveats » Dependencies / dimensions » Units / SI conversion factor / Valid min. & max. / Fill values » Display attributes (label, axis, …)

HDF and HDF-EOS Workshop November 30, 2006 Global attributes Variable attributes

HDF and HDF-EOS Workshop November 30, 2006 Variable naming convention Variable name construction: 3 part construction VARIABLE_NAME + VARIABLE_MODE (+ VARIABLE_DESCRIPTOR) Example: » The variable name for O 3 measured by an O 3 sonde and a Brewer spectrometer are: O3_CONCENTRATIONfor the ozonesonde O3_VERTICAL.SOLARfor the Brewer in direct sun mode O3_VERTICAL.ZENITHfor the Brewer in Umkehr mode » The RMS for a FTIR ozone measurement can be expressed as: O3_VERTICAL.ZENITH_UNCERTAINTY.RMS

HDF and HDF-EOS Workshop November 30, 2006 Similar conventions for Attribute entries Instrument naming File names …

HDF and HDF-EOS Workshop November 30, 2006 First Implementation: ESA Spring 2000: ESA Envisat mission was is it’s final validation planning: » 10 instruments, focus on land, atmosphere and oceans. ESA required: » Fully relational database for calibration and validation » File “QA/QC” on all incoming files » Specifically COSE concept, including HDF 4, to be implemented for atmosphere and ocean instrument validation

HDF and HDF-EOS Workshop November 30, 2006 Envisat Cal/Val Implemented in October 2000 » Meets all ESA requirements, including QA and the RDB » 200+ users, intercontinental participation » Backbone of the coordinated validation efforts of 3 atmospheric and 2 ocean instruments » Data sources collected from ship-based to satellites » Extended to other Envisat/ESA validation activities (2004) One issue: heterogeneous reporting -the ozonesonde problem was back!

HDF and HDF-EOS Workshop November 30, present EOS-Aura validation activities The Network for Detection of Atmospheric Composition Change (NDACC - former NDSC)

HDF and HDF-EOS Workshop November 30, 2006 Aura Validation Data Center Support the NASA EOS-Aura mission » 4 atmospheric instruments » troposphere to mesosphere measurements » Share validation communities with Envisat AVDC uses the same concept/approach as Envisat » Maintain Envisat files compatibility Operational since October 2004 » ~300 users - many new participants » ~2TB validation data

HDF and HDF-EOS Workshop November 30, 2006 AVDC modifications Refined file and metadata requirements through thorough analysis of Envisat Cal/Val in early 2004 » Eliminate common user errors and misconceptions » Added HDF5 implementation (user request due to HDF5 satellite data) Implemented rigid measurement reporting templates » Collaborative effort with NDACC » Measurement specific and defined by expert community » Results in truly homogeneous files Document describing changes by Bojkov et al., 2006 available through AVDC web site

HDF and HDF-EOS Workshop November 30, 2006 Future of the AVDC/Envisat implementation AVDC refinements are being implemented into Envisat Cal/Val AVDC/Envisat format being phased into NDACC network » LIDAR, MWR, FTIR completed AVDC concept to be extended to NASA A-train (and most probably NPP wrt validation data) Format is ESA requirement for ESA/EC GMES program » To be extended to land and radar altimetry NASA and ESA to remain synchronized

HDF and HDF-EOS Workshop November 30, 2006 Closing remarks This type of work takes time » Began in 1998, and will be ongoing Requires proactive involvement by all parties » Validation teams, instrument PIs, satellite teams, … Requires rigid guidelines - down to the measurement level Once implemented, it is extremely useful for all involved as experienced for the Envisat and Aura missions when validation relies on very different data sources

HDF and HDF-EOS Workshop November 30,