The HDF Group HDF/HDF-EOS Workshop XV1 Tools to Improve the Usability of NASA HDF Data Kent Yang and Joe Lee The HDF Group April 17, 2012

Tutorial Outline Introduction 5 Tools 1.H4CF Conversion Toolkit (h4cf) 2.h4toh5 Conversion Tool (h4toh5) 3.HDF-EOS5 to netCDF-4 Converter (eos52nc4) 4.HDF-EOS5 Augmentation Tool (aug_eos5) 5.HDF-EOS2 Dumper (eos2dump) Demo HDF/HDF-EOS Workshop XV2April 17, 2012

Who Said Users Don’t Matter? April 17, 2012HDF/HDF-EOS Workshop XV3

NASA HDF/HDF-EOS Data Usability of NASA HDF Data April 17, 2012HDF/HDF-EOS Workshop XV4 IDLMATLAB Panoply IDV A school teacher’s view Panoply IDV A school teacher’s expectation

The 5 Tools April 17, 2012HDF/HDF-EOS Workshop XV5 Improve the usability of NASA Data Supported by The HDF Group Funded by NASA / Raytheon

hdf4 5 Tools that Improve Usability April 17, 2012HDF/HDF-EOS Workshop XV6 eos2 eos5 ASCII NETCDF-4 NETCDF-3 Panoply / IDV MATLAB IDL NCL

H4CF Conversion Toolkit Follow CF convention Both Library and Tools April 17, 2012HDF/HDF-EOS Workshop XV7 hdf4 eos2 NETCDF-4 NETCDF-3 h4cf Panoply / IDV

H4CF Conversion Toolkit Two ways to build: 1.Use configure script. 2.Edit Makefile template. April 17, 2012HDF/HDF-EOS Workshop XV8

H4CF Conversion Library Variables (datasets) list Attributes (metadata) list All Vars & Attrs follow CF conventions Develop your own CF application! April 17, 2012HDF/HDF-EOS Workshop XV9

H4CF Library APIs Example // open the example HDF file h4cf_open("geo.hdf"); // HDF file attributes are obtained in a C++ STL map object const map file_attrs = h4cf_get_file_attrs(); // HDF file variables are obtained in a C++ STL list object const list pvars = h4cf_get_vars(); // close the HDF file h4cf_close(); April 17, 2012HDF/HDF-EOS Workshop XV10

H4CF Conversion Tools 2 Tools based on H4CF APIs: cdldumper h4tonccf April 17, 2012HDF/HDF-EOS Workshop XV11

cdldumper Usage: cdldumper filename.hdf Dump file in Common Data Language. Useful for quick check and debugging purpose. April 17, 2012HDF/HDF-EOS Workshop XV12

cdldumper Demo April 17, 2012HDF/HDF-EOS Workshop XV13 Number of attributes=1 _FillValue=255

cdldumper Demo April 17, 2012HDF/HDF-EOS Workshop XV14 %./cdldumper -H AMSR_E_L3_5DaySnow_V09_ hdf netCDF AMSR_E_L3_5DaySnow_V09_ hdf { dimensions: XDim_Northern_Hemisphere = 721, XDim_Southern_… variables: ubyte Flags_NorthernPentad(YDim_Northern_Hemis… Flags_NorthernPentad:_FillValue = 255; Flags_NorthernPentad:origname = "/Northern… Flags_NorthernPentad:long_name = "/Northern… Flags_NorthernPentad:coordinates = "latitude… Number of attributes=4 _FillValue=255 orig_name long_name coordinates

h4tonccf Usage: h4tonccf filename.hdf [filename.nc] generates either netcdf-3 or netcdf-4 file. follows netcdf classic model. April 17, 2012HDF/HDF-EOS Workshop XV15

h4tonccf Demo April 17, 2012HDF/HDF-EOS Workshop XV16

Don’t Follow CF Conventions Blindly April 17, 2012HDF/HDF-EOS Workshop XV17 [1] CF: scale * value + offset [2] MODIS: scale * (value – offset) Scale = 0.01 Value = 5000 Offset = [1] 0.01 * (-15000) = [2] 0.01 * (5000 – (-15000)) = 200

Kelvin? April 17, 2012HDF/HDF-EOS Workshop XV18

MODIS Scale / Offset Correction HDF/HDF-EOS Workshop XV19April 18, 2012

Converted File Size Comparison April 17, 2012HDF/HDF-EOS Workshop XV20 HDF-EOS2 Netcdf-4 Netcdf-3 9X

Limitation – No Support Yet April 17, 2012HDF/HDF-EOS Workshop XV21 MISR (SOM/LAMAZ) MERRA AVHRR LISTO QuikSCAT/SeaWind MOPITT L2 etc.

h4toh5 Conversion Tool Convert any hdf4 to hdf5. April 17, 2012HDF/HDF-EOS Workshop XV22 eos2 NETCDF-4 h4toh5 Usage: h4toh5 –eos –nc4 in.hdf out.nc Static Binary w/ –eos & -nc4:

h4tonccf vs. h4toh5 April 17, 2012HDF/HDF-EOS Workshop XV23 h4tonccfh4toh5 w/ -eos –nc4 Handle non-eos2 hdf4 YesNo Follow CF conventions YesNo Output netcdf classic YesNo Output netcdf enhanced NoYes

eos52nc4 converts hdf-eos5 to netcdf-4. follows netcdf enhanced model. does not follow CF conventions. Usage: eos52nc4 test.he5 test.nc April 17, 2012HDF/HDF-EOS Workshop XV24 eos5 NETCDF-4 eos52nc4

aug_eos5 augments hdf-eos5 to netcdf-4. Both HDF-EOS5 and NetCDF4 can access it. Usage: aug_eos5 [-i|-f map] test.h5 modifies the input file directly. does not follow CF conventions. April 17, 2012HDF/HDF-EOS Workshop XV25 eos5 NETCDF-4 aug_eos5

hdf5 netcdf4 Augmentation Concept April 17, 2012HDF/HDF-EOS Workshop XV26 hdf-eos5 HDF-EOS5 API NETCDF-4 API aug_eos5 augmentation hdf-eos5 HDF5 API

Why NetCDF-4 Doesn’t Work April 17, 2012HDF/HDF-EOS Workshop XV27 [nLevels][nTimes] NETCDF-4 API hdf-eos5

What aug_eos5 Does April 17, 2012HDF/HDF-EOS Workshop XV28 [nLevels][nTimes] NETCDF-4 API hdf-eos5

Options in aug_eos5 1.Default option -i option 2.-f map_file option April 17, 2012HDF/HDF-EOS Workshop XV29

Default option in aug_eos5 Grid XDim, YDim: calculated Other dims: default fill value Swath / ZA All dims: default fill value April 17, 2012HDF/HDF-EOS Workshop XV30

Better -i option in aug_eos5 Grid XDim, YDim: calculated Other dims: 1,2,…,n Swath / ZA All dims: 1,2,…,n April 17, 2012HDF/HDF-EOS Workshop XV31

Best -f option in aug_eos5 Needs an input map file. Input file specifies 3 rules : 1.What to be filled in 2.How to fill in 3.Where to get the values Most useful and flexible option April 17, 2012HDF/HDF-EOS Workshop XV32

aug_eos5 Input File Format Each line in input file contains 1.How: 0,1,2 2.What: dim name 3.Where: data file or existing variable in file. 4.(Optional) Multiple grid/swath/za case only: grid/swath/za name: April 17, 2012HDF/HDF-EOS Workshop XV33

aug_eos5 Input File Example 0 nTimes Time 1 nLevels data.txt 2 nCandidates April 17, 2012HDF/HDF-EOS Workshop XV34 Copy the data Read file data.txt Fill 1,2,…, n HowWhatWhere 95% cases – Copy the data

Multiple Swath Case swath name: BrO 0 nTimes Time 1 nLevels data1.txt 2 nCandidates April 17, 2012HDF/HDF-EOS Workshop XV35 HowWhatWhere swath name: BrO column 2 nTimes 1 nLevels data2.txt Multiple

aug_eos5 vs eos52nc4 April 17, 2012HDF/HDF-EOS Workshop XV36 aug_eos5eos52nc4 hdfeos-5YesNo netcdf-4Yes

HIRDLS/MLS Team Use aug_eos5 April 17, 2012HDF/HDF-EOS Workshop XV37 So You Can!

eos2 dump HDF-EOS2 Dumper Tool Useful for tools that can’t handle some projections. April 17, 2012HDF/HDF-EOS Workshop XV38 eos2 ASCII MATLAB IDL NCL

Important Tool for Zoo Maintenance April 17, 2012HDF/HDF-EOS Workshop XV39

Why Tools Need HDF-OES2 Dumper April 17, 2012HDF/HDF-EOS Workshop XV40 Lat / Lon in ASCII MATLAB IDL NCL HDF-EOS2 dumper LaRC MISR SOM Projection HDF-EOS2 Lat / Lon

HDF-EOS2 Dumper Options Usage: eos2dump options filename.hdf What: 1=lat / 2 =lon Format: m=5 values per line April 17, 2012HDF/HDF-EOS Workshop XV41

eos2dump Latitude Example eos2dump –c 1 filename.hdf > latitude.txt What: lat Format: one value per line (default) April 17, 2012HDF/HDF-EOS Workshop XV

eos2dump Longitude Example eos2dump –c2 m filename.hdf > longitude.txt What: lon Format: 5 values per line MATLAB cannot read it. April 17, 2012HDF/HDF-EOS Workshop XV

Got Tools? April 17, 2012HDF/HDF-EOS Workshop XV44

hdf4 Enjoy the NASA HDF Data! April 17, 2012HDF/HDF-EOS Workshop XV45 eos2 eos5 ASCII NETCDF-4 NETCDF-3 Panoply / IDV MATLAB IDL NCL NASA HDF data users

… and observe the Earth! April 17, 2012HDF/HDF-EOS Workshop XV46

The HDF Group HDF/HDF-EOS Workshop XV47 Thank you ! April 17, 2012

Acknowledgements HDF/HDF-EOS Workshop XV48April 17, 2012 This work was supported by Subcontract number under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.

The HDF Group Questions/comments? HDF/HDF-EOS Workshop XV49April 17, 2012