1. www.hdfgroup.org The HDF Group HDF/HDF-EOS Workshop XV1 Tools to Improve the Usability of NASA HDF Data Kent Yang and Joe Lee The HDF Group April 17, 2012

2. www.hdfgroup.org Tutorial Outline Introduction 5 Tools 1.H4CF Conversion Toolkit (h4cf) 2.h4toh5 Conversion Tool (h4toh5) 3.HDF-EOS5 to netCDF-4 Converter (eos52nc4) 4.HDF-EOS5 Augmentation Tool (aug_eos5) 5.HDF-EOS2 Dumper (eos2dump) Demo HDF/HDF-EOS Workshop XV2April 17, 2012

3. www.hdfgroup.org Who Said Users Don’t Matter? April 17, 2012HDF/HDF-EOS Workshop XV3

4. www.hdfgroup.org NASA HDF/HDF-EOS Data Usability of NASA HDF Data April 17, 2012HDF/HDF-EOS Workshop XV4 IDLMATLAB Panoply IDV A school teacher’s view Panoply IDV A school teacher’s expectation

5. www.hdfgroup.org The 5 Tools April 17, 2012HDF/HDF-EOS Workshop XV5 Improve the usability of NASA Data Supported by The HDF Group Funded by NASA / Raytheon http://hdfeos.org/software/tool.php

6. www.hdfgroup.org hdf4 5 Tools that Improve Usability April 17, 2012HDF/HDF-EOS Workshop XV6 eos2 eos5 ASCII NETCDF-4 NETCDF-3 Panoply / IDV MATLAB IDL NCL

7. www.hdfgroup.org H4CF Conversion Toolkit Follow CF convention Both Library and Tools http://hdfeos.org/software/h4cflib April 17, 2012HDF/HDF-EOS Workshop XV7 hdf4 eos2 NETCDF-4 NETCDF-3 h4cf Panoply / IDV

8. www.hdfgroup.org H4CF Conversion Toolkit Two ways to build: 1.Use configure script. 2.Edit Makefile template. April 17, 2012HDF/HDF-EOS Workshop XV8

9. www.hdfgroup.org H4CF Conversion Library Variables (datasets) list Attributes (metadata) list All Vars & Attrs follow CF conventions Develop your own CF application! April 17, 2012HDF/HDF-EOS Workshop XV9

10. www.hdfgroup.org H4CF Library APIs Example // open the example HDF file h4cf_open("geo.hdf"); // HDF file attributes are obtained in a C++ STL map object const map file_attrs = h4cf_get_file_attrs(); // HDF file variables are obtained in a C++ STL list object const list pvars = h4cf_get_vars(); // close the HDF file h4cf_close(); April 17, 2012HDF/HDF-EOS Workshop XV10

11. www.hdfgroup.org H4CF Conversion Tools 2 Tools based on H4CF APIs: cdldumper h4tonccf April 17, 2012HDF/HDF-EOS Workshop XV11

12. www.hdfgroup.org cdldumper Usage: cdldumper filename.hdf Dump file in Common Data Language. Useful for quick check and debugging purpose. April 17, 2012HDF/HDF-EOS Workshop XV12

13. www.hdfgroup.org cdldumper Demo April 17, 2012HDF/HDF-EOS Workshop XV13 Number of attributes=1 _FillValue=255

14. www.hdfgroup.org cdldumper Demo April 17, 2012HDF/HDF-EOS Workshop XV14 %./cdldumper -H AMSR_E_L3_5DaySnow_V09_20050126.hdf netCDF AMSR_E_L3_5DaySnow_V09_20050126.hdf { dimensions: XDim_Northern_Hemisphere = 721, XDim_Southern_… variables: ubyte Flags_NorthernPentad(YDim_Northern_Hemis… Flags_NorthernPentad:_FillValue = 255; Flags_NorthernPentad:origname = "/Northern… Flags_NorthernPentad:long_name = "/Northern… Flags_NorthernPentad:coordinates = "latitude… Number of attributes=4 _FillValue=255 orig_name long_name coordinates

15. www.hdfgroup.org h4tonccf Usage: h4tonccf filename.hdf [filename.nc] generates either netcdf-3 or netcdf-4 file. follows netcdf classic model. April 17, 2012HDF/HDF-EOS Workshop XV15

16. www.hdfgroup.org h4tonccf Demo April 17, 2012HDF/HDF-EOS Workshop XV16

17. www.hdfgroup.org Don’t Follow CF Conventions Blindly April 17, 2012HDF/HDF-EOS Workshop XV17 [1] CF: scale * value + offset [2] MODIS: scale * (value – offset) Scale = 0.01 Value = 5000 Offset = -15000.0 [1] 0.01 * 5000 + (-15000) = -14950 [2] 0.01 * (5000 – (-15000)) = 200

18. www.hdfgroup.org -14950 Kelvin? April 17, 2012HDF/HDF-EOS Workshop XV18

19. www.hdfgroup.org MODIS Scale / Offset Correction HDF/HDF-EOS Workshop XV19April 18, 2012

20. www.hdfgroup.org Converted File Size Comparison April 17, 2012HDF/HDF-EOS Workshop XV20 HDF-EOS2 Netcdf-4 Netcdf-3 9X

21. www.hdfgroup.org Limitation – No Support Yet April 17, 2012HDF/HDF-EOS Workshop XV21 MISR (SOM/LAMAZ) MERRA AVHRR LISTO QuikSCAT/SeaWind MOPITT L2 etc.

22. www.hdfgroup.org h4toh5 Conversion Tool Convert any hdf4 to hdf5. http://hdfgroup.org/h4toh5 April 17, 2012HDF/HDF-EOS Workshop XV22 eos2 NETCDF-4 h4toh5 Usage: h4toh5 –eos –nc4 in.hdf out.nc Static Binary w/ –eos & -nc4: http://hdfeos.org/software/h4toh5/bin/

23. www.hdfgroup.org h4tonccf vs. h4toh5 April 17, 2012HDF/HDF-EOS Workshop XV23 h4tonccfh4toh5 w/ -eos –nc4 Handle non-eos2 hdf4 YesNo Follow CF conventions YesNo Output netcdf classic YesNo Output netcdf enhanced NoYes

24. www.hdfgroup.org eos52nc4 converts hdf-eos5 to netcdf-4. follows netcdf enhanced model. does not follow CF conventions. Usage: eos52nc4 test.he5 test.nc http://hdfeos.org/software/convert_hdfeos5.php April 17, 2012HDF/HDF-EOS Workshop XV24 eos5 NETCDF-4 eos52nc4

25. www.hdfgroup.org aug_eos5 augments hdf-eos5 to netcdf-4. Both HDF-EOS5 and NetCDF4 can access it. Usage: aug_eos5 [-i|-f map] test.h5 modifies the input file directly. does not follow CF conventions. http://hdfeos.org/software/aug_hdfeos5.php April 17, 2012HDF/HDF-EOS Workshop XV25 eos5 NETCDF-4 aug_eos5

26. www.hdfgroup.org hdf5 netcdf4 Augmentation Concept April 17, 2012HDF/HDF-EOS Workshop XV26 hdf-eos5 HDF-EOS5 API NETCDF-4 API aug_eos5 augmentation hdf-eos5 HDF5 API

27. www.hdfgroup.org Why NetCDF-4 Doesn’t Work April 17, 2012HDF/HDF-EOS Workshop XV27 [nLevels][nTimes] NETCDF-4 API hdf-eos5

28. www.hdfgroup.org What aug_eos5 Does April 17, 2012HDF/HDF-EOS Workshop XV28 [nLevels][nTimes] NETCDF-4 API hdf-eos5

29. www.hdfgroup.org Options in aug_eos5 1.Default option -i option 2.-f map_file option April 17, 2012HDF/HDF-EOS Workshop XV29

30. www.hdfgroup.org Default option in aug_eos5 Grid XDim, YDim: calculated Other dims: default fill value Swath / ZA All dims: default fill value April 17, 2012HDF/HDF-EOS Workshop XV30

31. www.hdfgroup.org Better -i option in aug_eos5 Grid XDim, YDim: calculated Other dims: 1,2,…,n Swath / ZA All dims: 1,2,…,n April 17, 2012HDF/HDF-EOS Workshop XV31

32. www.hdfgroup.org Best -f option in aug_eos5 Needs an input map file. Input file specifies 3 rules : 1.What to be filled in 2.How to fill in 3.Where to get the values Most useful and flexible option April 17, 2012HDF/HDF-EOS Workshop XV32

33. www.hdfgroup.org aug_eos5 Input File Format Each line in input file contains 1.How: 0,1,2 2.What: dim name 3.Where: data file or existing variable in file. 4.(Optional) Multiple grid/swath/za case only: grid/swath/za name: April 17, 2012HDF/HDF-EOS Workshop XV33

34. www.hdfgroup.org aug_eos5 Input File Example 0 nTimes Time 1 nLevels data.txt 2 nCandidates April 17, 2012HDF/HDF-EOS Workshop XV34 Copy the data Read file data.txt Fill 1,2,…, n HowWhatWhere 95% cases – Copy the data

35. www.hdfgroup.org Multiple Swath Case swath name: BrO 0 nTimes Time 1 nLevels data1.txt 2 nCandidates April 17, 2012HDF/HDF-EOS Workshop XV35 HowWhatWhere swath name: BrO column 2 nTimes 1 nLevels data2.txt Multiple

36. www.hdfgroup.org aug_eos5 vs eos52nc4 April 17, 2012HDF/HDF-EOS Workshop XV36 aug_eos5eos52nc4 hdfeos-5YesNo netcdf-4Yes

37. www.hdfgroup.org HIRDLS/MLS Team Use aug_eos5 April 17, 2012HDF/HDF-EOS Workshop XV37 So You Can!

38. www.hdfgroup.org eos2 dump HDF-EOS2 Dumper Tool Useful for tools that can’t handle some projections. April 17, 2012HDF/HDF-EOS Workshop XV38 eos2 ASCII MATLAB IDL NCL http://hdfeos.org/zoo

39. www.hdfgroup.org Important Tool for Zoo Maintenance April 17, 2012HDF/HDF-EOS Workshop XV39

40. www.hdfgroup.org Why Tools Need HDF-OES2 Dumper April 17, 2012HDF/HDF-EOS Workshop XV40 Lat / Lon in ASCII MATLAB IDL NCL HDF-EOS2 dumper LaRC MISR SOM Projection HDF-EOS2 Lat / Lon

41. www.hdfgroup.org HDF-EOS2 Dumper Options Usage: eos2dump options filename.hdf What: 1=lat / 2 =lon Format: m=5 values per line April 17, 2012HDF/HDF-EOS Workshop XV41

42. www.hdfgroup.org eos2dump Latitude Example eos2dump –c 1 filename.hdf > latitude.txt What: lat Format: one value per line (default) April 17, 2012HDF/HDF-EOS Workshop XV42 90 80 70 60 50 40 30 20 10 0 -10 -20 -30

43. www.hdfgroup.org eos2dump Longitude Example eos2dump –c2 m filename.hdf > longitude.txt What: lon Format: 5 values per line MATLAB cannot read it. April 17, 2012HDF/HDF-EOS Workshop XV43 180 175 170 165 160 155 150 145 140 135 130 125 120 115 110

44. www.hdfgroup.org Got Tools? April 17, 2012HDF/HDF-EOS Workshop XV44

45. www.hdfgroup.org hdf4 Enjoy the NASA HDF Data! April 17, 2012HDF/HDF-EOS Workshop XV45 eos2 eos5 ASCII NETCDF-4 NETCDF-3 Panoply / IDV MATLAB IDL NCL NASA HDF data users

46. www.hdfgroup.org … and observe the Earth! April 17, 2012HDF/HDF-EOS Workshop XV46

47. www.hdfgroup.org The HDF Group HDF/HDF-EOS Workshop XV47 Thank you ! April 17, 2012

48. www.hdfgroup.org Acknowledgements HDF/HDF-EOS Workshop XV48April 17, 2012 This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.

49. www.hdfgroup.org The HDF Group Questions/comments? HDF/HDF-EOS Workshop XV49April 17, 2012