1. Page 1 HDF-EOS Tools Abe Taaheri, Raytheon IIS ESIP Meeting Chapel Hill, NC July 9, 2013

2. Page 2 Outline Overview of current support under EED project Need for HDF-EOS tools HEG & hdf-eos Plug-in for HDFView Demo ?

3. Page 3 Overview The HDF has been selected by the EOSDIS Project as the format of choice for standard product distribution of EOS data Under NASA contracts (ECS, EMD, EEB, EED) we developed HDF-EOS, and performed other development and maintenance activities

4. Page 4 Overview – Libraries  HDF-EOS 2 & 5  SDP & MTD Toolkits – Tools  HDF-EOS2 to GeoTIFF Converter (HEG)  HDF-EOS plug-in for THG HDFView  HDF-EOS2 to HDF-EOS5 conversion – User support  Helpdesk for Toolkit/HDF-EOS/HEG

5. Page 5 Tools Why we need some HDF-EOS tools ?

6. Page 6 HDF was selected for EOS data since – Many of the HDF defined datatypes map well to EOS datatypes  Raster images, multi-dimensional arrays, and text blocks – HDF provides efficient data management  Storage & performance Why HDF?

7. Page 7 Some other EOS datatypes do not map directly to HDF datatypes, particularly ̶ geolocated datatypes Therefore, some additions to traditional HDF are required to fully support these datatypes. HDF-EOS Datatypes

8. Page 8 HDF-EOS Datatypes To bridge the gap between the needs of EOS data products and the capabilities of HDF, 4 new EOS specific datatypes were introduced:  Point, Swath, and Grid  Zonal Average (hdf-eos5)

9. Page 9 HDF-EOS Datatypes Point : Unorginzed data (spatial or temporal) that has associated geolocation information Swath: Time-ordered data such as satellite swaths (time-ordered series of scanlines), or profilers (time-ordered series of profiles). Grid: Data stored in a rectilinear array based on a well defined and explicitly supported projection. ZA: Swath like datatype without geolocation mapping

10. Page 10 HDF4 File Contents – User View Objects & Relationships User Metadata Object Data

11. Page 11 HDF4 File Contents – Format View Vgroup name = variable_name class = Var0.0 NDG SDD SD NT variable name = variable_name rank type storagetype data Vdata name = attribute_name class = Attr0.0 1 1 0…* 1 1 attribute name = attribute_name 1 1 1 1 1 1 1 1 1 1 1 0...1 1 byte order, chunked storage, compression, … Object Data HDF4 Mapping Project Update - Ruth Aydt

12. Page 12 HDF/HDF-EOS are Complicated! HDF is not efficient for Users since they need to learn - how to use HDF & HDF-EOS libraries - how to process the data correctly Licenses for commercial tools are expensive The HDF Group & Raytheon have developed some tools to help users

13. Page 13 HDF & HDF-EOS Tools For most users HDF readers or NetCDF/CF conversion applications are enough - hdp, h5dump, eos2dump(for lat/lon), HDFView - h4tonccf, h4cf, OpeNDAP Handlers But some need more processing and tools appropriate for EOS data HDF-EOS to GeoTIFF Conversion Tool (HEG) HDF-EOS plug-in for HDFView

14. Page 14

15. Page 15  Name says HEG is: A utility to convert EOSDIS data from HDF-EOS2 to GeoTIFF format.  But in practice: It can do more than just conversion to GeoTIFFHEG

16. Page 16 HEG Functionality Conversion – HDF-EOS (Swath or Grid ) to a single-band or multi-band GeoTIFF’s – HDF-EOS Swath to HDF-EOS Grid --> GeoTIFF OR – ASTER Swath is written directly into GeoTIFF – HDF-EOS Swath or Grid to generic Binary ( with metadata file )

17. Page 17 HEG Functionality Subsetting – spatial: using Swath/Grid corner lat/lon or pixel numbers – field: Grid, Swath, and Field selection – band: Selection of data layers in 3-D and 4-D datasets ( 3-D to 6-D MISR SOM datasets )

18. Page 18 HEG Functionality Stitching – ( mosaicing ) + subsetting + Reprojecting Reprojection ̶ To: UTM, PS, TM, STP, LCC, LAMAZ, GEO, SIN, Albers, CEA Subsampling ̶ Subsample stacks

19. Page 19 HEG Functionality Running from Command Line (batch jobs) or with Java-based GUI Metadata preservation/creation.

20. Page 20 Conversion GUI HEG

21. Page 21 HEG Stitch GUI

22. Page 22 Bilinear Nearest Neighbor Cubic Convolution Geographic Polar Stereographic UTM Mercator Transverse Mercator Lambert Azimuthal Lambert Conformal Conic Sinusoidal Albers State Plane Cylindrical Eqyual-Area No-Reprojection Conversion GUI HEG Lat-Lon Rows-Columns

23. Page 23 HEG HEG Parameter File Used for running HEG on command line.

24. Page 24 Status log for HEG run

25. Page 25 HEG Data/Platform Support Currently supports MODIS, MISR, ASTER, AIRS, AMSR-E products on TERRA and AQUA – more than 210 products Operable on Sun, Windows, Linux, MAC

26. Page 26  Versions Available:  Stand-alone: A downloadable desktop version with a User Interface ̶ Portable, written in Java. ̶ Not dependent on COTS (eg. IDL). HEG Versions

27. Page 27 HEG Versions  Integrated into ECS Data Pool: Access through NASA archive online storage ̶ Reduces the transfer time of HDF-EOS data sets (if subset is requested) ̶ Provides the end-user with the exact file required by their application

28. Page 28 Examples (MISR) Mt. Etna Eruption: MISR L1B Stitched Blocks

29. Page 29 Examples (MODIS) MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5 2004 Hurricane Francis, MODIS

30. Page 30 Examples (ASTER) Two Swaths Stitched ….. ….. and Subsetted AST_L1B, 2003 California Fires Geo. projection, Multi-Band GeoTIFF (3 bands): Visible Near Infrared (VNIR)

31. Page 31 Examples ( Examples ( AMSR_E_L2_Ocean) swath Swath Conversion to Geographic Grid/HDF-EOS output

32. Page 32 Future Work

33. Page 33

34. Page 34 HDFView A Java-based browser from The HDF Group for HDF4 and HDF5 based data.

35. Page 35 HDFView with HDF-EOS Plug-in

36. Page 36 HDFView with HDF-EOS Plug-in – Extends HDFView to browse HDF-EOS data in both versions (2 & 5). – Is a plug-in module does not change HDFView functionality – Identical look and feel when displaying HDF and HDF-EOS objects – Access to complete breakdown of HDF-EOS objects. e.g. grid info, projection info, dims, datafields, and attributes

37. Page 37 HDFView Plug-in What else HDF-EOS plug-in adds to HDFView?

38. Page 38 Swath, Grid, and Point Objects 38 Plug-in file treeview Swath/Grid Lat/Lon display

39. Page 39 HE2 to HE5 Conversion One click to convert HE2 to HE5 file format 39

40. Page 40 AMSR-E Swath 40 Getting Image of selected pixels in a table Getting lat/lon for a selected pixel in a table

41. Page 41 MODIS (HE2) and OMI (HE5) Grids 41 Drawing Coastlines & International Borders on HE2 & HE5 Grids

42. Page 42 Coastlines & International borders for HE2 & HE5 swath Adding single-band/multi-band GeoTIFF writer Adding GeoTIFF reader with geolocating capability Future work

43. Page 43 Availability Download, Documentation, Help : HDFView: http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/HDFView_hdfeos_plugin.html HEG: http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html

44. Page 44 Help, Comments Questions, Comments, Requests:  Abe_Taaheri@raytheon.com  PGSTLKIT@raytheon.com forums/index.php  http://hdfeos.org/ forums/index.php

45. Page 45 Thank You! Questions/comments?