1 HDF-EOS APIs, tools, etc. Larry Klein, Abe Taaheri, and Cid Praderas L-3 Communications Government Services, Inc. November 30, 2005

2 What is HDF-EOS ? HDF (Hierarchical Data Format) is the format chosen by NASA as the standard for the EOS program HDF-EOS is an extension to HDF which standardizes storage of and access to common Earth Sciences data structures –By providing a single interface to data structures common to earth science –By providing a container for EOS inventory, archive and product specific metadata –By establishing a consistent relationship between geolocation and science data.

3 HDF-EOS Grid Structure Usage - Data which is organized by regular geographic spacing, specified by projection parameters. Structure –Any number of 2-D to 8-D data arrays per structure, one per data type (e.g. temperature) –Geolocation information contained in projection formula, coupled by structural metadata. –Any number of Grid structures per file allowed.

4 HDF-EOS Swath Structure Data which is organized by time, or other track parameter. Spacing can be irregular. Structure –Geolocation information stored explicitly in Geolocation Field (2-D array) –Data stored in 2-D or 3-D arrays –Time stored in 1-D or 2-D array, TAI is standard. –Geolocation/science data connected by structural metadata

5 Swath Data Example

6 HDF-EOS Point Structure Data is specified temporally and/or spatially, but with no particular organization Structure –Tables used to store science data at a particular Lat/Long/Height –Up to eight levels of data allowed. Structural metadata specifies relationship between levels.

7 Zonal Average Structure Generalized array structure with no geolocation linkage Data can be organized by time or track parameter Data spacing can be irregular Structure –Data stored in multidimensional arrays –Time stored in 1-D or 2-D array

8 HDF-EOS 2 HDF4 - based, storage format for EOS standard products. Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments Support for Grid/Point/Swath structures HDF4 and HDF-EOS 2 will be supported by NCSA/NASA for the fore-seeable future.

9 HDF-EOS 5 Based on HDF5, a complete rewrite of HDF4 with a different user interface. –First released in –Format used by EOS Aura instruments Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. –Supports same data structures –Added prefix ‘HE5_’ to HDF-EOS 2 functions. –Doesn’t preclude HDF5 functionality.

10 Top Level of HDF-EOS 5 File STRUC. METADATA HDFEOS SWATH GRID POINT Root -- “/” ADDITIONAL Global (File) Attributes Global (File) Attributes HDFEOS INFORMATION HDFEOS INFORMATION ARRAY ECS METADATA

11 Basic HDF-EOS Functions Access - Open, create, close objects/files Definition - Set key parameters of data sets Input/Output - Read/Write data Inquiry - Return information about data Subset - Read data from specified region

12 File/Object Access GRID: HE5_GDopen/he5_gdopen HE5_GDcreate/he5_gdcreate HE5_GDattach/he5_gdattach HE5_GDdetach/he5_gddetach HE5_GDclose/he5_gdclose

13 File/Object Access POINT: HE5_PTopen/he5_ptopen HE5_PTcreate/he5_ptcreate HE5_PTattach/he5_ptattach HE5_PTdetach/he5_ptdetach HE5_PTclose/he5_ptclose

14 File/Object Access SWATH: HE5_SWopen/he5_swopen HE5_SWcreate/he5_swcreate HE5_SWattach/he5_swattach HE5_SWdetach/he5_swdetach HE5_SWclose/he5_swclose

15 File/Object Access ZA: HE5_ZAopen/he5_zaopen HE5_ZAcreate/he5_zacreate HE5_ZAattach/he5_zaattach HE5_ZAdetach/he5_zadetach HE5_ZAclose/he5_zaclose

16 Definition GRID: HE5_GDdefdim/he5_gddefdim HE5_GDdefproj/he5_gddefproj HE5_GDdeforigin/he5_gddeforigin HE5_GDdefpixreg/he5_gd_defpreg

17 Definition POINT: HE5_PTdeflevel/he5_ptdeflevel HE5_PTdeflinkage/he5_ptdeflinkage

18 Definition SWATH: HE5_SWdefdim/he5_swdefdim HE5_SWdefdimmap/he5_swdefmap HE5_SWdefidxmap/he5_swdefimap HE5_SWdefgeofield/he5_swdefgfld HE5_SWdefdatafield/he5_swdefdfld

19 Definition ZA: HE5_ZAdefdim/he5_zadefdim HE5_ZAdefine/he5_zadefine HE5_ZAdefcomp/he5_zadefcomp HE5_ZAdefchunk/he5_zadefchunk

20 Input/Output GRID: HE5_GDwritefield/he5_gdwrfld HE5_GDreadfield/he5_gdrdfld HE5_GDwriteattr/he5_gdwrattr HE5_GDreadattr/he5_gdrdattr

21 Input/Output POINT: HE5_PTwritelevel/he5_ptwritelevel HE5_PTreadlevel/he5_ptreadlevel HE5_PTwriteattr/he5_ptwrattr HE5_PTreadattr/he5_ptrdattr

22 Input/Output SWATH: HE5_SWwritefield/he5_swwrfld HE5_SWreadfield/he5_swrdfld HE5_SWwriteattr/he5_swwrattr HE5_SWreadattr/he5_swrdattr

23 Input/Output ZA: HE5_ZAwrite/he5_zawrite HE5_ZAread/he5_zaread HE5_ZAwriteattr/he5_zawrattr HE5_ZAreadattr/he5_zardattr

24 Inquiry GRID: HE5_GDinqdims/he5_gdinqdims HE5_GDdiminfo/he5_gddiminfo HE5_GDprojinfo/he5_gdprojinfo HE5_GDorigininfo/he5_gdorigininfo HE5_GDpixreginfo/he5_gdpreginfo

25 Inquiry POINT: HE5_PTlevelinfo/he5_ptlevelinfo HE5_PTinqattrs/he5_ptinqattrs HE5_PTattrinfo/he5_ptattrinfo

26 Inquiry SWATH: HE5_SWinqdatafields/he5_swinwdflds HE5_SWinqgeofields/he5_swinqgflds HE5_SWinqattrs/he5_swinqattrs HE5_SWattrinfo/he5_swattrinfo

27 Inquiry ZA: HE5_ZAinquire/he5_zainquire HE5_ZAinfo/he5_zainfo HE5_ZAinqattrs/he5_zainqattrs HE5_ZAattrinfo/he5_zaattrinfo

28 Programming Model Open the HDF-EOS5 file Create or Attach to a specified Grid/Swath/Point/ZA object Define key object features and/or perform necessary operations (with the data sets, attributes, etc) Detach from the Grid/Swath/Point/ZA object Close the HDF-EOS5 file

29 HDF-EOS Functionality Basic File I/O Swath Interface Grid Interface Point Interface Profile Interface Generalized Array Interface (Zonal Avg. ( HDF5 )) Subsetting Fill Values Compression (gzip, szip, shuffling) Chunking/Tiling Aliases Global (File), Group & Local Attributes Threadsafe Version

30 Applications HE5View (HDF5 - based browser) EOSView (HDF4 - based browser) HDFView (with HDF-EOS Plug-in), Java EOS Browser (HDF-EOS 2 and 5 access) heconvert (converts HDF4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents) –Compliments HDF4 -> HDF5 conversion tool –Not tested on all products HEG (HDF-EOS to GeoTIFF Converter/Subsetter)

31 HDFView HDFView: A Java-based browser from NCSA for HDF4 and HDF5 – based data. HDF-EOS plug-in extends HDFView to browse HDF-EOS data in both versions. –One reason: EOSView is becoming obsolete The extension is a plug-in module and does not change NCSA HDFView functionality. All copy, save, editing, image display, etc. for HDF4 and 5 objects remains. Identical look and feel when displaying HDF and HDF-EOS objects. Users can access a complete breakdown of HDF- EOS objects. e.g. grid information, projection information, dimensions, datafields, and attributes are displayed.

32 HDFView (Cont.) Release 2.2 October A separate HDF-EOS plug-in: Users install HDFView 2.2 and integrates HDF-EOS plug-in into it. - Ability to do "Save as" for HDF-EOS5: modify individual values in grid or swath data. Delete entire grids or swaths. Save the result. - Separate help window for HDF-EOS objects Release 2.3 January Point Object viewing capability - Geolocating pixels for swath and grid images  Future Plans: - Ability to modify input HDF-EOS2 granules. - Ability to cut/paste objects, modify/delete attributes. - Ability to create new grids/swaths

33 HDFView showing ASTER L1B swath

34 A utility that converts EOSDIS data from HDF-EOS format to common Geographical Information System (GIS)-compatible formats. Two versions available: –A downloadable desktop version –Access through NASA archive online storage (Data Pools) –Implemented also in prototypes for Stand-alone Data Pool, OGC (Open Geospatial Consortium) for Web Services Chaining, and DOWS (Deploy OGC Web Services) on the DAACs Details of internal file structures are transparent to users. HDF-EOS to GeoTIFF Converter (HEG)

35 HEG Functionality Subsetting ( spatial, field, band ) Performs conversion to a single-band or multi- band GeoTIFF’s HDF-EOS Swath to HDF-EOS Grid and Swath HDF-EOS Swath or Grid to GeoTIFF HDF-EOS Swath or Grid to generic Binary ( with metadata file ) Stitching ( mosaicing ) + subsetting + Reprojecting Running from Command Line (batch jobs) or with Java-based GUI Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ, GEO, SIN ) Subsampling ( Subsample stacks )

36 HEG Functionality Metadata preservation/creation. Currently supports MODIS, MISR, ASTER, AIRS and AMSR-E products on TERRA and AQUA (>110 products), AURA/HDF-EOS5 support in progress Allows file selection from users local storage Operable on Sun, SGI, Win, Linux (64-bit soon), MAC Integrated into ECS Data Pool –Reduces the transfer time of HDF-EOS data sets (if subset is requested) –Provides the end-user with the exact file required by their application User Interface –Portable, written in Java. –Not dependent on COTS (eg. IDL).

37 Availability Access to libraries and applications:

38 Mt. Etna Eruption : MISR L1B Swaths: Stitched

39 Mt. Etna : Stitched, Subsetted ASTER L1B

40 MODIS L1B: Reprojected, Co-Registered with ASTER, La Plata Tornado

41 La Plata Tornado Damage: ASTER L1B reprojected/stitched/subsetted

42 MLS L2GP-H2O Product (2004, day 243), Geographic Projection, Full Globe

43 MLS L2GP-H2O Product, PS Projection, Northern Hemisphere

Hurricane Francis, MODIS (MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5)

45 AST_L1B, 2003 California Fires Geo. projection, Multi-Band GeoTIFF, (3 bands: Visibile Near Infrared (VNIR)) Two Swaths Stitched ….. ….. and Subsetted