Page 1 Current Status of HDF-EOS, Maintenance, Developments, and Tools Abe Taaheri, Raytheon IIS HDF & HDF-EOS Workshop XI November 2007

Page 2 TOOLKIT / HDF-EOS Support Maintenance support is funded on EMD Evolution through May 2008 One year extension until May 2009 Current staff - One full-time developer for maintaining:  SDP/MTD Toolkit  HDF-EOS2 & 5  HDF-EOS plug-in for HDFView  HDF-EOS2 to GeoTIFF Converter (HEG)  Other Toolkit/HDF-EOS related software  Toolkit/HDF-EOS/HEG helpdesk -One half-time developer for HEG maintenance

Page 3 ECS Support of HDF-EOS Both HDF4 and HDF5 based flavors of HDF-EOS are part of ECS baseline. OS’s: Solaris (9, 10), Irix6.5 (64, n32), Windows XP, Linux (including 64-bit Opteron and Itanium), Mac OS X ( MTD Toolkit, HDF-EOS2 & 5 ) (Dropped support for Solaris 8, HP, HP11, Dec, IBM for next release) adding Mac Intel support for 2007 release and full Cygwin support for 2008 Compilers: Fortran 77/90 & g77/pgf90, C, C++, gcc, g++

Page 4 Downloadable Software and Applications SDP/MTD Toolkit (Science Data Processing Tools) HDF-EOS2 (HDF4 based) HDF-EOS5 (HDF5 based) HE5View* (HDF5 - based browser) EOSView* (HDF4 - based browser) HDFView, Java-based browser (HDF4, HDF5, HDF-EOS 2 and 5 access) heconvert (converts HDF4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents) HEG (GeoTIFF conversion, Spatial subsetting, Field Subsetting, reprojection, stitching, sub-sampling, etc.) * Dropped support for OS updates on July 2006

Page 5 Software Releases Toolkit/HDF-EOS: Last release – March TOOLKIT , HDF-EOS 2.14, HDF-EOS with HDF4.2r1, HDF post5, Szip2.0 Next release – November TOOLKIT , HDF-EOS 2.15, HDF-EOS with new releases of HDF4, HDF5, and SZIP HDF-EOS Plug-in for HDFView: Last release – May 2006 – Version 2.3 Next release – Dec – HDFView2.4 was released October 31 by The HDF Group

Page 6 Software Releases (Cont.) HEG (HDF-EOS2 to GeoTIFF Converter): Last release – July 2007 – Version 2.8 Next release – May 2008 – Version 2.9

Page 7 SDP Toolkit Science Data Production (SDP) Toolkit is a collection of tools used by - Science software developers who produce code to process instrument data. - HDF-EOS users who need to Access metadata in HDF-EOS files Perform Time/Date conversion Access Digital Elevation Model Data Access Ancillary Data, such as Digital Chart of the World database (DCW), Olson World Ecosystem files, DEM data files, etc.

Page 8 SDP Toolkit (cont.) Tools provided in (SDP, MTD) Toolkit: - AA(Ancillary Data Access) - CBP(CELESTIAL Body Position) - CSC(Coordinate System Conversion) - CUC(Constant and Unit Conversions) - DEM(Digital Elevation Model Access) - EPH(Ephemeris Data Access) - GCT(Geo Coordinate Transformation) - IO(Input Output (File I/O)) - MEM(Memory Management) - MET(Metadata Access) - PC(Process Control) - SMF(Status Message File (Error/Status)) - TD(Time and Date Conversion)

Page 9 What is HDF-EOS ? HDF-EOS is an extension to HDF which standardizes storage of and access to common Earth Sciences data structures – By providing a single interface to data structures common to earth science – By providing a container for EOS inventory, archive and product specific metadata – By establishing a consistent relationship between geolocation and science data.

Page 10 HDF-EOS 2 HDF4 - based, storage format for EOS standard products. Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments Support for Grid/Point/Swath structures HDF4 and HDF-EOS 2 will be supported by THG/NASA for the fore-seeable future.

Page 11 HDF-EOS 5 Based on HDF5, a complete rewrite of HDF-EOS2 with a different user interface. – First released in – Format used by EOS Aura instruments Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. – Supports same data structures – Added prefix ‘HE5_’ to HDF-EOS 2 functions.

Page 12 HDFView HDFView: A Java-based browser from THG for HDF4 and HDF5 – based data. HDF-EOS plug-in extends HDFView to browse HDF- EOS data in both versions. The extension is a plug-in module and does not change THG HDFView functionality. All copy, save, editing, image display, etc. for HDF4 and 5 objects remains. Identical look and feel when displaying HDF and HDF-EOS objects. Users can access a complete breakdown of HDF- EOS objects. e.g. grid information, projection info., dimensions, datafields, and attributes are displayed. Current Support: SUN, SGI, LINUX, Mac, WINDOWS platforms

Page 13 Swath, Grid, and Point Objects

Page 14 AMSR-E and OMI Point objects

Page 15 AMSR-E Swath New Features in Next Release Version 2.4

Page 16 HEG: Utility that converts EOSDIS data from HDF-EOS format to common Geographical Information System (GIS)-compatible formats. Two versions available : – A downloadable desktop version – Access through NASA archive online storage (Data Pools) – Implemented also in prototypes for Stand-alone Data Pool, OGC (Open Geospatial Consortium) for Web Services Chaining, and DOWS (Deploy OGC Web Services) on the DAACs HDF-EOS to GeoTIFF Converter (HEG)

Page 17 HEG Functionality Conversion: – HDF-EOS (Swath or Grid ) to a single-band or multi- band GeoTIFF’s – HDF-EOS Swath to HDF-EOS Grid and Swath – HDF-EOS Swath or Grid to generic Binary ( with metadata file ) Subsetting ( spatial, field, band ) Stitching ( mosaicing ) + subsetting + Reprojecting Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ, GEO, SIN, Albers ) Subsampling ( Subsample stacks ) Running from Command Line (batch jobs) or with Java-based GUI

Page 18 HEG Functionality Metadata preservation/creation. Currently supports MODIS, MISR, ASTER, AIRS and AMSR-E products on TERRA and AQUA (>110 products), Allows file selection from users local storage Operable on Sun, SGI, Win, Linux, MAC Integrated into ECS Data Pool – Reduces the transfer time of HDF-EOS data sets (if subset is requested) – Provides the end-user with the exact file required by their application User Interface – Portable, written in Java. – Not dependent on COTS (eg. IDL).

Page 19 Mt. Etna Eruption: MISR L1B Stitched Blocks

Page Hurricane Francis, MODIS MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5

Page 21 AST_L1B, 2003 California Fires Geo. projection, Multi-Band GeoTIFF, (3 bands: Visible Near Infrared (VNIR)) Two Swaths Stitched ….. ….. and Subsetted

Page 22  SZIP Insatallation: Default decoder, Choice for encoder  Choice for configuring HDF4 with/without SZIP  Choice for disabling HDF4’s internal netCDF  Choice for installing Toolkit/HDF-EOS in 32 or 64-bit modes in 64-bit Linux platforms  Support for new releases of HDF4/HDF5/SZIP (HDF4.2r2, HDF , SZIP-2.1)  Installation fix for a problem caused by limits.h in some Linux platforms  Auto Configuration capability (like HDF) added for both HDF-EOS2 and HDF-EOS5  Support for Mac Intel platform. Drop of support for Sun5.8, Dec, HP, HP 11, IBM  Fixes for Lat/Lon to pixel conversions in DEM tools for Linux to get results similar to those in Unix platforms Major Development/Fixes Next Release: November 2007

Page 23 Major Development/Fixes (cont.) HDF-EOS Plug-in for HDFView: Current Support: SUN, SGI, LINUX, WINDOWS, and MAC platforms Last Release: Version 2.3 – Display Point Objects – Display pixel's lat/lon for swath and grid images – Scientific notation in Table data Next Release: Version 2.4 – Auto installation of plug-in into HDFView running self- extracting archives. – Display image for selected pixels in the viewed tables – Display Lat/Lon for a selected pixel in the data field tables – Mac OSX fixes for one-button mice – Better display of Plug-in User’s Guide – New icons for activating Lat/Lon display for Swath & Grid

Page 24 Major Development/Fixes (cont.) HEG: Last Release (Version 2.8) – Added Support for: MOD02SSH, MYD02SSH, MOD14A1.v005, MYD14A1.005 MOD29P1D.V005, MOD08_M3.v005, MOD11_L2.v005 AMSR-E Snow and Ice, AMSR-E Ocean – MISR Bug fixes & enhancements:  Added “No-reprojection” for ouput in SOM projection  Added “Nearest Neighbour” for sampling type  Fixed problem with missing metadata for HDF-EOS outputs – Added Albers Projection – New "Subsetter" functionality for Parameters (aka Fields). This results in: Output to HDF-EOS SWATH from input HDF-EOS SWATH No alteration of input data because of projection operation.

Page 25 HDF-EOS/Toolkit Future Plans : Auto Configuration for TOOLKIT Full Support for Cygwin platform Support for HDF5-1.8 Performance improvement, if possible, for reading HDF format ephemeris/attitude files in Toolkit Performance improvement in HDF-EOS5 by redesigning some functions More functionalities in HDF-EOS Plug-in for HDFView: -Format conversion: HDF-EOS2 -> HDF-EOS5 - Improved Help Window -Ability to modify input HDF-EOS2 granules. - Ability to cut/paste objects, modify/delete attributes. - Ability to create new grids/swaths

Page 26 Availability Access to libraries, applications, Documentation : TOOLKIT: HDFView: HEG: – –

Page 27

Page 28 HDF-EOS5:  Added a routine to return list of aliases for a field  Used HDF5’s H5T_C_S1 datatype for string type  Added Fortran wrapper for SWwritedatameta() routine  Replaced symbol “H5E_NONE_MAJOR” with other symbols for better description of errors  Ported to Solaris 9 and Solaris 10  Fixed POINT object for: - linkage between levels -updating levels -writing Forward and Backward Linkage data -char field inquiry Major Development/Fixes Last Release

Page 29  Fixed subsetting problem in HDF-EOS5 Grid (correction to conversion of DMS degree to Decimal degree).  Set LD_LIBRARY_PATH for SZIP/HDF5 shared libraries in installation and environment setting scripts TOOLKIT/HDF-EOS2:  Ported to Solaris 9 and Solaris 10  Gdapi.c changes for adding more significant digits after the decimal point to projection parameters if needed.  chkeph utility fixes for reading binary ephemeris files (files with different endiannes than the machine)  SGI compiler support  orbsim fixes for Linux to create files similar to those in UNIX platforms (Random number generation fixes) Major Development/Fixes (cont.) Last Release