1. Kent Yang, Mike Folk The HDF Group March 31, 2009
ESDIS 2008 tasks status
Kent Yang, Mike Folk
The HDF Group
March 31, 2009
6/26/2018
Annual briefing to ESDIS

2. Annual briefing to ESDIS
Summary of tasks
Total 28 tasks:
Most tasks received from the annual briefing and the workshop
4 NPP/NPOESS tasks transferred to NPOESS project
10 tasks have been finished
6 tasks are on-going activities
8 tasks are in progress
Almost one-to-one mapping
6/26/2018
Annual briefing to ESDIS

3. Finished tasks highlights
6/26/2018
Annual briefing to ESDIS

4. HDF-EOS automatic configuration
Both HDF-EOS2 and HDF-EOS5
Added the szip support
Both encoder and decoder
Decoder only
Without szip
Added Fortran wrapper support
Updated source code under Fortran testdrivers directory
Updated installation files for automatic configuration
6/26/2018
Annual briefing to ESDIS

5. Investigation of HDF Support in Some Open Source Software Packages
6/26/2018
Annual briefing to ESDIS

6. Five open source packages
PyHDF
Python interface to HDF4
Geospatial Data Abstraction Library (GDAL)
Translator library for Raster Geospatial Data Formats
Supports about 100 file formats
NCAR Common Language (NCL)
Interpreted Language for Data Analysis and Visualization
Grid Analysis and Display System (GrADS)
GNU Data Language (GDL)
Data Analysis and Visualization
6/26/2018
Annual briefing to ESDIS

7. Evaluation - File Formats
HDF4
HDF5
NCL
SDS
NetCDF4 classic
GrADS
PyHDF
SDS, Vdata, Vgroup
GDL
SDS, Vdata
General HDF5 objects
GDAL
SDS, GR, EOS2
6/26/2018
6/26/2018
Annual briefing to ESDIS 7 7

8. Evaluation - Documentation
Example
NCL
sufficient
Many examples with screenshots
GrADS
Several screenshots
PyHDF
simple, but sufficient
Several simple examples
GDL
IDL documents
GDAL
Sufficient
A few examples in Tutorial
Working applications included
6/26/2018
6/26/2018
Annual briefing to ESDIS 8 8

9. Evaluation - Installation
binaries
source build
NCL
AIX, IRIX, Linux, MacOS X, Solaris, Cygwin
not tested
GrADS
Linux, MacOS X, SunOS
PyHDF
Requires few settings
GDL
Requires several packages
Assumes HDF4 and HDF5 without SZIP
GDAL
Requires one patch
6/26/2018
6/26/2018
Annual briefing to ESDIS 9 9

10. Annual briefing to ESDIS
Usage example - GrADS
$ grads
ga> xdfopen AE_RnGd.xdf
ga> set lon
ga> set gxout shaded
ga> display RrLandRain
ga> draw title Total Rain Rate over Land in July 2007
6/26/2018
Annual briefing to ESDIS

11. Performance evaluation of using SSHFS-FUSE to access HDF5 files
6/26/2018
Annual briefing to ESDIS

12. Annual briefing to ESDIS
Using NFS
remote system
local system
NFS requires a special permission. General users are not allowed.
NFS
6/26/2018
Annual briefing to ESDIS

13. 1. Downloading a whole file
remote system
local system
ftp, sftp, …
What if only a small part is necessary from a huge file?
local
filesystem
6/26/2018
Annual briefing to ESDIS

14. Annual briefing to ESDIS
2. Using FUSE+SSHFS
remote system
local system
Given that permission to access FUSE is granted, general users can mount remote filesystems.
SSHFS
FUSE
6/26/2018
Annual briefing to ESDIS

15. Accessing HDF5 through SSHFS-FUSE
Experiment:
For 4 files,
Copy a whole file
Copy one dataset in a file
Copy one small hyperslab in a dataset
6/26/2018
Annual briefing to ESDIS

16. Results of the experiment
elapsed time ratio
- SSHFS / downloading a whole file and subsetting
sshfs/download&local
File 1
Whole file
3.42
One dataset
0.23
One hyperslab
0.10
SSHFS
consumed
more time
Almost one-to-one mapping
SSHFS
consumed
less time
6/26/2018
Annual briefing to ESDIS

17. Annual briefing to ESDIS
Other finished tasks
The third dimension support of HDF-EOS
The documentation of HDF-EOS needed to be updated
HDF4.2r4 release
Static library with static external libraries
C shared libraries
HDFView 2.5 release
Fixed bugs of ‘switch base index’ and ‘fill value’
Help build EOS toolkit on solaris
NetCDF4/NetCDF3/HDF5 performance study
6/26/2018
Annual briefing to ESDIS

18. On-going tasks overview
6/26/2018
Annual briefing to ESDIS

19. On-going tasks overview
Applications/other technologies(in separate talks)
Support OPeNDAP
HDF4 handler for OPeNDAP
Independent HDF4 mapping
HDF5 library
Support backward/forward compatibility
Tests with other software packages
Maintain HDF-EOS2 and HDF-EOS5 daily tests
Regression tests with IDL and Matlab
6/26/2018
Annual briefing to ESDIS

20. Annual briefing to ESDIS Annual briefing to ESDIS
Tasks in progress
Some tasks almost finished
Some tasks we need to know the next step
Some tasks are re-scoped into our next year’s plan
6/26/2018
Annual briefing to ESDIS
6/26/2018
Annual briefing to ESDIS 20

21. Annual briefing to ESDIS
Special value RFC
90% done.
We have written the RFC
We then sent out to NASA users and hdf-forum
A decision has been made based on feedback
Provide examples to demonstrate how to handle special values
Hi Kent,
Thank you for your message. I appreciate your taking the time to
communicate with us.
I appreciate even more that your decision was guided by the needs,
concerns, and interests expressed by hdf5 data producers. - Paul
6/26/2018
Annual briefing to ESDIS

22. High Level Language Support
Motivation
A better high-level language support for HDF5 such as Python, Perl, Ruby etc.
Progress
Investigated SWIG-HDF5 interface
Held a discussion in HDF-EOS workshop
Formed a High level language wiki to further discuss the general issue
Suggestions ?
6/26/2018
Annual briefing to ESDIS

23. Simplifying installation procedures
Request
Investigate simplifying build installation procedures, including 3rd party libraries
Progress
The HDF group is investigating the use of RPM to simplify the whole process
A RFC is being written
6/26/2018
Annual briefing to ESDIS

24. Using NetCDF4 APIs to read HDF-EOS5
Motivation:
To allow the access of HDF-EOS5 files via NetCDF4 APIs
To ensure that HDF-EOS5 files still accessed by the HDF-EOS5 library
6/26/2018
Annual briefing to ESDIS

25. Using NetCDF4 APIs to read HDF-EOS5
Progress:
Wrote an investigation report on this issue
Discussed with Aura, netcdf and EOS developers
Finalized a solution
Augmented the EOS5 file to support dimensions
Relaxed creation order requirements of NetCDF4 to access HDF-EOS5 data
A prototyped augmentation tool is implemented
Work to be done:
Wait for NetCDF4 developer to release a snapshot to support the relax of creation order
Write a netCDF4 fortran example to read EOS5 files
6/26/2018
Annual briefing to ESDIS

26. HDF5 to jpeg tool prototype
Request from HDF-EOS5 to support the conversion in HDF-EOS2 to HDF-EOS5 tool
h52jpeg traverses an HDF5 file and saves each HDF5 image found to a jpeg file
h52jpeg is only a prototype
Not fully tested
Not fully documented
Not released with HDF5 library
Source code is available only at a special branch
Is there any request to make this tool fully supported?
6/26/2018
Annual briefing to ESDIS

27. Command line format conversion tools
Request:
In order of priority - netCDF, geotiff, grib and ascii
Progress:
This is a big task.
We sent out a newsletter to ask if there are any available tools; no useful responses received.
We were implementing an HDF-EOS5 to netCDF4 conversion tool.
Questions:
Is this task still necessary?
If yes, what’s the priority of this task?
6/26/2018
Annual briefing to ESDIS

28. EOS access example codes and tools
Requests:
Tools such as HEG, IDL , Matlab etc.
Well-commented example codes
Progress:
Some experimental work has been done
Plan:
The work is re-scoped to be the focus of the new year
6/26/2018
Annual briefing to ESDIS

29. Review dimension scale model
Requests:
Review dimension scale model and identify use cases
Plan:
We will start investigating this after the briefing
Any suggestions?
6/26/2018
Annual briefing to ESDIS

30. Annual briefing to ESDIS
Thank You!
6/26/2018
Annual briefing to ESDIS

31. Annual briefing to ESDIS
Acknowledgements
This work was supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX08A077A.
Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.
6/26/2018
Annual briefing to ESDIS

32. Annual briefing to ESDIS
Questions/comments?
6/26/2018
Annual briefing to ESDIS