1. Access HDF5 Datasets via OPeNDAP’s Data Access Protocol (DAP)
I will introduce a new joint project between the HDF Group and OPeNDAP. The goal of this project is to provide a server and utility so that people can have access to HDF5 Datasets via OPeNDAP.
MuQun Yang
The HDF Group
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

2. HDF and HDF-EOS Workshop X, Landover, MD
OPeNDAP
A software framework that allows simple access to remote data
Client-server model
Servers make the local data accessible remotely
Clients make applications easily access to remote served data
Widely used by Earth Science Community
URL:
First a brief introductions of OPeNDAP.
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

3. HDF and HDF-EOS Workshop X, Landover, MD
The Data Request
Data are requested with a URL.
Protocol Machine name OPeNDAP server Directory File name
?sst[10:10][0:90][0:180]
Constraint
The user can impose a constraint on the data that s/he would like to acquire from a data set by appending a constraint expression to the end of the URL.
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

4. HDF and HDF-EOS Workshop X, Landover, MD
OPeNDAP Data Types
OPeNDAP provides three basic data objects:
data descriptor structure (.dds) - syntactic metadata - rigid
data attribute structure (.das) - semantic metadata - flexible
data (.dods) - the actual data in a binary structure
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

5. HDF and HDF-EOS Workshop X, Landover, MD
OPeNDAP Servers
Data
Matlab
HDF4
JDBC
FreeFrom
FITS
CDF
CEDAR
netCDF
DSP
JGOFS
Tables
SQL
Flat
Binary
CODAR
ESML
General
No fully-supported HDF5 server
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

6. HDF and HDF-EOS Workshop X, Landover, MD
OPeNDAP Clients
netCDF C
netCDF Java
IDV
Ferret
GrADS
VisAD
ncBrowse
Matlab
Excel
IDL
Access
Client
No HDF5 client
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

7. Why HDF5-OPeNDAP server and client?
Users!
Users!!
Users!!!
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

8. HDF and HDF-EOS Workshop X, Landover, MD
A User Case
Task:
10 elements of 17 MB HDF5 data
200 elements of 10 MB NetCDF data
150 elements of 5 MB HDF4 data
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

9. HDF and HDF-EOS Workshop X, Landover, MD
General approach
HDF5 data
FTP
HDF4 data
FTP
NetCDF data
FTP
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

10. HDF and HDF-EOS Workshop X, Landover, MD
Approach with OPeNDAP
HDF5 data
FTP
HDF4 data
server
DAP
client
NetCDF data
DAP
server
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

11. HDF and HDF-EOS Workshop X, Landover, MD
Conclusion
HDF5-DAP server is required
HDF5-DAP client is strongly recommended
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

12. Previous work - A Prototype of DAP2-HDF5 server
Incomplete mapping from HDF5 to OPeNDAP
No comprehensive test suite
Only tested a few examples on two platforms
Incomplete documentation
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

13. HDF and HDF-EOS Workshop X, Landover, MD
An example from SSMI
From HDF4
server at JPL
From HDF5
server at NCSA
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

14. Current HDF5-OPeNDAP Support
No Client
Prototyped Server
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

15. HDF and HDF-EOS Workshop X, Landover, MD
HDF5-OPeNDAP Project
Two-year project awarded by NASA ROSES NRA
Funding much less than expected
Original proposed work kept
with significant contributions from ESDIS
Starting date: Nov. 15th, 2006
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

16. Proposed Work – software development
Implementation
HDF5-DAP2 server
HDF5-DAP4 server
Deployment
Release servers to OPeNDAP community
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

17. HDF and HDF-EOS Workshop X, Landover, MD
HDF5-DAP2 server
Update and complete the prototype, including make files, instructions, documentation, etc
Add a few selected features to handle NASA Aura data and NPOESS data
Map HDF5 group to DAP(?)
Map Object/regional reference to DAP(?)
Implement a comprehensive test suite
Test on platforms supported by OPeNDAP
Demonstrate with sample NASA data
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

18. Fully-Supported HDF5 Server
DAP
HDF5 server
HDF5
Matlab/Ferret Client
DAP
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

19. Proposed Work – software development
Implementation
DAP4 to HDF5 conversion utility
Deployment
Release utility to OPeNDAP community
Design study
Feasibility of DAP-aware HDF5 library
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

20. Proposed Work - DAP to HDF5 utility
A command line tool that converts DAP to HDF5 file
individual DAP variable
subsetting
Conversion tool
HDF5
HDF5 server
DAP
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

21. Proposed work - documentations
DAP4 specification
HDF5-DAP4 mapping document
DAP4 to HDF5 specification
RFC for DAP-aware HDF5 library
Submit specifications to ESDSWG
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD

22. HDF and HDF-EOS Workshop X, Landover, MD
Acknowledgement
This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNG05GC60A.  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. 
11/24/2018
HDF and HDF-EOS Workshop X, Landover, MD