1. HDF5 OPeNDAP Project Update and Demo
MuQun Yang and Hyo-Kyung Lee (The HDF Group)
James Gallagher (OPeNDAP, Inc.)
We’ll present the HDF5 OPeNDAP project update and demo.
This is a joint project between the HDF Group and James Gallagher at OPeNDAP, Inc.
We’ll present an introduction to OPeNDAP and then present how HDF5 can be served effectively in OPeNDAP.

2. OPeNDAP Review
A software framework that allows simple access to remote data
Data Access Protocol
Client-server model
Widely used by Earth Science Community
Now let’s see how the idea of DAP-way can be implemented.
DAP is a mere protocol like FTP and OPeNDAP is an open and free software implementation of DAP-way.
It is a server-client model and provides APIs for both.
And the biggest advantage of OPeNDAP is that it is very popular among Earth Science Community.

3. OPeNDAP Visualization Client
Example Usage
HDF4
OPeNDAP
Client
OPeNDAP
HDF4 Server
Actual Content of Data
Here’s an example usage of OPeNDAP.
First, here is a HDF4 file that you want to allow people to see.
Once the OPeNDAP HDF4 server is installed and running, this file becomes available world-wide through Data Access Protocol and you can view its content with a standard web browser.
Data Access Protocol provides three ways of viewing data.
First you can view the syntactic structure of data.
Second you can view the semantic meaning of data.
Finally, you can view the real content of data you want to retrieve.
Semantic Meaning of Data
Syntactic Structure of Data
OPeNDAP Visualization Client

4. Why Important? HDF5: NASA EOS / NPOESS NASA’s MISSION: ACCESS!
Our MISSION: Build a DAP-HDF5 Bridge
Why is HDF5 part so important?
The main reason is that new satellite data are being stored and will be stored in HDF5 in the future.
And one of the most important mission of NASA is to provide a full access to its huge data center.
So our mission is to bridge the gap between data provider and data user through OPeNDAP.

5. Goals 1st: Access HDF5 data via OPeNDAP
2nd: Construct HDF5 data from OPeNDAP
So, here are our project goals.
First, we want to provide way to access HDF5 in remote location like NASA in DAP-way.
Yet, we want to lose as little as possible during this transformation process.

6. OPeNDAP HDF5 Server Map HDF5 Datatypes into DAP Datatypes
Goal: Access HDF5 data via OPeNDAP
Map HDF5 Datatypes into DAP Datatypes
Plus, tweaks for OPeNDAP Clients
Plus, tweaks for HDF-EOS Files
And, we built the HDF5 server to provide a standard DAP-way of accessing HDF5.
Plus, we did some additional work for some OPeNDAP clients and HDF-EOS files. In the next few slides, we’ll see why such tweaks are necessary.

7. Mapping HDF5 Datatypes Prototype server in 2001
NASA Grant from Nov. 2006
Support for Compound Datatype
Support for Group
Support for References / Links
First Product Release in Mar. 2008
The first effort was made in 2001 and we received a NASA grant in 2006 to make it a robust product.
Mapping HDF5 in DAP-way required matching HDF5 objects with DAP objects. We enhanced the prototype server by adding support for HDF5 objects like Group, Compound and References.
The first product was released in March 2008 through OPeNDAP site.

8. OPeNDAP HDF5 Server Map HDF5 Datatypes in DAP Datatypes
Goal: Access HDF5 data via OPeNDAP
Map HDF5 Datatypes in DAP Datatypes
Plus, tweaks for OPeNDAP Clients
Plus, tweaks for HDF-EOS Files
After building the HDF5 server to provide a standard DAP-way of accessing HDF5,
we did some additional work for some OPeNDAP clients and HDF-EOS files. In the next few slides, we’ll see why such tweaks are necessary.

9. OPeNDAP Clients I hate long variable names.
Not all OPeNDAP clients are created equal!
I hate long variable names.
I need special attributes on dataset.
I care only Grid datatypes.
I care only well-formed attributes.
Some OPeNDAP clients are very picky in terms of what DAP-server can provide.
They may even ask something that the original HDF5 doesn’t have in the attribute.
They simply reject what the standard DAP protocol allows.

10. Tweaks for OPeNDAP Clients
Two configuration options
--enable-short-name
--enable-CF
Thus, we provided some configuration options during installation.
This can make the most picky OPeNDAP client happy.
However, enabling these options are risky due to you may not serve some dataset in HDF5 through server.

11. OPeNDAP HDF5 Server Map HDF5 Datatypes in DAP-way
Goal: Transform HDF5 access in DAP-way
Map HDF5 Datatypes in DAP-way
Plus, tweaks for OPeNDAP Clients
Plus, tweaks for HDF-EOS5 Files
Yet, it was impossible to view HDF-EOS5 directly on a map. We needed an Array to Grid tweaking.

12. HDF EOS Grid with No Dimension data
Not all HDF5 files are created for DAP!
Grid with No Dimension data
Clients expect Grid with Dimension data
Some Attributes stored in Extremely Long String(s) (e.g. StructMetada.0)
Clients expect well-structured attributes
This kind of situation happened with NASA HDF-EOS5 case.
The dataset in HDF EOS files that NASA produce cannot be served directly in DAP-way that vis. clients like.
Since HDF5 data producer like NASA doesn’t have to keep DAP-way in mind, they created HDF-EOS5 files in a way that they feel convenient.

13. Tweaks for HDF-EOS Two more configuration options --enable-eos-grid
--enable-eos-meta
Thus, we believe it’s our job to make the raw HDF5 data into a new form that clients will like.
--enable-eos-grid processes the raw data into Grid that can client can consume easily.
--enable-eos-meta chops the long string into a better format that can client can handle.

14. Day After Server Tweaks
Finally, Happy Clients!
IDL
ncBrowse
Ferret
MATLAB
NCL
IDV
So after applying the tweaks for clients and hdf-eos, our server can serve many clients. Here’s a live demo and movie.
GrADS
ODC

15. Goals 1st : Access HDF5 via OPeNDAP
2nd: Construct HDF5 data from OPeNDAP
Bonus: Help clients to better access EOS Swath data
So we could achieve the first goal.
Achieving the first goal
has some side effects and give another challenge of preservation.

16. HDF5 Served in OPeNDAP Maximize
There are some things money can’t buy. For everything else, there is MasterCardTM.
Minimize
There are some things DAP can’t serve. For everything else, there is _________?
If you ever watched the “priceless” MasterCard commercial, you’ve heard this famous phrase:
“There are …. “.
We have found a similar situation when we tried to serve HDF5 in DAP-way.
If HDF5 is served in DAP-way, it was inevitable to lose some details that the original HDF5 has simply because DAP doesn’t know how to represent.
However, we think it would be great if we can minimize such things and find a clever solution like MasterCard that can cover and deliver maximum features in HDF5 through DAP-way.
Maximize

17. Some Things
(that OPeNDAP HDF5 server can’t serve)
Hard: Opaque, Bitmap, Enum, 64 bit Integer, Variable Length types
Illegal: Reserved characters in DAP are used in Dataset/Group name in HDF5
Here are some limitations that the current server can’t handle.
Certain types like Opaque, Bitmap, Enum are not available in the current DAP protocol so it’s impossible to map them.
Another interesting case is that there are some reserved characters in DAP yet they are perfectly legal in HDF5.
In these cases, DAP protocol itself can be changed to have new additional types or use special hex encoding of characters.

18. For Everything Else Let there be HDF5-Friendly OPeNDAP client library!
Package and Deliver in DAP first.
Then, let the client library handle it.
Concept of Trojan Horse
However, it’s hard to modify the protocol itself since it’s standard so
this is our current solution for everything else in HDF5.
We’d like to make a special HDF5-friendly OPeNDAP client dedicated for our OPeNDAP HDF5 server.
The concept is quite simple. We simply package and deliver them in DAP-way and let the client handle it.

19. Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.
Example: Group in HDF5
HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5
Here’s really one good example.
In DAP, an attribute plays a role like comment in computer programming.
Although DAP doesn’t have any concept of group, the group may play an important role in HDF5 like disambiguation of same variable name.
This entire group structure is being sent as a single attribute in DAP.
Thus, if a user wants to re-construct HDF5 using DAP client, this group attribute is essential and HDF5-friendly OPeNDAP client library should handle it properly.

20. Example: Reference in HDF5
Object / Regional Reference
Map to DAP URL at server
No de-referencing of URL at client library
Important for NPOESS
Another good example is the reference in HDF5.
There are two types of reference in HDF5 – object and regional.
Both object and regional reference can be mapped to a special data type called URL in DAP.
However, the current OPeNDAP client library doesn’t support the de-referencing of the URL.
That is, there is no way for a OPeNDAP client can access the dataset that the URL points to.
This is particularly important for NPOESS data since it uses tons of HDF5 reference inside.

21. HDF5-Friendly OPeNDAP Client Library
One more reason: Help Clients to view Swath
Only ODC can display Swath properly.
No Latitude and Longitude
Easy but Rigid OPeNDAP NC Client Library!
easy: same as netCDF C interface
rigid: follows netCDF model
Our client library must be Easy but Flexible
There’s another reason that we want to pursue HDF5-friendly OPeNDAP client library.
A swath is a 3-D scan of very small region on the earth.
Earth Scientists are not much interested in seeing 2-D Grid data but a vertical profile of a swath like this.
However, among the six OPeNDAP clients that we tried, only one client could display a vertical profile of swath data properly.
The main reason is that many DAP clients impose un-necessary restrictions in the format that the DAP Protocol produces. One such restriction is a CF convention that we’ve discussed before.
Thus, our goal is to build a client library that shifts the burden of meeting client’s requirement from the OPeNDAP HDF5 server to a client.
Courtesy of NASA

22. NC-Friendly OPeNDAP Client Library
HDF5-Friendly OPeNDAP Client Library
View G/R/Swath
???
View NetCDF
Group/Ref/Swath
NetCDF
View HDF5
HDF5
GrADS
GrADS
OPeNDAP
HDF5 Server
OPeNDAP
NC Server
liboc-dap
libnc-dap
dapserver
However, it’s hard to modify the protocol itself since it’s standard so
this is our current solution for everything else.
We’d like to make a special DAP client dedicated for OPeNDAP HDF5 server.
The concept is quite simple. We simply package and deliver them in DAP-way and let the client handle it.
We want to provide a more generic OPeNDAP client library.
libdap

23. Example: dap2h5 ORIGINAL CONSTRUCTED
A test application for the client library
It can construct HDF5 from DAP output
ORIGINAL
CONSTRUCTED
To prove our client library concept, we created a demo application called “dap2h5”.
As a prototype, it has a limited capability but this demo shows how to re-construct group information and dataset from DAP output successfully.
Since it is a prototype, attributes on groups are ignored by “dap2h5” at this point.

24. Demo: GrADS GrADS is modified to use our library
The client library provides grid mapping from swath data
GrADS displays swath through HDF-EOS5 specific client library API calls
Again, to prove the usefulness of our client library, we compiled GrADS with it and the new grads could display MLS level2 swath data directly.
This demo shows the carbon monoxide level near Beijing before and during Olympics. CO level is high but later goes down significantly due to air quality control enforcement by Chinese government.

25. Caution Our Client Library is ONLY PROTOTYPE!
It does NOT support all DAP data types
It does NOT support all AURA files
It does NOT support all Visualization clients
Here’s a summary of our project.
Providing a DAP-way access of HDF5 is easy, efficient and cool.
However, it can lose some information that HDF5 originally has due to the nature of DAP-way transformation.
This requires either the modification of DAP or the creation of HDF5-friendly client.
Since it’s hard to modify the well-established DAP, we think it’s easier to implement HDF5-friendly OPeNDAP client library.
When it’s done properly, we showed that the client library could serve the HDF5 files better and help the visualization clients to visualize HDF5-EOS data directly via DAP.

26. Summary HDF5 Access via OPeNDAP Is Easy / Efficient / Cool
Loses some things
Requires HDF5-friendly OPeNDAP Client Library
HDF5-Friendly OPeNDAP Client Library Prototype
Serves HDF5 better
Benefits visualization clients
Here’s a summary of our project.
Providing a DAP-way access of HDF5 is easy, efficient and cool.
However, it can lose some information that HDF5 originally has due to the nature of DAP-way transformation.
This requires either the modification of DAP or the creation of HDF5-friendly client.
Since it’s hard to modify the well-established DAP, we think it’s easier to implement HDF5-friendly OPeNDAP client library.
When it’s done properly, we showed that the client library could serve the HDF5 files better and help the visualization clients to visualize HDF5-EOS data directly via DAP.

27. Future Work HDF5 to DAP2 Mapping Document
Release HDF5-friendly OPeNDAP Client Library Prototype
URL:
Here’s our future work.
First, we’ll give a detailed document on mapping between HDF5 and DAP.
Second, we’ll finish and release the HDF5-Friendly OPeNDAP Client Library prototype and test it with one more client like NCL.
By the way, the library will remain as only prototype due to the limited funding availability.

28. Credits Mike Folk (THG) Robert McGrath (NCSA)
Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)
Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang (NASA)
Jennifer Adams (GrADS)
Dave Brown (UCAR)
We’d like to thank these people.
They have provided us the right direction in development, an early access to data files and numerous feedback.

29. Acknowledgement
This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX06AG75A. .  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.