HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) We’ll present the HDF5 OPeNDAP project update and demo. This is a joint project between the HDF Group and James Gallagher at OPeNDAP, Inc. We’ll present an introduction to OPeNDAP and then present how HDF5 can be served effectively in OPeNDAP.

OPeNDAP Review A software framework that allows simple access to remote data Data Access Protocol Client-server model Widely used by Earth Science Community Now let’s see how the idea of DAP-way can be implemented. DAP is a mere protocol like FTP and OPeNDAP is an open and free software implementation of DAP-way. It is a server-client model and provides APIs for both. And the biggest advantage of OPeNDAP is that it is very popular among Earth Science Community.

OPeNDAP Visualization Client Example Usage HDF4 OPeNDAP Client OPeNDAP HDF4 Server Actual Content of Data Here’s an example usage of OPeNDAP. First, here is a HDF4 file that you want to allow people to see. Once the OPeNDAP HDF4 server is installed and running, this file becomes available world-wide through Data Access Protocol and you can view its content with a standard web browser. Data Access Protocol provides three ways of viewing data. First you can view the syntactic structure of data. Second you can view the semantic meaning of data. Finally, you can view the real content of data you want to retrieve. Semantic Meaning of Data Syntactic Structure of Data OPeNDAP Visualization Client

Why Important? HDF5: NASA EOS / NPOESS NASA’s MISSION: ACCESS! Our MISSION: Build a DAP-HDF5 Bridge Why is HDF5 part so important? The main reason is that new satellite data are being stored and will be stored in HDF5 in the future. And one of the most important mission of NASA is to provide a full access to its huge data center. So our mission is to bridge the gap between data provider and data user through OPeNDAP.

Goals 1st: Access HDF5 data via OPeNDAP 2nd: Construct HDF5 data from OPeNDAP So, here are our project goals. First, we want to provide way to access HDF5 in remote location like NASA in DAP-way. Yet, we want to lose as little as possible during this transformation process.

OPeNDAP HDF5 Server Map HDF5 Datatypes into DAP Datatypes Goal: Access HDF5 data via OPeNDAP Map HDF5 Datatypes into DAP Datatypes Plus, tweaks for OPeNDAP Clients Plus, tweaks for HDF-EOS Files And, we built the HDF5 server to provide a standard DAP-way of accessing HDF5. Plus, we did some additional work for some OPeNDAP clients and HDF-EOS files. In the next few slides, we’ll see why such tweaks are necessary.

Mapping HDF5 Datatypes Prototype server in 2001 NASA Grant from Nov. 2006 Support for Compound Datatype Support for Group Support for References / Links First Product Release in Mar. 2008 The first effort was made in 2001 and we received a NASA grant in 2006 to make it a robust product. Mapping HDF5 in DAP-way required matching HDF5 objects with DAP objects. We enhanced the prototype server by adding support for HDF5 objects like Group, Compound and References. The first product was released in March 2008 through OPeNDAP site.

OPeNDAP HDF5 Server Map HDF5 Datatypes in DAP Datatypes Goal: Access HDF5 data via OPeNDAP Map HDF5 Datatypes in DAP Datatypes Plus, tweaks for OPeNDAP Clients Plus, tweaks for HDF-EOS Files After building the HDF5 server to provide a standard DAP-way of accessing HDF5, we did some additional work for some OPeNDAP clients and HDF-EOS files. In the next few slides, we’ll see why such tweaks are necessary.

OPeNDAP Clients I hate long variable names. Not all OPeNDAP clients are created equal! I hate long variable names. I need special attributes on dataset. I care only Grid datatypes. I care only well-formed attributes. Some OPeNDAP clients are very picky in terms of what DAP-server can provide. They may even ask something that the original HDF5 doesn’t have in the attribute. They simply reject what the standard DAP protocol allows.

Tweaks for OPeNDAP Clients Two configuration options --enable-short-name --enable-CF Thus, we provided some configuration options during installation. This can make the most picky OPeNDAP client happy. However, enabling these options are risky due to you may not serve some dataset in HDF5 through server.

OPeNDAP HDF5 Server Map HDF5 Datatypes in DAP-way Goal: Transform HDF5 access in DAP-way Map HDF5 Datatypes in DAP-way Plus, tweaks for OPeNDAP Clients Plus, tweaks for HDF-EOS5 Files Yet, it was impossible to view HDF-EOS5 directly on a map. We needed an Array to Grid tweaking.

HDF EOS Grid with No Dimension data Not all HDF5 files are created for DAP! Grid with No Dimension data Clients expect Grid with Dimension data Some Attributes stored in Extremely Long String(s) (e.g. StructMetada.0) Clients expect well-structured attributes This kind of situation happened with NASA HDF-EOS5 case. The dataset in HDF EOS files that NASA produce cannot be served directly in DAP-way that vis. clients like. Since HDF5 data producer like NASA doesn’t have to keep DAP-way in mind, they created HDF-EOS5 files in a way that they feel convenient.

Tweaks for HDF-EOS Two more configuration options --enable-eos-grid --enable-eos-meta Thus, we believe it’s our job to make the raw HDF5 data into a new form that clients will like. --enable-eos-grid processes the raw data into Grid that can client can consume easily. --enable-eos-meta chops the long string into a better format that can client can handle.

Day After Server Tweaks Finally, Happy Clients! IDL ncBrowse Ferret MATLAB NCL IDV So after applying the tweaks for clients and hdf-eos, our server can serve many clients. Here’s a live demo and movie. GrADS ODC

Goals 1st : Access HDF5 via OPeNDAP 2nd: Construct HDF5 data from OPeNDAP Bonus: Help clients to better access EOS Swath data So we could achieve the first goal. Achieving the first goal has some side effects and give another challenge of preservation.

HDF5 Served in OPeNDAP Maximize There are some things money can’t buy. For everything else, there is MasterCardTM. Minimize There are some things DAP can’t serve. For everything else, there is _________? If you ever watched the “priceless” MasterCard commercial, you’ve heard this famous phrase: “There are …. “. We have found a similar situation when we tried to serve HDF5 in DAP-way. If HDF5 is served in DAP-way, it was inevitable to lose some details that the original HDF5 has simply because DAP doesn’t know how to represent. However, we think it would be great if we can minimize such things and find a clever solution like MasterCard that can cover and deliver maximum features in HDF5 through DAP-way. Maximize

Some Things (that OPeNDAP HDF5 server can’t serve) Hard: Opaque, Bitmap, Enum, 64 bit Integer, Variable Length types Illegal: Reserved characters in DAP are used in Dataset/Group name in HDF5 Here are some limitations that the current server can’t handle. Certain types like Opaque, Bitmap, Enum are not available in the current DAP protocol so it’s impossible to map them. Another interesting case is that there are some reserved characters in DAP yet they are perfectly legal in HDF5. In these cases, DAP protocol itself can be changed to have new additional types or use special hex encoding of characters.

For Everything Else Let there be HDF5-Friendly OPeNDAP client library! Package and Deliver in DAP first. Then, let the client library handle it. Concept of Trojan Horse However, it’s hard to modify the protocol itself since it’s standard so this is our current solution for everything else in HDF5. We’d like to make a special HDF5-friendly OPeNDAP client dedicated for our OPeNDAP HDF5 server. The concept is quite simple. We simply package and deliver them in DAP-way and let the client handle it.

Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it. Example: Group in HDF5 HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5 Here’s really one good example. In DAP, an attribute plays a role like comment in computer programming. Although DAP doesn’t have any concept of group, the group may play an important role in HDF5 like disambiguation of same variable name. This entire group structure is being sent as a single attribute in DAP. Thus, if a user wants to re-construct HDF5 using DAP client, this group attribute is essential and HDF5-friendly OPeNDAP client library should handle it properly.

Example: Reference in HDF5 Object / Regional Reference Map to DAP URL at server No de-referencing of URL at client library Important for NPOESS Another good example is the reference in HDF5. There are two types of reference in HDF5 – object and regional. Both object and regional reference can be mapped to a special data type called URL in DAP. However, the current OPeNDAP client library doesn’t support the de-referencing of the URL. That is, there is no way for a OPeNDAP client can access the dataset that the URL points to. This is particularly important for NPOESS data since it uses tons of HDF5 reference inside.

HDF5-Friendly OPeNDAP Client Library One more reason: Help Clients to view Swath Only ODC can display Swath properly. No Latitude and Longitude Easy but Rigid OPeNDAP NC Client Library! easy: same as netCDF C interface rigid: follows netCDF model Our client library must be Easy but Flexible There’s another reason that we want to pursue HDF5-friendly OPeNDAP client library. A swath is a 3-D scan of very small region on the earth. Earth Scientists are not much interested in seeing 2-D Grid data but a vertical profile of a swath like this. However, among the six OPeNDAP clients that we tried, only one client could display a vertical profile of swath data properly. The main reason is that many DAP clients impose un-necessary restrictions in the format that the DAP Protocol produces. One such restriction is a CF convention that we’ve discussed before. Thus, our goal is to build a client library that shifts the burden of meeting client’s requirement from the OPeNDAP HDF5 server to a client. Courtesy of NASA

NC-Friendly OPeNDAP Client Library HDF5-Friendly OPeNDAP Client Library View G/R/Swath ??? View NetCDF Group/Ref/Swath NetCDF View HDF5 HDF5 GrADS GrADS OPeNDAP HDF5 Server OPeNDAP NC Server liboc-dap libnc-dap dapserver However, it’s hard to modify the protocol itself since it’s standard so this is our current solution for everything else. We’d like to make a special DAP client dedicated for OPeNDAP HDF5 server. The concept is quite simple. We simply package and deliver them in DAP-way and let the client handle it. We want to provide a more generic OPeNDAP client library. libdap

Example: dap2h5 ORIGINAL CONSTRUCTED A test application for the client library It can construct HDF5 from DAP output ORIGINAL CONSTRUCTED To prove our client library concept, we created a demo application called “dap2h5”. As a prototype, it has a limited capability but this demo shows how to re-construct group information and dataset from DAP output successfully. Since it is a prototype, attributes on groups are ignored by “dap2h5” at this point.

Demo: GrADS GrADS is modified to use our library The client library provides grid mapping from swath data GrADS displays swath through HDF-EOS5 specific client library API calls Again, to prove the usefulness of our client library, we compiled GrADS with it and the new grads could display MLS level2 swath data directly. This demo shows the carbon monoxide level near Beijing before and during Olympics. CO level is high but later goes down significantly due to air quality control enforcement by Chinese government.

Caution Our Client Library is ONLY PROTOTYPE! It does NOT support all DAP data types It does NOT support all AURA files It does NOT support all Visualization clients Here’s a summary of our project. Providing a DAP-way access of HDF5 is easy, efficient and cool. However, it can lose some information that HDF5 originally has due to the nature of DAP-way transformation. This requires either the modification of DAP or the creation of HDF5-friendly client. Since it’s hard to modify the well-established DAP, we think it’s easier to implement HDF5-friendly OPeNDAP client library. When it’s done properly, we showed that the client library could serve the HDF5 files better and help the visualization clients to visualize HDF5-EOS data directly via DAP.

Summary HDF5 Access via OPeNDAP Is Easy / Efficient / Cool Loses some things Requires HDF5-friendly OPeNDAP Client Library HDF5-Friendly OPeNDAP Client Library Prototype Serves HDF5 better Benefits visualization clients Here’s a summary of our project. Providing a DAP-way access of HDF5 is easy, efficient and cool. However, it can lose some information that HDF5 originally has due to the nature of DAP-way transformation. This requires either the modification of DAP or the creation of HDF5-friendly client. Since it’s hard to modify the well-established DAP, we think it’s easier to implement HDF5-friendly OPeNDAP client library. When it’s done properly, we showed that the client library could serve the HDF5 files better and help the visualization clients to visualize HDF5-EOS data directly via DAP.

Future Work HDF5 to DAP2 Mapping Document Release HDF5-friendly OPeNDAP Client Library Prototype URL: http://hdfgroup.org/projects/opendap/index.html Here’s our future work. First, we’ll give a detailed document on mapping between HDF5 and DAP. Second, we’ll finish and release the HDF5-Friendly OPeNDAP Client Library prototype and test it with one more client like NCL. By the way, the library will remain as only prototype due to the limited funding availability.

Credits Mike Folk (THG) Robert McGrath (NCSA) Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET) Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang (NASA) Jennifer Adams (GrADS) Dave Brown (UCAR) We’d like to thank these people. They have provided us the right direction in development, an early access to data files and numerous feedback.

Acknowledgement This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX06AG75A. .  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.