1. HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.)

2. Question of the Day How can I get One Sub from SUBWAY? C hicken teriyaki with no onion How can I get One Subset of EOS data image from NASA? Today’s satellite image right above the roof

3. Subway Easy: Easy: anywhere / accepts Visa|MC Efficient: Efficient: your own recipe Cool: Cool: eat fresh and look good Easy: Easy: anywhere / accepts Visa|MC Efficient: Efficient: your own recipe Cool: Cool: eat fresh and look good How can I get One Sub from SUBWAY?

4. Dumb-way Difficult: Difficult: bring a USB drive and visit NASA Inefficient: Inefficient: download 1 G-byte and search for 1 K-byte dataset Not cool: Not cool: wrong or useless dataset Difficult: Difficult: bring a USB drive and visit NASA Inefficient: Inefficient: download 1 G-byte and search for 1 K-byte dataset Not cool: Not cool: wrong or useless dataset How can I get One Subset of EOS data image from NASA?

5. DAP-way! Easy: Easy: anywhere / accepts IE|FF Efficient: Efficient: WYSIWYG – less fat! Cool: Cool: Visualization clients – see fresh! Easy: Easy: anywhere / accepts IE|FF Efficient: Efficient: WYSIWYG – less fat! Cool: Cool: Visualization clients – see fresh! How can I get One Subset of EOS data image from NASA?

6. DAP-way Implementation OPeNDAP: Free DAP-way Software Server-Client Model APIs Widely used by Earth Science Community

7. OPeNDAP Market Dominance: http://www.opendap.org OPeNDAP server sites Courtesy of OPeNDAP, Inc.

8. OPeNDAP Servers Data Matlab HDF4JDBC FreeFromFITS CDFCEDAR Data netCDF HDF4Matlab Data DSP Data JGOFS TablesSQLFITSCDF Flat Binary CEDAR Data CODAR Data ESML GeneralCODAR Courtesy of OPeNDAP, Inc.

9. OPeNDAP Clients netCDF C netCDF Java IDVFerretGrADSVisADncBrowse MatlabExcelIDLAccess Matlab Client IDL Client Courtesy of OPeNDAP, Inc.

10. How OPeNDAP works OPeNDAP Client OPeNDAP HDF4 Server HDF4 OPeNDAP NetCDF Server NetCDF

11. Example Usage OPeNDAP Client OPeNDAP HDF4 Server HDF4 Syntactic Structure of Data Semantic Meaning of Data Actual Content of Data OPeNDAP Visualization Client

12. What about HDF5? OPeNDAP Client OPeNDAP HDF4 Server HDF4 OPeNDAP NetCDF Server NetCDF OPeNDAP HDF5 Server HDF5

13. Why Important? HDF5: NASA EOS / NPOESS ACCESS! NASA’s MISSION : ACCESS! Our MISSION: Build DAP-way Bridge

14. Goals 1st: Transform HDF5 access in DAP-way 2nd: Yet preserve the beauty of HDF5

15. OPeNDAP HDF5 Server 1.Map HDF5 Datatypes in DAP-way 2.Plus, tweaks for OPeNDAP Clients 3.Plus, tweaks for HDF-EOS Files Goal: Transform HDF5 access in DAP-way

16. Mapping HDF5 in DAP-way Prototype server in 2001 NASA Grant from Nov. 2006 Support for Compound Datatype Support for Group Support for References / Links First Product Release in Mar. 2008

17. OPeNDAP HDF5 Server 1.Map HDF5 Datatypes in DAP-way 2.Plus, tweaks for OPeNDAP Clients 3.Plus, tweaks for HDF-EOS Files Goal: Transform HDF5 access in DAP-way

18. OPeNDAP Clients I hate long variable names. I need special attributes on dataset. I care only Grid datatypes. I care only well-formed attributes. Not all OPeNDAP clients are created equal!

19. Tweaks for OPeNDAP Clients Two configuration options --enable-short-name --enable-CF

20. OPeNDAP HDF5 Server 1.Map HDF5 Datatypes in DAP-way 2.Plus, tweaks for OPeNDAP Clients 3.Plus, tweaks for HDF-EOS5 Files Goal: Transform HDF5 access in DAP-way

21. HDF EOS Grid with No Dimension data Clients expect Grid with Dimension data Metadata Attribute in Two Strings Clients expect One Structured Format Not all HDF5 files are created for DAP-way!

22. Tweaks for HDF-EOS Two more configuration options --enable-eos-grid --enable-eos-meta

23. Day After Server Tweaks Finally, Happy Clients!

24. Live Demo Ozone concentration level over the south pole

25. Goals 1st: Transform HDF5 access in DAP-way 2nd: Yet preserve the beauty of HDF5

26. HDF5 Served in DAP-way (Yet Preserve the beauty of HDF5) There are some things money can’t buy. For everything else, there is MasterCard TM. There are some things DAP can’t serve. For everything else, there is _________? Minimize Maximize

27. Some Things Hard: Opaque, Bitmap, Enum, 64 bit Integer, Variable Length types Illegal: Reserved characters in DAP are used in Dataset/Group name in HDF5 (that OPeNDAP HDF5 server can’t serve)

28. For Everything Else Let there be HDF5-friendly OPeNDAP client library! Package and Deliver in DAP-way first. Then, let the client library handle it. Concept of Trojan Horse

29. Example: Group in HDF5 Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it. HDF5-friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5

30. Example: Reference in HDF5 Object / Regional Reference Map to DAP URL at server No de-referencing of URL at client library Important for NPOESS

31. HDF5-Friendly OPeNDAP Client Library One more reason: Tame Clients Only ODC can display Swath properly. Easy but Evil OPeNDAP NC Client Library!  easy: nc style API  evil: fixed dimensional attributes Our client library must be Easy but Good No Latitude and Longitude Courtesy of NASA

32. libnc-dap ??? liboc-dap NC-Friendly OPeNDAP Client Library libdap dapserver OPeNDAP NC Server GrADS NetCDFView NetCDF HDF5 Group/Ref/Swath OPeNDAP HDF5 Server View HDF5 GrADS View G/R/Swath HDF5-Friendly OPeNDAP Client Library

33. Sample Prototype APIs Generic OPeNDAP o oc_open() / oc_inq() HDF5 specific o oc_hdf5_is_hdf5() / oc_hdf5_find_group () HDF-EOS5 specific o oc_he5_is_eos5() / oc_he5_grid_to_swath() HDF5-Friendly OPeNDAP Client Library

34. Demo: dap2h5 A test application for the client library It can re-build HDF5 from DAP output

35. Demo: GrADS GrADS is modified to use our library The client library provides grid mapping from swath data GrADS displays swath through HDF-EOS5 specific API calls

36. Summary DAP-way access of HDF5 Is Easy / Efficient / Cool Loses some things Requires HDF5-friendly OPeNDAP Client Library HDF5-friendly OPeNDAP Client Library Serves HDF5 better Benefits visualization clients

37. Future Work HDF5 DAP Mapping Document Release HDF5-friendly OPeNDAP Client Library prototype

38. Credits Mike Folk (THG) Robert McGrath (NCSA) Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET) Christopher Lynnes, James Johnson, Denis Nadeau (NASA) Jennifer Adams (GrADS) Dave Brown (UCAR)

39. Acknowledgement This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNG05GC60A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.