1. HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015

2. Outline HDF5 – OPeNDAP work review HDF4 – OPeNDAP handler progress report Future work plan 10/31/2015Annual briefing to ESDIS 2

3. Users OPeNDAP Clients IDV OPeNDAP Servers HDF4 Files Hyrax AIRS DAP libnc-dap Visualization Tools Access data via OPeNDAP HDF HANDLER 10/31/2015 3 Annual briefing to ESDIS

4. HDF5 – OPeNDAP Work 4 Annual briefing to ESDIS10/31/2015

5. HDF5-OPeNDAP handler Update with new Hyrax Help GES DISC serve TES and OMI data Finish HDF5 – DAP2 mapping document and submit it as a technical note to NASA SPG 10/31/2015Annual briefing to ESDIS 5

6. 10/31/2015Annual briefing to ESDIS 6 HDF5-Friendly OPeNDAP Client Library Retrieve the key HDF5 information Easy to access other HDF5 data (e.g. EOS5 swath)

7. Example: Groups in HDF5 Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it. HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5 7 Annual briefing to ESDIS10/31/2015

8. Annual briefing to ESDIS 8 MLS swath via GrADS CO at these elevations is a measure of pollution lower stratosphere upper stratosphere Before After During CO level near Beijing, China before, during and after Olympics

9. HDF4 – OPeNDAP Work 9 Annual briefing to ESDIS10/31/2015

10. Current HDF4-OPeNDAP handler Contributed by JPL in 1998 Part of the OPeNDAP distribution Visualization client tools cannot display most HDF-EOS2 and HDF4 data 10/31/2015Annual briefing to ESDIS 10

11. Enhance the HDF4-OPeNDAP handler More NASA HDF-EOS2 and HDF4 data can be accessed via visualization clients 11 Annual briefing to ESDIS10/31/2015 Our work

12. Challenges – more products 12 Annual briefing to ESDIS10/31/2015 Hundreds of HDF-EOS2 and HDF4 Data products

13. Challenges – Requirement from Clients CF conventions - Clients expect a certain data structure - Clients expect some key attributes 13 Annual briefing to ESDIS10/31/2015

14. Various HDF-EOS2 Grid projections Geographic Sinusoidal Polar stereographic … 10/31/2015Annual briefing to ESDIS 14 Challenges – Grid Data [4][12] Lon[12] Data [4][8] Lon[4][8] Geographic Sinusoidal

15. HDF-EOS2 Swath - Dimension map 10/31/2015Annual briefing to ESDIS 15 Challenges - Swath Data field has three columnsLongitude field has two columns Longitude should have three columns

16. Challenges – various products “Special” HDF-EOS2 files Hybrid HDF-EOS2 files -Some HDF4 objects that cannot be read by EOS library HDF4 data products 10/31/2015Annual briefing to ESDIS 16

17. Enhanced HDF4 Handler 10/31/2015Annual briefing to ESDIS 17 Follow CF conventions  Generate the correct DAP data type.  Add shared geo-location information.  Insert additional key attributes. How: Parse Metadata or Use EOS2 library

18. Implementation choice Use Metadata parser or HDF-EOS2 library Metadata parser Pro: simple Con: limited products HDF-EOS2 library Pro: maximum products Con: install HDF-EOS2 library first 10/31/2015Annual briefing to ESDIS 18

19. Solution Comparison 19 Annual briefing to ESDIS10/31/2015

20. Quick Implementation Parse Metadata – hdf5_handler Use HDF-EOS2 library – h4toh5 20 Annual briefing to ESDIS10/31/2015

21. Current Status MODIS / AIRS Grid and Swath Other Projections (Polar, Sinusoidal) Testing 21 Annual briefing to ESDIS10/31/2015

22. HDF-EOS2 in CF via DAP 22 10/31/2015 AIRS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdf http://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10 MODIS Grid MOD08_D3.A2008284.005.2008289105354.hdf http://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html AIRS Swath ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G08 331121849.hdf MODIS Swath MOD06_L2.A2001222.0025.004.2003127111841.hdf http://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.html Annual briefing to ESDIS

23. More challenges: Missing Attributes MODIS Grid – Rain fall 23 Annual briefing to ESDIS10/31/2015 No Unit! AURA OMI Grid - Ozone

24. More challenges Match dimension names Horizontal geolocation information: - Non-standard names ( e.g. nlon, nlat) How to match? Vertical geolocation information: - Users provide the geolocation name and vaule but no conventions are enforced. How to know? 10/31/2015Annual briefing to ESDIS 24

25. More challenges – Missing Values In a swath file, a data field has three dimensions. But no dimension value for the third dimension. 25 Annual briefing to ESDIS10/31/2015 MISR_AM1_TC_CLASSIFIERS_P020_O000995_F06_0011.hdf

26. A Solution to challenges Provide an option for the handler to generate fully CF-compliant output Create a separate XML input file for the handler and feed it into the handler Insert attributes, renaming rules, and missing values by putting them in the XML input This requires a knowledge about the characteristics of data in advance 26 Annual briefing to ESDIS10/31/2015

27. One Solution An input file from data provider Example: Attributes, Renaming, Values 27 Annual briefing to ESDIS10/31/2015 inches 0,1,2,3,4,5,6,7,8,9

28. Future Work (Short Term) Expand test-suite with data pool Release through OPeNDAP website Support for Swath via Parsing Support for Point data type via HDF- EOS2 library 28 Annual briefing to ESDIS10/31/2015

29. Future Work (Long Term) General HDF4 Files Support Using an input XML file to provide full CF support HDF-EOS2 to DAP2 Mapping Document 29 Annual briefing to ESDIS10/31/2015

30. Thank You! October 15-18, 2008Workshop Name (edit footer to change) 30

31. Acknowledgements This work was supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A, NNX08A077A and NNX06AG75A. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. October 15-18, 2008 31 Workshop Name (edit footer to change)

32. Questions/comments? October 15-18, 2008Workshop Name (edit footer to change) 32