1. www.hdfgroup.org The HDF Group HDF/HDF-EOS Workshop XIV1 Easy Remote Access via OPeNDAP Kent Yang and Joe Lee The HDF Group The 14 th HDF/HDF-EOS Workshop September 28, 2010

2. www.hdfgroup.org Easy HDF/HDF-EOS Workshop XIV2September 28,2010

3. www.hdfgroup.org Outline Introduction to OPeNDAP What’s New? o HDF4-OPeNDAP Handler (major) o HDF5-OPeNDAP Handler (minor) HDF/HDF-EOS Workshop XIV3September 28,2010

4. www.hdfgroup.org Client Library (libnc_dap) Data Access Protocol (DAP) via Internet Server (Hyrax) Handler (hdf4/5_handler) Remote Data (HDF4/5) View Data (User) OPeNDAP Concept 4 Visualization Tools (gradsdap) Typical Remote Access FTP/HTTP Visualization tools (e.g., grads) HDF/HDF-EOS Workshop XIVSeptember 28,2010

5. www.hdfgroup.org Access HDF via OPeNDAP HDF/HDF-EOS Workshop XIV5 Users OPeNDAP Clients IDV OPeNDAP Servers HDF Files Hyrax AIRS/OMI DAP libnc-dap Visualization Tools HDF HANDLER Aqua/Aura September 28,2010

6. www.hdfgroup.org Outline Introduction to OPeNDAP What’s New? o HDF4-OPeNDAP Handler (major) o HDF5-OPeNDAP Handler (minor) HDF/HDF-EOS Workshop XIV6September 28,2010

7. www.hdfgroup.org What’s New? HDF 4 -OPeNDAP Handler Work HDF/HDF-EOS Workshop XIV7September 28,2010

8. www.hdfgroup.org HDF4-OPeNDAP Handler Work Problems Our Work Limitations HDF/HDF-EOS Workshop XIV8September 28,2010

9. www.hdfgroup.org Existing HDF4 Handler Problem Some OPeNDAP Visualization Tools CAN NOT display most NASA HDF- EOS2 and HDF4 data HDF/HDF-EOS Workshop XIV9September 28,2010

10. www.hdfgroup.org Existing HDF4 Handler Problem HDF/HDF-EOS Workshop XIV10 Users OPeNDAP Clients IDV OPeNDAP Servers HDF4 Files Hyrax AIRS Grid DAP libnc-dap Aqua Existing HDF4 HANDLER Visualization Tools September 28,2010

11. www.hdfgroup.org HDF4-OPeNDAP Handler Work Problems Our Work Limitations HDF/HDF-EOS Workshop XIV11September 28,2010

12. www.hdfgroup.org Our Work Let Visualization Clients Access HDF- EOS2 and HDF4 files HDF/HDF-EOS Workshop XIV12September 28,2010

13. www.hdfgroup.org Why Visualization Fail? My house costs 1. HDF/HDF-EOS Workshop XIV13 My house in Hawaii costs $1M. Information – Earth Science - Content Geo-Location, Geo-Location, Geo-Location Unit, Scale factor September 28,2010

14. www.hdfgroup.org Why Visualization Fails? Clients expect certain data structures o Shared & named dimensions Clients expect some key attributes o CF conventions o coordinates attribute o unit attribute HDF/HDF-EOS Workshop XIV14September 28,2010

15. www.hdfgroup.org Extreme Makeover My house in HI costs 1M. HDF/HDF-EOS Workshop XIV15 My house in Hawaii costs $1M. Enhanced HDF4 Hander Interoperability! September 28,2010

16. www.hdfgroup.org Challenges Hundreds of HDF-EOS2 and HDF4 products o Grid Projections o Swath Dimension Map o Product Specific: TRMM / TOMS / MERRA / CERES / Etc. September 28,2010HDF/HDF-EOS Workshop XIV16

17. www.hdfgroup.org Solution Step 1: Retrieve Key Information Step 2: Transform Output File Structure following CF conventions and DAP 2.0 September 28,2010HDF/HDF-EOS Workshop XIV17

18. www.hdfgroup.org Step1: Retrieve Key Information Coordinate variable values in an HDF-EOS2 file September 28,2010HDF/HDF-EOS Workshop XIV18 Calculate Lat/Lon

19. www.hdfgroup.org Step1: Retrieve Key Information(continued) Coordinate variable values in an HDF4 file September 28,2010HDF/HDF-EOS Workshop XIV19 Latitude and Longitude are Provided outside the file.

20. www.hdfgroup.org Step2: Transform Output Flatten Structure and handle special characters September 28,2010HDF/HDF-EOS Workshop XIV20 DAP DAS Output following CF An HDF file Flatten CF attributes

21. www.hdfgroup.org Shift of Burden From Users to Handler Developers Client processing to Server processing September 28,2010HDF/HDF-EOS Workshop XIV21

22. www.hdfgroup.org Enhanced HDF4 Handler HDF/HDF-EOS Workshop XIV22 Users OPeNDAP Clients IDV OPeNDAP Servers HDF4 Files Hyrax DAP libnc-dap Aura MODIS Swath w/ Dim. Map Enhanced HDF4 HANDLER Visualization Tools September 28,2010

23. www.hdfgroup.org Demo: HDF-EOS2 Library Solution HDF/HDF-EOS Workshop XIV23 Swath with Dimension Map September 28,2010

24. www.hdfgroup.org HDF-EOS2 Products Supported GES DISC AIRS / MERRA / TOMS LAADS / LP DAAC / NSIDC Many MODIS products NSIDC AMSR_E / NISE LaRC MISR / MOPITT / some CERES(TRMM) HDF/HDF-EOS Workshop XIV24September 28,2010

25. www.hdfgroup.org Pure HDF4 Products Supported GES DISC TRMM L1B, L2B Swath TRMM L3 Grid 42B & 43B products OBPG (Ocean Color) SeaWiFS / MODIST / MODISA CZCS / OCTS HDF/HDF-EOS Workshop XIV25September 28,2010

26. www.hdfgroup.org Pure HDF4 Products Supported CERES CER_AVG CER_ES4 CER_ISCCP-D2like-Day CER_ISCCP-D2like-GEO CER_SRBAVG CER_SYN CER_ZAVG HDF/HDF-EOS Workshop XIV26September 28,2010

27. www.hdfgroup.org HDF4-OPeNDAP Handler Work Problems Our Work Limitations HDF/HDF-EOS Workshop XIV27September 28,2010

28. www.hdfgroup.org Their Limitations Reasons for Visualization Failures IDV / Panoply / Java Heap Memory Network / DAP (e.g., max HTTP header size) Can’t retrieve key information correctly (e.g., LAMAZ projection) from data Unable to harmonize with CF-conventions (e.g., FillValue inside latitude and longitude) HDF/HDF-EOS Workshop XIV28September 28,2010

29. www.hdfgroup.org Our Limitations HDF-EOS2 + additional HDF4 objects - Additional HDF4 objects will be ignored. The rest of pure HDF4 products not listed in previous slides are not tested and may not be visualized by Java OPeNDAP Clients. September 28,2010HDF/HDF-EOS Workshop XIV29

30. www.hdfgroup.org What’s New? HDF 5 -OPeNDAP Handler Work HDF/HDF-EOS Workshop XIV30September 28,2010

31. www.hdfgroup.org HDF5-OPeNDAP Handler Update HDF/HDF-EOS Workshop XIV31 Users OPeNDAP Clients IDV OPeNDAP Servers HDF5 Files Hyrax OMI Grid DAP libnc-dap Aura OMI Swath HDF5 HANDLER Update Visualization Tools September 28,2010

32. www.hdfgroup.org HDF5-OPeNDAP Handler Update Support Aura HDF-EOS5 OMI Swath Support Aura HDF-EOS5 MLS HDF/HDF-EOS Workshop XIV32September 28,2010

33. www.hdfgroup.org HDF/HDF-EOS Workshop XIV33September 28,2010 OPeNDAP HDF4/HDF5 handlers Source codes and RPMs can be found under http://opendap.org Demo HDF5/HDF4 servers http://hdfeos.org/zoo/hdf5_handler http://hdfeos.org/zoo/hdf4_handler

34. www.hdfgroup.org The HDF Group HDF/HDF-EOS Workshop XIV34 Thank you ! September 28,2010

35. www.hdfgroup.org Acknowledgements This work was supported by cooperative agreement number NNX08AO77A from the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space Administration. HDF/HDF-EOS Workshop XIV35September 28,2010

36. www.hdfgroup.org The HDF Group Questions/comments? HDF/HDF-EOS Workshop XIV36September 28,2010