The HDF Group HDF/HDF-EOS Workshop XIV1 Easy Remote Access via OPeNDAP Kent Yang and Joe Lee The HDF Group The 14 th HDF/HDF-EOS Workshop September 28, 2010

Easy HDF/HDF-EOS Workshop XIV2September 28,2010

Outline Introduction to OPeNDAP What’s New? o HDF4-OPeNDAP Handler (major) o HDF5-OPeNDAP Handler (minor) HDF/HDF-EOS Workshop XIV3September 28,2010

Client Library (libnc_dap) Data Access Protocol (DAP) via Internet Server (Hyrax) Handler (hdf4/5_handler) Remote Data (HDF4/5) View Data (User) OPeNDAP Concept 4 Visualization Tools (gradsdap) Typical Remote Access FTP/HTTP Visualization tools (e.g., grads) HDF/HDF-EOS Workshop XIVSeptember 28,2010

Access HDF via OPeNDAP HDF/HDF-EOS Workshop XIV5 Users OPeNDAP Clients IDV OPeNDAP Servers HDF Files Hyrax AIRS/OMI DAP libnc-dap Visualization Tools HDF HANDLER Aqua/Aura September 28,2010

Outline Introduction to OPeNDAP What’s New? o HDF4-OPeNDAP Handler (major) o HDF5-OPeNDAP Handler (minor) HDF/HDF-EOS Workshop XIV6September 28,2010

What’s New? HDF 4 -OPeNDAP Handler Work HDF/HDF-EOS Workshop XIV7September 28,2010

HDF4-OPeNDAP Handler Work Problems Our Work Limitations HDF/HDF-EOS Workshop XIV8September 28,2010

Existing HDF4 Handler Problem Some OPeNDAP Visualization Tools CAN NOT display most NASA HDF- EOS2 and HDF4 data HDF/HDF-EOS Workshop XIV9September 28,2010

Existing HDF4 Handler Problem HDF/HDF-EOS Workshop XIV10 Users OPeNDAP Clients IDV OPeNDAP Servers HDF4 Files Hyrax AIRS Grid DAP libnc-dap Aqua Existing HDF4 HANDLER Visualization Tools September 28,2010

HDF4-OPeNDAP Handler Work Problems Our Work Limitations HDF/HDF-EOS Workshop XIV11September 28,2010

Our Work Let Visualization Clients Access HDF- EOS2 and HDF4 files HDF/HDF-EOS Workshop XIV12September 28,2010

Why Visualization Fail? My house costs 1. HDF/HDF-EOS Workshop XIV13 My house in Hawaii costs $1M. Information – Earth Science - Content Geo-Location, Geo-Location, Geo-Location Unit, Scale factor September 28,2010

Why Visualization Fails? Clients expect certain data structures o Shared & named dimensions Clients expect some key attributes o CF conventions o coordinates attribute o unit attribute HDF/HDF-EOS Workshop XIV14September 28,2010

Extreme Makeover My house in HI costs 1M. HDF/HDF-EOS Workshop XIV15 My house in Hawaii costs $1M. Enhanced HDF4 Hander Interoperability! September 28,2010

Challenges Hundreds of HDF-EOS2 and HDF4 products o Grid Projections o Swath Dimension Map o Product Specific: TRMM / TOMS / MERRA / CERES / Etc. September 28,2010HDF/HDF-EOS Workshop XIV16

Solution Step 1: Retrieve Key Information Step 2: Transform Output File Structure following CF conventions and DAP 2.0 September 28,2010HDF/HDF-EOS Workshop XIV17

Step1: Retrieve Key Information Coordinate variable values in an HDF-EOS2 file September 28,2010HDF/HDF-EOS Workshop XIV18 Calculate Lat/Lon

Step1: Retrieve Key Information(continued) Coordinate variable values in an HDF4 file September 28,2010HDF/HDF-EOS Workshop XIV19 Latitude and Longitude are Provided outside the file.

Step2: Transform Output Flatten Structure and handle special characters September 28,2010HDF/HDF-EOS Workshop XIV20 DAP DAS Output following CF An HDF file Flatten CF attributes

Shift of Burden From Users to Handler Developers Client processing to Server processing September 28,2010HDF/HDF-EOS Workshop XIV21

Enhanced HDF4 Handler HDF/HDF-EOS Workshop XIV22 Users OPeNDAP Clients IDV OPeNDAP Servers HDF4 Files Hyrax DAP libnc-dap Aura MODIS Swath w/ Dim. Map Enhanced HDF4 HANDLER Visualization Tools September 28,2010

Demo: HDF-EOS2 Library Solution HDF/HDF-EOS Workshop XIV23 Swath with Dimension Map September 28,2010

HDF-EOS2 Products Supported GES DISC AIRS / MERRA / TOMS LAADS / LP DAAC / NSIDC Many MODIS products NSIDC AMSR_E / NISE LaRC MISR / MOPITT / some CERES(TRMM) HDF/HDF-EOS Workshop XIV24September 28,2010

Pure HDF4 Products Supported GES DISC TRMM L1B, L2B Swath TRMM L3 Grid 42B & 43B products OBPG (Ocean Color) SeaWiFS / MODIST / MODISA CZCS / OCTS HDF/HDF-EOS Workshop XIV25September 28,2010

Pure HDF4 Products Supported CERES CER_AVG CER_ES4 CER_ISCCP-D2like-Day CER_ISCCP-D2like-GEO CER_SRBAVG CER_SYN CER_ZAVG HDF/HDF-EOS Workshop XIV26September 28,2010

HDF4-OPeNDAP Handler Work Problems Our Work Limitations HDF/HDF-EOS Workshop XIV27September 28,2010

Their Limitations Reasons for Visualization Failures IDV / Panoply / Java Heap Memory Network / DAP (e.g., max HTTP header size) Can’t retrieve key information correctly (e.g., LAMAZ projection) from data Unable to harmonize with CF-conventions (e.g., FillValue inside latitude and longitude) HDF/HDF-EOS Workshop XIV28September 28,2010

Our Limitations HDF-EOS2 + additional HDF4 objects - Additional HDF4 objects will be ignored. The rest of pure HDF4 products not listed in previous slides are not tested and may not be visualized by Java OPeNDAP Clients. September 28,2010HDF/HDF-EOS Workshop XIV29

What’s New? HDF 5 -OPeNDAP Handler Work HDF/HDF-EOS Workshop XIV30September 28,2010

HDF5-OPeNDAP Handler Update HDF/HDF-EOS Workshop XIV31 Users OPeNDAP Clients IDV OPeNDAP Servers HDF5 Files Hyrax OMI Grid DAP libnc-dap Aura OMI Swath HDF5 HANDLER Update Visualization Tools September 28,2010

HDF5-OPeNDAP Handler Update Support Aura HDF-EOS5 OMI Swath Support Aura HDF-EOS5 MLS HDF/HDF-EOS Workshop XIV32September 28,2010

HDF/HDF-EOS Workshop XIV33September 28,2010 OPeNDAP HDF4/HDF5 handlers Source codes and RPMs can be found under Demo HDF5/HDF4 servers

The HDF Group HDF/HDF-EOS Workshop XIV34 Thank you ! September 28,2010

Acknowledgements This work was supported by cooperative agreement number NNX08AO77A from the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space Administration. HDF/HDF-EOS Workshop XIV35September 28,2010

The HDF Group Questions/comments? HDF/HDF-EOS Workshop XIV36September 28,2010