HDF4 OPeNDAP Project Progress Report MuQun Yang and Hyo-Kyung Lee 1 HDF Developers' Meeting11/24/2015

OPeNDAP Review A software framework that allows simple access to remote data Data Access Protocol Client-server model Widely used by Earth Science Community 2 HDF Developers' Meeting11/24/2015

Users OPeNDAP Clients IDV OPeNDAP Servers HDF4 Files Hyrax AIRS DAP libnc-dap Visualization Tools OPeNDAP Reivew

Client Library (libnc-dap) DAP Protocol (via http) Server (apache) Handler (hdf4_handler) Remote Data (HDF4) View Data (User) OPeNDAP Concept 4 HDF Developers' Meeting11/24/2015 Visualization Tools (gradsdap) Typical Remote Access FTP/HTTP Visualization tools (e.g. grads)

Better Access for HDF-EOS2 via DAP Faster download through selection Quicker analysis through visualization 5 HDF Developers' Meeting11/24/2015 Goals

You bought a HDTV but... No HD signal Needs more Channels in HD Cable Service Provider Problem 6 HDF Developers' Meeting11/24/2015 HDTV Problems

You downloaded a viewer but... No CF convention Needs more HDF4 data in CF Data Service Provider Problem 7 HDF Developers' Meeting11/24/2015 HDF4 Problems

Challenges: HDF-EOS2 CF convention (e.g. NetCDF model) Syntax Clients expect a certain data structure Semantics Clients expect some key attributes 8 HDF Developers' Meeting11/24/2015 HD Signal Challenge

Challenges: HDF-EOS2 DVD/VHS/VCD/LD >>> Blue-Ray Can all titles be delivered in HD? HDF4/HDF-EOS2 >>> HDF5/HDF-EOS5 Can all HDF-EOS2 be delivered in CF? 9 HDF Developers' Meeting11/24/2015 More Channels in HD Challenge

Solution: New HDF4 Handler 10 HDF Developers' Meeting11/24/2015 HD Signal Solution Syntax  Generate the correct DAP data type.  Add shared geo-location information. Semantics  Insert additional key attributes.

Solution: New HDF4 Handler Use Metadata parser Pro: simple Con: limited channels Use HDF-EOS2 library Pro: maximum channels Con: install HE2 library first 11 HDF Developers' Meeting11/24/2015 More Channels in HD Solution

Solution Comparison 12 HDF Developers' Meeting11/24/2015

Quick Implementation Parse Metadata – hdf5_handler Use HDF-EOS2 library – h4toh5 13 HDF Developers' Meeting11/24/2015

Current Status MODIS / AIRS Grid and Swath Other Projections (Polar, Sinusoidal) Testing 14 HDF Developers' Meeting11/24/2015

HDF-EOS2 in CF via DAP Finally, Happy Data Consumer! 15 HDF Developers' Meeting11/24/2015 AIRS Grid AIRS L3.RetStd001.v G hdf MODIS Grid MOD08_D3.A hdf AIRS Swath ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/AIRS L2.RetStd.v G hdf MODIS Swath MOD06_L2.A hdf

Limitations – Missing Attributes MODIS Grid – Rain fall 16 HDF Developers' Meeting11/24/2015 No Unit! AURA OMI Grid - Ozone

Limitations – Missing Names How to Match Dimension Names? 17 HDF Developers' Meeting11/24/2015 HDF4 Handler NetCDF4 Handler pressureStd → Level

Limitations – Missing Values How to Supply Missing Dimension Values? 18 HDF Developers' Meeting11/24/2015 MISR_AM1_TC_CLASSIFIERS_P020_O000995_F06_0011.hdf HTTP/ OK XDODS-Server: XOPeNDAP-Server: XDAP: 3.1 Content-Type: text/plain Content-Description: dods_error Cache-Control: no-cache Error { code = 1002; message = "An internal error was encountered: This HDF-EOS2 file has multiple Grids with different dimension sizes. Please report this to "; };

Solution – Customization Provide an option for handler customization to generate fully CF- compliant output Create a separate XML input file for handler and feed it into the handler. Insert attributes, renaming rules, and missing values by putting them in the XML input. This requires a knowledge about the characteristics of data in advance. 19 HDF Developers' Meeting11/24/2015

Solution – Customization Example: Attributes, Renaming, Values 20 HDF Developers' Meeting11/24/2015 inches 0,1,2,3,4,5,6,7,8,9

Summary HDF4 Access via OPeNDAP Is Easy, Efficient and Fast Is Cool like HDTV Adding CF is not straightforward Use HDF-EOS2 library Is Essential Maximize the data coverage 21 HDF Developers' Meeting11/24/2015

Future Work (Short Term) Expand test-suite with data pool Release through OPeNDAP website Support for Swath via Parsing Support for Point data type via HDF- EOS2 library 22 HDF Developers' Meeting11/24/2015

Future Work (Long Term) General HDF4 Files Support Customization for CF option HE2 to DAP Mapping Documentation 23 HDF Developers' Meeting11/24/2015