The HDF Group HDF/HDF-EOS Workshop XV1 HDF-OPeNDAP Project Update Joe Lee and Kent Yang The HDF Group April 18, 2012

Outline Introduction to OPeNDAP HDF5 Handler Update HDF4 Handler Update Live Demo HDF/HDF-EOS Workshop XV2April 18, 2012

Introduction to OPeNDAP Goal: Background on Update Why OPeNDAP? What is HDF Handler? How important CF conventions? April 18, 2012HDF/HDF-EOS Workshop XV3

Access HDF via FTP HDF/HDF-EOS Workshop XV4 Users FTP Clients IDL FTP Servers HDF Files AIRS/OMI FTP Local HDF Files Aqua/Aura April 18, 2012

Client Library (libnc_dap) Data Access Protocol (DAP) via Internet Server (Hyrax) Handler (hdf4/5_handler) Remote Data (HDF4/5) View Data (User) OPeNDAP concept 5 Visualization Tools (gradsdap) Typical remote access problem FTP/HTTP Visualization tools (e.g., GrADS) HDF/HDF-EOS Workshop XVApril 18, 2012

Access remote data easily April 18, 2012HDF/HDF-EOS Workshop XV6

without worrying about data plans. April 18, 2012HDF/HDF-EOS Workshop XV7

Visualize remote data easily, too. April 18, 2012HDF/HDF-EOS Workshop XV8

HDF handler is key HDF/HDF-EOS Workshop XV9 Users OPeNDAP Clients IDV OPeNDAP Servers HDF Files Hyrax AIRS/OMI DAP NetCDF Java Visualization Tools HDF HANDLER Aqua/Aura April 18, 2012

because if handler doesn’t follow CF April 18, 2012HDF/HDF-EOS Workshop XV10 No Visualization Bonus!

If handler gives up extra bonus April 18, 2012HDF/HDF-EOS Workshop XV11 HDF / HDF-EOS DAP Server DAP handler development is easier.

What’s involved in extra bonus April 18, 2012HDF/HDF-EOS Workshop XV12 DAP Server CF Conventions DAP DAP Client Visualization Earth Scientist HDF / HDF-EOS

How handler developers feel April 18, 2012HDF/HDF-EOS Workshop XV13

Because we care all NASA products. April 18, 2012HDF/HDF-EOS Workshop XV14 ? AURA NPP AQUA TERRA MISR

Developers’ Dream April 18, 2012HDF/HDF-EOS Workshop XV15 … all data are created equal, that they shall be visualized by all client tools easily…

What if all people agreed… April 18, 2012HDF/HDF-EOS Workshop XV16 UDF (Imaginary) DAP Server DAP DAP Client Visualization Earth Scientist development might have been much easier. Check once.

Lessons from handler development April 18, 2012HDF/HDF-EOS Workshop XV17 Communicate and work together among different groups. Standardize everything from data products to visualization tools. Follow conventions and standards.

Outline Introduction to OPeNDAP HDF5-OPeNDAP Handler Update HDF4-OPeNDAP Handler Update Live Demo HDF/HDF-EOS Workshop XV18April 18, 2012

Redesign of HDF5 handler Goal: Support all NASA HDF5 products. Modularize for future product support. Follow CF conventions robustly. April 18, 2012HDF/HDF-EOS Workshop XV19 Product

Why did we redesign? April 18, 2012HDF/HDF-EOS Workshop XV20 In 2007, was enough for AURA Grid HDF5 handler

Why did we redesign? April 18, 2012HDF/HDF-EOS Workshop XV21 In 2011, wasn’t enough for AURA Grid AURA Swath AURA ZA GSSTF

Why did we redesign? April 18, 2012HDF/HDF-EOS Workshop XV22 In 2011, wasn’t enough for AURA Grid AURA Swath AURA ZA GSSTF

and more… so we gave up! April 18, 2012HDF/HDF-EOS Workshop XV23 GOSAT ICESat-2 Aquarius SMAP

We rebuilt one that can scale up April 18, 2012HDF/HDF-EOS Workshop XV24

by modularizing the design April 18, 2012HDF/HDF-EOS Workshop XV25 HDF5 handlerCFHDF5GOSATAquariusSMAP HDF- EOS5 GRIDSWATHZAGeneric with a big room for future products.

Key updates on HDF5 handler April 18, 2012HDF/HDF-EOS Workshop XV26 version1.5.x1.6.x CF-convention Support PrimitiveRobust Range of NASA Data Products LimitedAll Mulitiple Grids/Swath/ZA NoYes Configuration Options Static BuildRun-time Support for Future Products RigidFlexible

HDF-EOS5 products supported OMI Swath, Grid MLS Swath HIRDLS Swath, ZA TES Swath, Grid GSSTF April 18, 2012HDF/HDF-EOS Workshop XV27

HDF5 products supported Aquarius Grid MEaSUREs SeaWiFS DeepBlue SBUV2 Ozone (O3) Profile SMAP (Simulation) April 18, 2012HDF/HDF-EOS Workshop XV28

HIRDLS Zonal Average April 18, 2012HDF/HDF-EOS Workshop XV29

New TES Swath April 18, 2012HDF/HDF-EOS Workshop XV30

MEaSUREs GSSTF April 18, 2012HDF/HDF-EOS Workshop XV31

Aquarius Grid (Simulation) April 18, 2012HDF/HDF-EOS Workshop XV32

MEaSUREs SeaWiFS DeepBlue April 18, 2012HDF/HDF-EOS Workshop XV33

Ozone Profile (BUV) April 18, 2012HDF/HDF-EOS Workshop XV34

SMAP Swath (Simulation) April 18, 2012HDF/HDF-EOS Workshop XV35

SMAP Swath (Simulation) April 18, 2012HDF/HDF-EOS Workshop XV36

No CF magic for some products Tool / CF Limitation GOSAT/acos: 1-D trajectory ICESat-2 OMI L2G: Lat/Lon with 3-D Java-NetCDF Client Limitation Dataset w/ long string April 18, 2012HDF/HDF-EOS Workshop XV37

It doesn’t mean handler failure April 18, 2012HDF/HDF-EOS Workshop XV38 because other tools like can still access and visualize data.

MATLAB and IDL will work too. April 18, 2012HDF/HDF-EOS Workshop XV39 HDF5 Examples under zoo will work. Visit Replace local file access with OPeNDAP URL access. MATLAB: Try new OPeNDAP functions. The rest will be same.

Read if you can’t see April 18, 2012HDF/HDF-EOS Workshop XV40 Aura_OMI_Level3/OMTO3e.003/2012/OMI-Aura_L3- OMTO3e_2012m0416_v m0418t he5.ascii?lon[1029:1:1029],lat[513:1:513],_HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3[513:1:513][1029:1:1029] Dataset: OMI-Aura_L3-OMTO3e_2012m0416_v m041 8t he5 lon, lat, _HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3.lon, _HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3._HDFEOS_GRIDS_O MI_Column_Amount_O3_Data_Fields_ColumnAmountO3[_HDFEOS_GRIDS_OMI_Column_Amount_ O3_Data_Fields_ColumnAmount O3.lat=38.375], 309.6

or hear from Siri? April 18, 2012HDF/HDF-EOS Workshop XV41

Outline Introduction HDF5-OPeNDAP Handler Update HDF4-OPeNDAP Handler Update Live Demo HDF/HDF-EOS Workshop XV42April 18, 2012

HDF4 OPeNDAP handler update Corrected the handling the scale/offset for MODIS products. Added the support for Lambert azimuthal equal-area (LAMAZ). Added the support for Space-oblique Mercator projection (SOM). Other minor bug fixes. HDF/HDF-EOS Workshop XV43April 18, 2012

Key updates on HDF4 handler April 18, 2012HDF/HDF-EOS Workshop XV44 version Some MODIS Scale/offset WrongCorrected LAMAZ projection support NoYes SOM projection support NoYes

MODIS scale / offset correction HDF/HDF-EOS Workshop XV45April 18, 2012

Support for LAMAZ projection HDF/HDF-EOS Workshop XV46April 18, 2012

Support for SOM projection HDF/HDF-EOS Workshop XV47April 18, 2012 Dataset { UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512]; Int32 SOMBlockDimU2[SOMBlockDimD1 = 180]; Float32 LatitudeU3[XDimD2 = 128][YDimD3 = 512]; Float32 LongitudeU4[XDimD2 = 128][YDimD3 = 512]; … } Dataset { UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512]; Int32 SOMBlockDimU2[SOMBlockDimD1 = 180]; Float32 LatitudeU3[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512]; Float32 LongitudeU4[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512]; … }

HDF-EOS2 products supported GES DISC AIRS / MERRA / TOMS LAADS / LP DAAC / NSIDC Many MODIS products NSIDC AMSR_E / NISE / MODIS LaRC MISR / MOPITT / some CERES (TRMM) HDF/HDF-EOS Workshop XV48April 18, 2012

HDF4 products supported GES DISC TRMM L1B, L2B Swath TRMM L3 Grid 42B & 43B products OBPG (Ocean Color) SeaWiFS / MODIST / MODISA CZCS / OCTS HDF/HDF-EOS Workshop XV49April 18, 2012

HDF4 products supported CERES CER_AVG CER_ES4 CER_ISCCP-D2like-Day CER_ISCCP-D2like-GEO CER_SRBAVG CER_SYN CER_ZAVG HDF/HDF-EOS Workshop XV50April 18, 2012

HDF4 handler limitations HDF-EOS2 + additional HDF4 objects In some products, some additional HDF4 objects will be ignored. Other HDF4 products not listed in previous slides are not tested and may not be visualized by Java OPeNDAP clients. HDF/HDF-EOS Workshop XV51April 18, 2012

Other limitations CF conventions Limitation _FillValue inside latitude and longitude Java-NetCDF Limitation Java Heap Memory HDF/HDF-EOS Workshop XV52April 18, 2012

Outline Introduction HDF5-OPeNDAP Handler HDF4-OPeNDAP Handler Live Demo HDF/HDF-EOS Workshop XV53April 18, 2012

We’ve updated handlers HDF/HDF-EOS Workshop XV54April 18, 2012 OPeNDAP HDF Handlers HDF- EOS2 HDF5 HDF- EOS5 HDF4 … and will continue.

GES DISC Uses Handlers HDF/HDF-EOS Workshop XV55April 18, 2012

Please try HDF4/HDF5 handlers HDF/HDF-EOS Workshop XV56April 18, 2012 New hdf5_handler will be released in July. Source codes and RPMs: Demo HDF5/HDF4 servers:

and give us feedback. HDF/HDF-EOS Workshop XV57April 18, 2012

The HDF Group HDF/HDF-EOS Workshop XV58 Thank you! April 18, 2012

Acknowledgements HDF/HDF-EOS Workshop XV59April 18, 2012 This work was supported by Subcontract number under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.

The HDF Group Questions/comments? HDF/HDF-EOS Workshop XV60April 18, 2012