DAP+NETCDF Using the netCDF-4 Data Model Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009

Overview RECAP: netCDF-DAP Integration RECAP: Architecture DAP Elements Example DDS and DAS DAP to netCDF-4 Translation netCDF-3 versus netCDF-4 Translation Extending the URL Format Constraints: Projections Constraints: Selections Figuring Out Constraints Client Parameters Revisited Building and Installing netCDF-4 + DAP Debugging Notes

RECAP: NETCDF-DAP Integration Unidata has integrated the DAP protocol into the standard netCDF distribution. It replaces the OPeNDAP provided libnc-dap library It closely mimics the original libnc-dap translation Implemented completely in C, versus libnc-dap, which is C++ This talk is primarily about the support for the netCDF-4 data model

RECAP: Architecture / Flow GRIB netCDF-3 HDF5 . Internet DAP Converter Remote Data Server DAP to netCDF Converter NCO (NC operators) NCDUMP Other Programs Data Request Using DAP URL Data in DAP Format netCDF library Part of the netCDF Library Web Server

DAP Elements DDS - describes the format of the data DAS - describes associated attributes DATADDS - the actual data; aka DODS DATADDS contains two things: A subset of the DDS followed by The actual data

Web Debugging If you are knowledgeable about DAP, you can look at the underlying DDS, DAS, DATADDS, and more, using a web browser Procedure: Remove the client parameters Add to the base url one of the following extensions: .dds, .das, .asc (or .ascii), .html, .info

Example DDS and DAS Dataset { Sequence { String Drifter_ID; String Date_Sampled; Float64 Latitude; Float64 Longitude; Float64 SST; ... } Drifters; } EOS.DBO; Attributes { FacilityGlobal { String DataCenter "COAS"; String DrifterType "MetOcean"; } Drifter_ID { String Description “Instrument unique identifier Date_Sampled { String Description "Date/time"; String Timezone "GMT"; }...

DAP to NETCDF-4 Translation The client parameter [mode=netcdf4] will cause the DAP DDS and data to be translated to the netCDF-4 data model netCDF-4 model is more powerful than DAP, so the translation only uses part of netCDF-4 Versus the netCDF-3 (classic) translation Lossless All DAP primitive types handled (no upgrading) DAP sequences are translated to combinations of netCDF-4 compound types and vlen types Performance improvement No need for string or sequence dimensions

NETCDF-3 vs netCDF-4 translation netcdf Drifters { dimensions: Drifters = 5 ; // Sequence // dimension stringdim64 = 64 ; // String // dimension variables: char Drifters.Drifter_ID(Drifters, stringdim64) ; double Drifters.Latitude(Drifters); double Drifters.Longitude(Drifters); double Drifters.SST(Drifters); netcdf Drifters { types: compound Drifters_record_t { string Drifter_ID ; double Latitude ; double Longitude ; double SST; }; Drifters_record_t(*) Drifters_t ; variables: Drifters_t Drifters ; … …

Extending the DAP URL Format <clientparams><baseurl>?<constraint> Additional client parameters Constraint tells the OPeNDAP server what subset of the dataset to send Constraint is divided into two parts Projections Selections Currently disabled Data Specified By Full DDS Data Specified By Constraints

Client Parameters Revisited Explicitly specify [mode=netcdf4] in order to get the netcdf-4 translation Or specify NC_NETCDF4 mode to nc_open() Additional [show=…] cases [show=dds] (attribute “_dds”) [show=das] (attribute “_das”) [show=datadds] (attribute “_datadds”) String attribute defining the DDS, DAS, or DDS part of the DATADS, respectively, as obtained from the OPeNDAP server

Constraints: Projections Projections specify array slices of underlying OPeNDAP variables to send http://test.opendap.org/dods/dts?windW[0:10:2] Format: ?<projection>,<projection>,… Projection is DAP variable name followed by a slice specification [first:stride:last] Note: last, not count

Constraints: Selections Selection constraints tell the OPeNDAP server to subset parts of the dataset based on the values in the dataset Can only be applied to DAP Sequences http://test.opendap.org/dap/data/&CS02.light>0 Format: &<selection>&<selection>&… Multiple selections are treated as logical “and” Selections always follow the projections, if any in the URL

Figuring out Constraints How does one know what constraints to apply? The problem is that they are defined in terms of the underlying DAP data model => Breaks the netCDF abstraction For projections, the “show=projection” parameter can help by showing the underlying DAP variables For Selections, you need to know something about the values in the original dataset Look at the .info file?

Build and Install NETCDF4 + DAP Add --enable-dap and --enable-netcdf-4 to your list of ./configure flags Specify the curl library (libcurl) if ./configure cannot find it --with-curl=<path to curl library> As usual, of course, specify --with-hdf5 and --with-zlib

Example OPeNDAP Data Sources http://test.opendap.org:8080/dods/dts http://test.opendap.org http://test.opendap.org/dap/data/ff http://www.opendap.org/data/datasets.cgi ?xmlfilename=datasets.xml&exfunction=none

Questions?

Demo Sites http://test.opendap.org:8080/dods/dts/Drifters http://test.opendap.org/dap/data/ff http://www.usgodae.org/dods/GDS/coamps_cent_am http://www.opendap.org/data/datasets.cgi?xmlfilename=datasets.xml&exfunction=none