1. DAP+NETCDF Using the netCDF-4 Data Model
Dr. Dennis Heimbigner
Unidata netCDF Workshop
August 3-4, 2009

2. Overview RECAP: netCDF-DAP Integration RECAP: Architecture
DAP Elements
Example DDS and DAS
DAP to netCDF-4 Translation
netCDF-3 versus netCDF-4 Translation
Extending the URL Format
Constraints: Projections
Constraints: Selections
Figuring Out Constraints
Client Parameters Revisited
Building and Installing netCDF-4 + DAP
Debugging Notes

3. RECAP: NETCDF-DAP Integration
Unidata has integrated the DAP protocol into the standard netCDF distribution.
It replaces the OPeNDAP provided libnc-dap library
It closely mimics the original libnc-dap translation
Implemented completely in C, versus libnc-dap, which is C++
This talk is primarily about the support for the netCDF-4 data model

4. RECAP: Architecture / Flow
GRIB
netCDF-3
HDF5 .
Internet
DAP Converter
Remote Data Server
DAP to netCDF Converter
NCO
(NC operators)
NCDUMP
Other
Programs
Data Request
Using
DAP URL
Data in DAP
Format
netCDF library
Part of the
netCDF Library
Web Server

5. DAP Elements DDS - describes the format of the data
DAS - describes associated attributes
DATADDS - the actual data; aka DODS
DATADDS contains two things:
A subset of the DDS followed by
The actual data

6. Web Debugging
If you are knowledgeable about DAP, you can look at the underlying DDS, DAS, DATADDS, and more, using a web browser
Procedure:
Remove the client parameters
Add to the base url one of the following extensions:
.dds, .das, .asc (or .ascii), .html, .info

7. Example DDS and DAS Dataset { Sequence { String Drifter_ID;
String Date_Sampled;
Float64 Latitude;
Float64 Longitude;
Float64 SST;
...
} Drifters;
} EOS.DBO;
Attributes {
FacilityGlobal {
String DataCenter "COAS";
String DrifterType "MetOcean"; }
Drifter_ID {
String Description
“Instrument unique identifier
Date_Sampled {
String Description "Date/time";
String Timezone "GMT";
}...

8. DAP to NETCDF-4 Translation
The client parameter [mode=netcdf4] will cause the DAP DDS and data to be translated to the netCDF-4 data model
netCDF-4 model is more powerful than DAP, so the translation only uses part of netCDF-4
Versus the netCDF-3 (classic) translation
Lossless
All DAP primitive types handled (no upgrading)
DAP sequences are translated to combinations of netCDF-4 compound types and vlen types
Performance improvement
No need for string or sequence dimensions

9. NETCDF-3 vs netCDF-4 translation
netcdf Drifters {
dimensions:
Drifters = 5 ; // Sequence
// dimension
stringdim64 = 64 ; // String
// dimension
variables:
char Drifters.Drifter_ID(Drifters,
stringdim64) ;
double Drifters.Latitude(Drifters);
double Drifters.Longitude(Drifters);
double Drifters.SST(Drifters);
netcdf Drifters {
types:
compound Drifters_record_t {
string Drifter_ID ;
double Latitude ;
double Longitude ;
double SST; };
Drifters_record_t(*) Drifters_t ;
variables:
Drifters_t Drifters ; … …

10. Extending the DAP URL Format
<clientparams><baseurl>?<constraint>
Additional client parameters
Constraint tells the OPeNDAP server what subset of the dataset to send
Constraint is divided into two parts
Projections
Selections
Currently disabled
Data Specified
By Full DDS
Data Specified
By Constraints

11. Client Parameters Revisited
Explicitly specify [mode=netcdf4] in order to get the netcdf-4 translation
Or specify NC_NETCDF4 mode to nc_open()
Additional [show=…] cases
[show=dds] (attribute “_dds”)
[show=das] (attribute “_das”)
[show=datadds] (attribute “_datadds”)
String attribute defining the DDS, DAS, or DDS part of the DATADS, respectively, as obtained from the OPeNDAP server

12. Constraints: Projections
Projections specify array slices of underlying OPeNDAP variables to send
Format: ?<projection>,<projection>,…
Projection is DAP variable name followed by a slice specification [first:stride:last]
Note: last, not count

13. Constraints: Selections
Selection constraints tell the OPeNDAP server to subset parts of the dataset based on the values in the dataset
Can only be applied to DAP Sequences
Format: &<selection>&<selection>&…
Multiple selections are treated as logical “and”
Selections always follow the projections, if any in the URL

14. Figuring out Constraints
How does one know what constraints to apply?
The problem is that they are defined in terms of the underlying DAP data model
=> Breaks the netCDF abstraction
For projections, the “show=projection” parameter can help by showing the underlying DAP variables
For Selections, you need to know something about the values in the original dataset
Look at the .info file?

15. Build and Install NETCDF4 + DAP
Add --enable-dap and --enable-netcdf-4 to your list of ./configure flags
Specify the curl library (libcurl) if ./configure cannot find it with-curl=<path to curl library>
As usual, of course, specify --with-hdf5 and --with-zlib

16. Example OPeNDAP Data Sources
?xmlfilename=datasets.xml&exfunction=none

17. Questions?

18. Demo Sites http://test.opendap.org:8080/dods/dts/Drifters