1. Accessing Remote Datasets using the DAP protocol through the netCDF interface. Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009

2. Overview Goal: Access to Remote Data What is OPeNDAP and DAP? The DAP-netCDF Integration Architecture/Flow Example DDS and DAS netCDF Translation Glitches netCDF Translation Example Specifying a DAP Data Source General URL Format General Format: Client Parameters Defined Client Parameters General Format: Base URL Debugging Note Building and Installing NETCDF+DAP

3. Goal: Access to Remote Data Remote access allows the netCDF library to retrieve data that is stored on other machines in the internet. There are basically two ways to do this. Method 1: Copy the remote data using e.g. FTP or HTTP.  Great if the data is already a.nc file,  And, you have disk space to hold the data.  Otherwise, find and install conversion program to convert to netcdf format (tedious and error prone)

4. Remote Access (cont.) Method 2: Let others do the work for you. You tell netCDF where to get a dataset from some remote machine. The remote machine automatically converts the dataset to a standard form (called DAP) The DAP formatted data is sent to your machine. The netcdf library converts the DAP data to netcdf data for your use.

5. What is OPeNAP and DAP? DAP is a widely supported protocol and standard data format for accessing remote data Defined and maintained by the OPenDAP organization (http://www.opendap.org/) DAP was expressly designed to serve as intermediate format for accessing a wide variety of data sources

6. DAP (cont.) DAP represents a stream of data, not a file on a disk netCDF defines the structure and the attributes together, DAP divides them into two parts:  DDS - describes the format of the data  DAS - describes associated attributes Plus: DATADDS - the actual data; the data also includes a partial DDS at the front  aka DODS

7. The NETCDF-DAP Integration Unidata has integrated the DAP protocol into the standard netCDF distribution. It replaces the OPeNDAP provided libnc-dap library It closely mimics the original libnc-dap translation It is implemented completely in C, versus libnc-dap, which is C++ => You can use it with FORTRAN

8. Architecture / Flow GRIBnetCDF-3HDF5...... Internet DAP Converter Remote Data Server DAP to netCDF Converter NCO (NC operators) NCDUMP Other Programs Data Request Using DAP URL Data in DAP Format netCDF library Part of the netCDF Library Web Server

9. Example DDS and DAS Dataset { Sequence { String Drifter_ID; String Date_Sampled; Float64 Latitude; Float64 Longitude; Float64 SST;... } Drifters; } EOS.DBO; Attributes { FacilityGlobal { String DataCenter "COAS"; String DrifterType "MetOcean"; } Drifter_ID { String Description “Instrument unique identifier } Date_Sampled { String Description "Date/time"; String Timezone "GMT"; }...

10. NETCDF-3 Translation Glitches Loss’y: Some data is inaccessible String type: netCDF-3 no, DAP yes So, the netCDF variable is type as char, but it is given extra dimension to represent the string  String var(d1,d2) => char var(d1,d2,stringdim1)  Stringdim1 is fixed length (defaults to 64)

11. Translation Glitches (cont.) Unsigned types: netCDF-3 no, DAP yes So, we upgrade the type  E.g. unsigned short => int, unsigned int => double Sequence dimensions  Think of the number of records in a relational table  This number is represented as a dimension  Used to dimension variables that represent fields of the relational table

12. NETCDF-3 Translation Example The netCDF-3 translation of the previous DDS+DAS netcdf Drifters { dimensions: Drifters = 5 ; // Sequence dimension stringdim64 = 64 ; // String dimension variables: char Drifters.Drifter_ID(Drifters, stringdim64) ; Drifters.Drifter_ID:Description = “Instrument unique identifier”; double Drifters.Latitude(Drifters); double Drifters.Longitude(Drifters); double Drifters.SST(Drifters); // Global Attributes :DataCenter = "COAS“... DEMO

13. Specifying a DAP Data Source A DAP data source is specified using an extended URL syntax that refers to the DAP server containing that data Used in place of a file name in e.g. ncdump  Beware: Shell may require quotes around URL Example: [show=url;seqdims]http://test.opendap.org:8080/ dods/dts/Drifters

14. General URL Format This format extends normal URL format by adding optional client parameters The client parameters are specific to the DAP/netCDF integration  But, it has its roots in the original OPeNDAP provided C++ based libnc-dap library

15. General Format: Client Parameters One or more things enclosed in brackets e.g […][…]… A set of brackets contains one or more = pairs separated by commas e.g. [x=y,a=b] The value part may be empty or it may be a list of values separated by semicolons e.g [x,y=a;b;c] If a parameter name is repeated, the values are merged using semicolons e.g [a=b,a=c] is equivalent to [a=b;c]

16. Defined Client Parameters mode (e.g. mode=netcdf3) Specify if we want  DAP -> netcdf-3, or  DAP -> netcdf-4 Possible values (choose 1): libnc-dap | netcdf3 | netcdf-3 | classic netcdf4 | netcdf-4 | cdm Default is mode=netcdf3

17. Defined Client Parameters show (e.g. show=url;projection) Specify additional attributes to include in the translation. show=translate (attribute = “_translation”)  Displays what mode was used to translate.  E.g. netcdf3 vs netcdf4 show=url (attribute = “_url”)  Displays the url that was passed to nc_open.

18. Defined Client Parameters (cont.) show=seqdims (attribute = “_sequence_dims”)  List dimensions that actually come from underlying DAP sequences  Also, don’t compute the sequence length  Gives useful cost information

19. General Format: Base URL Little to say, it specifies the DAP server from which to retrieve the data But: should never end in.dds,.das, or.dods  DAP code will take care of the extensions DEMO

20. Debugging Note There is a DAP specific logging mechanism that can sometimes provide extra information  Set the environment variable OCLOGFILE  E.g. for /bin/sh export OCLOGFILE=“” If you ask for the whole dataset (using e.g. ncdump), some servers will not serve up that much data and you will get an error whose cause is not obvious unless you turn on logging If you send a support request: include logging

21. Build and Install NETCDF+DAP 1.Add --enable-dap to your list of./configure flags 2.DAP requires the curl library (libcurl) 3.If your./configure fails because it cannot find libcurl, then you must specify --with-curl= Exactly analogous to how you would specify the path to HDF5, for example 4.If you want to thoroughly test that dap is working, then add --enable-dap-remote-tests

22. Example OPeNDAP DATA Sources http://test.opendap.org:8080/dods/dts http://test.opendap.org http://test.opendap.org/dap/data/ff http://www.usgodae.org/dods/GDS_NC/gdem http://www.opendap.org/data/datasets.cgi ?xmlfilename=datasets.xml&exfunction=none http://www.opendap.org/data/datasets.cgi

23. Questions?

24. Demos Sites http://test.opendap.org:8080/dods/dts/Drifters http://test.opendap.org http://test.opendap.org/dap/data/ff http://www.usgodae.org/dods/GDS/coamps_cent_am http://www.opendap.org/data/datasets.cgi?xmlfilename=d atasets.xml&exfunction=none