1. Clients

2. Accessing Data with DAP
Web Browser
Already discussed…
Graphical clients
ncBrowse, ODC, Ferret, GrADS
Command-line clients
getdap (UNIX, win32), loaddap (Matlab, IDL), nco (UNIX, win32)
Custom clients
C++, C, Java, Python
netCDF

3. Using a Graphical Client
Example: The OPeNDAP Data Connector
Combines data location with retrieval and display
Shows the built URL, including constraint expression
Can be transferred to another application

4. Start the ODC

5. The ODC opens to the search pane
Five different panes
Choices within a pane

6. Use the dataset list to find the TPAC climatologies
Choose the Antarctic Cooperative
Research Centre TPAC/CISRO
Climatologies
…then hit ‘To Retrieve’ to move
the selection to the next pane

7. The Retrieve pane Double click ‘levitus_annual_97.nc’
To see the contents of the file in
The area on the right

8. The ODC shows the URL as it builds it.
Click the checkbox for SALT and O2.
For both, set the range of z_index to
‘0 to 0’. Make sure to hit tab/return in
The boxes.
…then hit ‘Output to’ to move to the View pane

9. Getting the data…

10. There are a number of ways to view
The data. Here the plotter has been
Chosen (the default).
Hit ‘Plot to’ to generate a plot using the
Default settings.

11. When the plot is made, the interface
Switches to the ‘Preview’ tab
Switch back to the ‘Variables’ tab to
Plot O2

12. Choose ‘O2’ from the menu, then hit
‘Plot to.’

13. Now that the data have been read and Cached, you can switch back and
Forth between variables quickly without
Any additional data transfers
When ready, go back to the ‘Retrieve’
Pane.

14. Choose ‘TEMP’
Set the constraint
…then plot

16. ODC Summary The ODC provides a way to search for, access and plot data
Acts as a ‘URL builder;’ the URLs can be pasted into other applications
We didn’t need to know anything about DAP, its Request or Response objects or how a URL is used to request data
The data set list often contains stale entries
Also supports using the GCMD for data location - more on this when we cover searching

17. Using a Command-line Client
Matlab - demonstration
NCO - a powerful tool developed and maintained by another group

18. Matlab Demonstration of custom-built Graphical interfaces for Matlab
Matlab scripting is used to build the interfaces and provide some dataset-specific processing
A Matlab command extension is used to read the data (written in C/C++).
Two things are required in addition to Matlab: The DAP command extension (‘loaddap’) and the graphical interface software.
For the Matlab demo, you can try our GUIs. They are on the svn site underﾊml-ocean-testbed. When you have the ml-ocean-testbed directory setup, get into Matlab, cd to the ml-ocean-testbed directory and type O<tab> and select the first program (OCEAN_TOOLBOX, assuming that you are on a Mac) and run it. It should do two things - add /usr/bin to your path (as well as a testbed subdirectories to your path and bring up the Master GUI which links to all the others. The part that adds /usr/bin to your path may not work, so you might want to do that anyway so that the demo doesn't screw up. It's where loaddap is. Then click on either Pathfinder4km, GOES, SeaWinds or OAFlux (the other two don't work at present). To test it, just click on "Load last request" in the upper right hand corner. That will populate the GUI fields. Then "Get Data". It will load data into the Matlab workspace. Pathfinder4km has a progress indicator - not great, but it works. When the data have been acquired, type whos to see whats in the workspace. There should be new variables called opendap_0001, 0002,... depending on how many 2-d fields there are. If you acquired some fields before with another (or the same) GUI, it will increment the opendap_nnnn names. You can merge them with the merge command, but no need to do that in the demo - unless you really want to. The best thing to do is to simply plot. Type op_<tab> and choose op_plotimage. Complete with (1,opendap_0001) so the command should look likeﾊ>> op_plotimage(1,opendap_0001)This will plot the appropriate fields from opendap_0001 into figure 1. It will generally ask you what you want to plot and list the fields that are appropriate. (Sometimes it list ones that you would never want to plot - I need to fix that.) Choose the appropriate field and presto the data are plotted. One of the variables in the opendap_nnnn structure is URL. You can always source that directly in loaddap. However, the GUI does some stuff on the data that will not be done if you do it from the command line, like rename things, replace missing values with nan, convert to geophysical units, etc. But it does make for a good demo to add that in.I used the Pathfinder4km data set as I was going through this set of instructions. One nice thing about the "Load last command" is that you can get a nice image before hand and then simply reload. That shows the sort of canned mode. You can then do another one that is not already loaded in, like change the year.

19. Running the Matlab Demonstration
Start Matlab
Download the command extension
Download the interface software
In Matlab change directory to the ‘ml-ocean-testbed’ directory.
Type ‘OCEAN_TOOLBOX’
The interface will start…

20. The Ocean Toolbox

21. I choose the Pathfinder
Open a dataset
I choose the Pathfinder
dataset

22. Fill in the information
SST & Quality fields
Load data into the Matlab
workspace

23. Load data into the Matlab
Get the data
Load data into the Matlab
workspace

24. Plot/Display the data

25. Using the loaddap command extension directly
Start Matlab
Add the directory with the extension to the Matlab command path
Verify the command extension is working
Feed it a URL
Plot the data

26. Pass a URL, constrain the response To the ‘u’ and ‘v’ vectors only
Plot those vectors; See Figure 1

27. Matlab Summary Command line client is the tool used to move the data
Easily used in Matlab scripts to hide the details and make custom interfaces
To the the command extension directory, user must know:
Data location (URL)
Internal structure of the data set (syntactic metadata - DDS/DDX)
How to write a constraint expression
>> loaddap('
>> whos
Name Size Bytes Class
SST x struct array
Grand total is elements using bytes
>> SST
SST = SST: [90x180 double]
TIME: e+03
COADSY: [90x1 double]
COADSX: [180x1 double]
>> nn=find(SST.SST(:)<-10);
>> SST.SST(nn) = NaN;
>> imagesc(SST.COADSY,SST.COADSX,SST.SST);set(gca,'ydir','normal');colorbar;
>>
Here's what they do and why you need to do them (a blow-by-blow description).
>> whos Name Size Bytes Class
The next line uses 'find' to locate each array location with a missing value (you have to read the attributes to know what the missing values are set to).
The next line sets each array location that has a missing value to 'NaN', so that Matlab's graphics scaling will ignore those values when generating a colormap.
The next line contains several separate commands, delimited with ';'.The first command 'imagesc' is the 2-d graphics command the render a psuedo-color image. The second command sets the 'y-axis' to normal mode, otherwise it puts the smaller axis values at the top rather than the bottom and in this case the array is rendered upside down (at least from the user's perspective). The third command adds a colorbar. Voila..
>> imagesc(SST.COADSX,SST.COADSY,SST.SST);set(gca,'ydir','normal');colorbar;

28. NetCDF Operators (NCO)
Unix command line client
Unlike the previous two clients, NCO uses the netCDF client library to read from a DAP server
A client library is a collection of functions which hide the mechanics of (most of) the interaction with a server so the client can go about its business
The NCO client is, in fact, just the NCO package linked to our (OPeNDAP’s) version of the netCDF library (aka. the netCDF client library)

29. Build the NCO Software Change directory to /root/src/nco-3.9.2
cd /root/src/nco-3.9.2
Run configure to build the Makefile, then build and install the software
./configure
make
make install

30. Use NCO to Convert the FNOC1 vectors into a speed
NCAP: NCO Arithmetic Processor
ncap2 -s ‘speed=sqrt(u^2,v^2)’ spd.nc
The URL is the input ‘file’ and wndspd.nc is the output
Use ncdump to look at the result file
ncdump -h spd.nc
ncdump -v speed spd.nc

31. View the Result: ncBrowse
We can use ncBrowse to look at the local neetCDFZ we just built
ncBrowse can also look at the DAP server directly
Built using the DAP-enabled Java netCDF library (a client library where access to DAP servers hides behind the netCDF API)

32. Start ncBrowse Double click on ‘speed’ - the new data
we made with the previous NCO example

33. Fix up the latitude and longitude axes, the
‘Graph Variable.’

34. We have to be somewhat savvy about the
units - check back and look at the attributes…

35. Custom clients What options exist to build clients C++ using libdap
C using Ocapi
C,Fortran using the netcdf client library
Python using PyDAP
Java using Java-OPeNDAP
Matlab & IDL using the respective versions of loaddap

36. Clients Summary
Custom clients offer an opportunity to develop for a specific audience or a particular problem/project.
Example: ComMIT Tsunami inundation model client developed by NOAA/PMEL and BOM
General purpose clients like loaddap can read any kind of data while clients built using the netCDF client library are limited to the semantics of netCDF
Example: Record access is slow because each access is separate network request

37. Clients Summary, cont.
ODC: A client built specifically to provide a ‘browse’ capability for any data source
Uses Java-OPeNDAP
Loaddap: a client built to read any data into an analysis application
Can be used as a building block for more sophisticated applications
Use libdap (C++, Matlab) or Ocapi (C, IDL)
netCDF client library: A client-building tool
convert ‘legacy’ code
provide a simple way to write new applications
C++, C, Fortran

38. Finding Data Ways to find data:
The OPeNDAP Data Set List
GCMD
TPAC
Google
THREDDS
We maintain a page with links to dataset searching sites:

39. Common Features
All of these data location features except Google depend on active community involvement in building catalogs of data
The solutions can be described as static documents or crawlers
Google and TPAC are crawlers
Crawlers can discover datasets without human intervention
They can make mistakes that seem silly
The The Dataset List, GCMD and THREDDS are static documents or collections of static documents
Static lists can be tailored by hand
They can go out of date quickly

40. Differentiating Features
Google & TPAC:
Google is just crawling HTML. If a server is not linked to a HTML page, it won’t be found.
TPAC is preset with server locations and picks up changes at those sites

41. Differentiating Features, cont.
The Static Lists:
The Dataset List has a very low metadata requirement
Not maintained as actively as either GCMD or THREDDS catalogs
GCMD:
The GCMD has a fairly high entry level threshold
Professional staff maintain the GCMD as their sole job
THREDDS
THREDDS catalogs are, or can be, located at the data - locality distributes maintenance
Quality varies from site to site

42. Finding Data Summary
Locating data seems like it would be the place to start building a system, but it’s far more varied than the one-size-fits-all approach most tried in the 1990’s
Crawlers and hierarchical lists show the most promise but maintained centralized lists are also useful