1. Session 2: Using OPeNDAP-enabled Applications to Access Australian Data Services and Repositories eResearch Australasia 2011, ½ Day Morning Workshop, Thursday 10 th November 2011

2. GENERAL INFORMATION This is a half-day workshop (9am to 12:30pm) 9:00am Introductions and Participants Goals 9:15am Session 1: Discovering OPeNDAP data access services 10:00am Session 2: Applicable use cases of OPeNDAP data services −10:30am Tea Break for 15 minutes 11:00am Session 3: OPeNDAP service protocols and features 11:45am Session 4: Accessing complementary features and services 12:30pm End of Workshop 10:00am Session 2: Applicable use cases of OPeNDAP data services

3. Session 2 Applicable use cases of OPeNDAP data services for data cataloging and data access using a variety of applications and tools. A short tutorial exploring data access using an OPeNDAP-enabled tool within a scripting language such as python. 45 minutes in length + 15 minute Tea Break at 10:30am

4. Spectrum of Use Cases Application Data Representation OGC data model domain specific geospatial, 1-D, 2-D DAP2 data model domain neutral n-D, time series **DAP4 data model domain neutral new data types and data structures streaming, compressed, chunked Common Data Model (CDM) domain specific Future data model domain neutral?? Application Types Programmatic / Langauge API FORTRAN, C/C++, JAVA, Python, NetCDF, Java NetCDF Programmatic / Tools NetCDF, NCO, PyDAP Custom Tools: OPeNDAP crawler, ocean_prep Interactive Data Viewer IDV, Panolopy, IDL, MATLAB, iPython (matplotlib), NCL, web browser (metadata) Interactive Analysis MATLAB, IDL, iPython, NCL Custom Application: Inudation Modeller Web Application Live Access Server IMOS Data Portal (WMS) Custom Java Servlet Programming DAP2 Legacy Code existing tools DAP2 New Code New tools **DAP4 programming legacy code support **DAP4 programming new data model and protocols streaming support **DAP4 programming Asynchronous access modes, server-side processing Data Access Protocol Metadata Request das, dds, ddx ASCII/Binary Data Request Simple data representation DAP Binary Object RequestNcML Data Request aggregation, virtual data sets **DAP4 server-side operations, async access mode, new data model, posting Syntax Return data set info file.nc.dds - readable file.nc.ddx - XML file.nc.asc - ASCII data return Select variables file.nc.dods?var1,var2,var3 subset arrays file.dods?var1(0:1:10) Return file translations file.nc.netcdf - NetCDF file Server-side operations file.nc?GEOLOC() Async access mode ?? Clients Programmatic Access Tsunami inudation modeller, NetCDF, NCO, PyDAP, PyNetCDF, MATLAB, IDL, … Interactive Access Web browser - Catalog MATLAB, IDL, Python, Panolopy,… Data Library & Catalog Service metadata harvesting directory listings remote THREDDS services Web Service Java servlet, Java applet Geospatial Information Service OPeNDAP data service Analysis Service Live Access Server Service Capabilities DAP2 response metadata, dods, ASCII / Binary **DAP4 Response async access mode, server- side, streaming, NcML Aggregation service Virtual Data Set Service Remote Data Access Metadata Conversion and RDF metadata definitions, translations (-> ISO) sematics, ontalogy CF->ISO, CF->WMS, CF->WCS Layered Services Catalogue service WMS, WCS services Authentication Conformance checks CF metadata check ISO metadata check **DAP4 features listed is my estimation and not the official specification

5. Workshop Use-Cases Application Data Representation DAP2 data model domain neutral n-D, time series Application Types Programmatic / Langauge API FORTRAN, C/C++, JAVA, Python, NetCDF, Java Netcdf, PyDAP Programmatic / Tools NetCDF, NCO, PyDAP Custom Tools: OPeNDAP crawler Interactive Data Viewer Panolopy, MATLAB, NCL, web browser Programming DAP2 Legacy Code existing tools: DAP2 New Code New tools Data Access Protocol Metadata Request das, dds, ddx ASCII/Binary Data Request Simple data representation DAP Binary Object RequestNcML Data Request aggregation Syntax Return metadata info file.nc.das - readable file.nc.dds - readable file.nc.ddx - XML metadata file.nc.help - help info Select vars and return data file.nc.asc?var1,var2,var3 file.nc.dods?var1,var2,var3 subset arrays, return data file.asc?var1(0:1:10) file.dods?var1(0:1:10) Return file translations file.nc.netcdf - NetCDF file Server-side operations file.nc?GEOLOC() Clients Programmatic Access NetCDF, NCO, PyDAP, PyNetCDF Interactive Access Web browser - Catalog Python, MATLAB, Panolopy Service Capabilities DAP2 response THREDDS data service Hyrax data service NcML Aggregation service Layered Services Catalog service WMS

6. Use Case limitations Time to access data is dependent on the following factors: Hardware and network performance Selection of variables and dimensions Number of data requests to be issued −Latency inherent in the data request Number of concurrent accesses to the server

7. Performance limitations to data delivery Network connection Network Bandwidth Data Transfer (MB per second) *Elapse Time (seconds) WiFi2 – 56 Mbps0.2 – 5 MBps500+ ADSL modem2 – 14 Mbps0.2 – 1.4 MBps357+ Home LAN100 Mbps10 MBps50 SATA Disk20 – 40 MBps12.5+ Office LAN1000 Mbps100 MBps~5.00 Disk Array120 – 240 MBps~3.00 Backbone Ethernet10 Gbps1,000 MBps~0.50 QDR Infiniband40 Gbps4,000 MBps~0.12 Lustre Parallel FS10,000 MBps0.05+ *Time to transfer a 500 MB data object

8. Performance limitations to data delivery Data RequestData Size*Elapse Time (seconds) Improved Access Complete File 3 x fields(3D) doubles 250 MB1781.0x One 3D field 5 vertical levels 50 MB35.75.0x 3 x 2D fields 30 MB21.48.3x One 2D field (1250x1000) 10 MB7.1424.9x Subset 3D field (500 x 500 x 5 ) 10 MB7.1424.9x Subset 2D field (500 x 500 ) 2 MB1.43124x Vertical Column (100 x 100 x 5 ) 0.4 MB0.28635x *Time to transfer a data object on an ADSL2 modem = 14 Mbps

9. DAP-enabled client tools/applications OPeNDAP Clients (partial list) http://opendap.org/whatClients To be demo’ed today. 1.Web browser returning ASCII data 2.Pydap - is a pure Python library implementation of the DAP2 3.NetCDF - is a set of software libraries and self-describing, machine-independent data formats with interfaces to Python, FORTRAN, C/C++, and Java languages 4.NCO – comprises a dozen standalone, command-line programs that take netCDF files as input 5.MATLAB – session 3 6.Panoply – session 4

10. Web Browser demo “.ascii” tells the OPeNDAP service to return the data in ASCII format. −http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/201111 06-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lonhttp://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/201111 06-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon Try accessing multiple variables such as latitude −http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/201111 06-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon,lathttp://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/201111 06-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon,lat What other variables are available in the file? −Try accessing “sst” and download to ascii

11. Tutorial - Subsetting Modify the variable indice (C array syntax 0..n-1 ), and request ascii data in the web browser http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106- ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[10:1:50]http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106- ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[10:1:50 What happens if the middle index number is changed from “1” to “2”? http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106- ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[10:2:50]http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106- ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[10:2:50

12. Tutorial – subsetting continued http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106- ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[0:1:1439http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106- ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[0:1:1439] Add a new variable to the above URL, separated by a comma, and make a request for ascii data in the web browser: http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106- ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[0:1:50],lat[0:1:30]http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106- ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[0:1:50],lat[0:1:30 Now do the same thing in the form and modify the indice range watch out for large indice ranges returning large amounts of data

13. Tutorial:.dods response Try the binary response “.dods” “.dods” tells the OPeNDAP service to return the data in binary format −http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/201111 06-ABOM-L4LRfnd-GLOB-v01-fv01.nc.dods?lonhttp://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/201111 06-ABOM-L4LRfnd-GLOB-v01-fv01.nc.dods?lon This is two part binary DAP data object which contains 1) meta data, and 2) binary data structure. This is the typical response for OPeNDAP enabled client applications.

14. Pydap Pydap is a pure Python library implementing the Data Access Protocol, also known as DODS or OPeNDAP. You can use Pydap as a client or server. http://pydap.org/ To install Pydap on Windows… see the next slide for Windows To install Pydap on Mac OS X… see the slide for Mac OS X To install Pydap on Linux… see the slide for Mac OS X

15. Pydap installation for Windows To install Pydap on Windows … 1.Install python onto Windows 2.Install easy_install: ez_setup.py 3.Install Pydap: easy_install Pydap

16. Pydap installation for Mac OS X To install Pydap on Mac OS X… 1.Python is install on Mac OS 10.5 and 10.6 by default 2.Install easy_install: ez_setup.py 3.Install Pydap: easy_install Pydap

17. Test Pydap client installation >>> from pydap.client import open_url >>> dataset = open_url('http://test.opendap.org/dap/data/nc/coads_climatology.nc') >>> var = dataset['SST'] >>> var.shape (12, 90, 180) >>> var.type >>> print var[0,10:14,10:14] # this will download data from the server with data [[ -1.26285708e+00 -9.99999979e+33 -9.99999979e+33 -9.99999979e+33] [ -7.69166648e-01 -7.79999971e-01 -6.75454497e-01 -5.95714271e-01] [ 1.28333330e-01 -5.00000156e-02 -6.36363626e-02 -1.41666666e-01] [ 6.38000011e-01 8.95384610e-01 7.21666634e-01 8.10000002e-01]] and axes 366.0 [-69. -67. -65. -63.] [ 41. 43. 45. 47.]

18. More Pydap client features See Pydap client: http://pydap.org/client.htmlhttp://pydap.org/client.html

19. NetCDF API and Tools NetCDF is a set of software libraries and self-describing, machine- independent data formats that support the creation, access, and sharing of array-oriented scientific data. http://www.unidata.ucar.edu/software/netcdf/ To install, go to http://www.unidata.ucar.edu/downloads/netcdf/index.jsp To use with python, build netCDF4 and its python module, or … easy_install netCDF4

20. NetCDF demo >>> import netCDF4 >>> url = 'http://test.opendap.org/dap/data/nc/coads_climatology.nc’ >>> dataset = netCDF4.Dataset(url) >>> var = dataset.variables['SST'] >>> var.shape (12, 90, 180) >>> print var[0,10:14,10:14] # this will download data from the server with data [[-1.26285707951 -- -- --] [-0.769166648388 -0.77999997139 -0.675454497337 -0.595714271069] [0.128333330154 -0.0500000156462 -0.0636363625526 -0.141666665673] [0.638000011444 0.895384609699 0.721666634083 0.810000002384]] >>> print var float32 SST('TIME', 'COADSY', 'COADSX') …

21. NetCDF demo Get metadata information about the following data set: ncdump -h http://opendap.bom.gov.au:8080/thredds/dodsC/nmoc/oceanmaps2_ofa m_fc/latest/ocean_fc_20111108_000_surface.nc http://opendap.bom.gov.au:8080/thredds/dodsC/nmoc/oceanmaps2_ofa m_fc/latest/ocean_fc_20111108_000_surface.nc

22. NCO Tools The netCDF Operators (NCO) comprise a dozen standalone, command-line programs that take netCDF files as input, then operate (e.g., derive new data, average, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats. NCO aids manipulation and analysis of gridded scientific data. http://nco.sourceforge.net/ To install NCO tools, go to http://nco.sourceforge.net/#Binaries

23. NCO tool demo Download the initial conditions for regional ocean model ncks -O -F -d xt_ocean,648,979 -d yt_ocean,467,798 http://opendap.bom.gov.au:8080/thredds/dodsC/oceanmaps_access_an alysis_ogcm/temp/2010/ocean_an_20100312_temp.nc -o ocean_temp_2010_03_12.nc http://opendap.bom.gov.au:8080/thredds/dodsC/oceanmaps_access_an alysis_ogcm/temp/2010/ocean_an_20100312_temp.nc -o ocean_temp_2010_03_12.nc View the original file metadata: ncdump –h http://opendap.bom.gov.au:8080/thredds/dodsC/oceanmaps_access_an alysis_ogcm/temp/2010/ocean_an_20100312_temp.nc -o ocean_temp_2010_03_12.nc http://opendap.bom.gov.au:8080/thredds/dodsC/oceanmaps_access_an alysis_ogcm/temp/2010/ocean_an_20100312_temp.nc -o ocean_temp_2010_03_12.nc View the subsetted file’s metadata ncdump –h ocean_temp_2010_03_12.ncocean_temp_2010_03_12.nc

24. NCO demo Download the initial conditions for regional ocean model using longitude and latitudes ranges for the dimensions ncks -O -F -d xt_ocean,143.55,176.66 -d yt_ocean,-28.35,4.75 http://opendap.bom.gov.au:8080/thredds/dodsC/oceanmaps_access_an alysis_ogcm/temp/2010/ocean_an_20100312_temp.nc -o ocean_temp2_2010_03_12.nc http://opendap.bom.gov.au:8080/thredds/dodsC/oceanmaps_access_an alysis_ogcm/temp/2010/ocean_an_20100312_temp.nc -o ocean_temp2_2010_03_12.nc Are the files the same (dimensions and lon/lat range)? ncdump –v xt_ocean ocean_temp_2010_03_12.ncocean_temp_2010_03_12.nc ncdump –v xt_ocean ocean_temp2_2010_03_12.ncocean_temp2_2010_03_12.nc

25. Tutorial: Pick a demo to try Please select from pydap, NetCDF, and NCO demos 1.Install the software on your machine 2.Run a test case and see if the software is installed correctly 3.Access a different file from a TDS or Hyrax data service 4.Get the metadata information 5.Get the coordinate axes data 6.Get a subset of data from an array

26. Thank you Authors: Tim F. Pugh 1, James Gallagher 2, Dave Fulker 3 1 Australian Bureau of Meteorology, Melbourne, Australia, t.pugh@bom.gov.au 2 OPeNDAP, Butte, Montana, USA, jgallagher@opendap.orgjgallagher@opendap.org 3 OPeNDAP, Boulder, Colorado, USA, dfulker@opendap.org