Weathertop Consulting, LLC Server-side OPeNDAP Analysis – Concrete steps toward a generalized framework via a reference implementation using F-TDS Roland Schweitzer Weathertop Consulting, LLC Steve Hankin and Ansley Manke NOAA/PMEL

Weathertop Consulting, LLC Stuff I thought I’d get done this summer, but now can only talk about and ask for help. Roland Schweitzer Weathertop Consulting, LLC

Highlights Server-side analysis Server-side analysis Motivation Motivation Implementation Implementation LAS as OPeNDAP Client and Server LAS as OPeNDAP Client and Server Implementation of the server (and why you might care) ‏ Implementation of the server (and why you might care) ‏ Community participation and “call to action” Community participation and “call to action” Summary Summary

Weathertop Consulting, LLC Server-side Analysis In general server-side analysis is a computation made by an OPeNDAP server at the request of a client. In general server-side analysis is a computation made by an OPeNDAP server at the request of a client. The specification of the computation is transmitted to the server via the OPeNDAP URL. The specification of the computation is transmitted to the server via the OPeNDAP URL.

Weathertop Consulting, LLC Motivation and Implementation We are interested in server-side analysis for use with the Live Access Server (LAS). We are interested in server-side analysis for use with the Live Access Server (LAS). Primarily to implement comparisons between data defined on different grids Primarily to implement comparisons between data defined on different grids Our implementation borrows heavily from GDS and our experience of running a legacy application (Ferret) from within a Java runtime environment. Our implementation borrows heavily from GDS and our experience of running a legacy application (Ferret) from within a Java runtime environment.

Weathertop Consulting, LLC A highly configurable Web server designed to provide flexible access to geo-referenced scientific data The Live Access Server (LAS) ‏

Weathertop Consulting, LLC LAS Architecture Product Server SQL Backend Service DRDS Backend Service Metadata (XML) ‏ Local RDBMS LAS Product Server client product metadata product request XML (REST)‏ back end request (SOAP)‏ netCDF data OPeNDAP server Remote RDBMS DRDS server Local netCDF data Ferret Backend Service Ferret

Weathertop Consulting, LLC Comparing OPeNDAP datasets metadata product request XML (REST)‏ back end request (SOAP)‏ Product Server SQL Backend Service DRDS Backend Service Metadata (XML) ‏ LAS user product netCDF data 2 OPeNDAP server netCDF data 1 OPeNDAP server Suppose the variables are on different grids? Ferret Backend Service Ferret

Weathertop Consulting, LLC LAS as an OPeNDAP Server Data on grids which are available via LAS are guaranteed to be geo-referenced and at least COARDS compliant. Data on grids which are available via LAS are guaranteed to be geo-referenced and at least COARDS compliant. We can often “repair” (including re-gridding) the data and/or metadata by writing a script of Ferret commands. We can often “repair” (including re-gridding) the data and/or metadata by writing a script of Ferret commands. Wouldn’t it be nice to make these “repaired” data available via OPeNDAP? Wouldn’t it be nice to make these “repaired” data available via OPeNDAP?

Weathertop Consulting, LLC The Ferret-THREDDS Data Server F-TDS makes this possible. F-TDS makes this possible. F-TDS provides an OPeNDAP view of the data being served by LAS and can make any transformations specified by the associated script before serving the data. F-TDS provides an OPeNDAP view of the data being served by LAS and can make any transformations specified by the associated script before serving the data. F-TDS also implements server-side analysis (including the ability to pass in external data sources). F-TDS also implements server-side analysis (including the ability to pass in external data sources).

Weathertop Consulting, LLC A GDS Digression The GrADS Data (DODS) Server was an early implementation of this concept. The GrADS Data (DODS) Server was an early implementation of this concept. Both GDS and F-TDS use the Java Runtime environment to invoke the associated legacy app (GrADS or Ferret) to do the heavy lifting. Both GDS and F-TDS use the Java Runtime environment to invoke the associated legacy app (GrADS or Ferret) to do the heavy lifting.

Weathertop Consulting, LLC F-TDS Capabilities F-TDS takes advantage of several characteristics of Ferret. F-TDS takes advantage of several characteristics of Ferret. New "virtual" data variables can be defined New "virtual" data variables can be defined Can build the metadata (netCDF header described by dimensions, coordinate variables and the structure of data variables) without performing any heavy calculations for both data read from files and “virtual” data variables Can build the metadata (netCDF header described by dimensions, coordinate variables and the structure of data variables) without performing any heavy calculations for both data read from files and “virtual” data variables Only performs calculations when the data are requested Only performs calculations when the data are requested Only calculates the minimal set needed to fulfil the current request Only calculates the minimal set needed to fulfil the current request

Weathertop Consulting, LLC F-TDS Evolution The Java netCDF library allows new data container formats to be plugged-in by implementing the I/O Service Provider interface. The Java netCDF library allows new data container formats to be plugged-in by implementing the I/O Service Provider interface. Once “plugged-in” clients using nj22 have access to the data from this container. Once “plugged-in” clients using nj22 have access to the data from this container. We implemented a Ferret I/O Service provider which can read Ferret command scripts and direct Ferret to perform the calculations as needed to satisfy data requests. We implemented a Ferret I/O Service provider which can read Ferret command scripts and direct Ferret to perform the calculations as needed to satisfy data requests.

Weathertop Consulting, LLC The THREDDS Data Server TDS is an OPeNDAP server based on netCDF Java. TDS is an OPeNDAP server based on netCDF Java. nj22 I/O Service Providers can be plugged in to TDS. nj22 I/O Service Providers can be plugged in to TDS. The combination of the Ferret I/O Service Provider and TDS (aka F-TDS) serves via OPeNDAP data which are represented by Ferret command scripts (both data read from disk by Ferret and virtual data computed on-the-fly by Ferret). The combination of the Ferret I/O Service Provider and TDS (aka F-TDS) serves via OPeNDAP data which are represented by Ferret command scripts (both data read from disk by Ferret and virtual data computed on-the-fly by Ferret).

Weathertop Consulting, LLC The Why You Should Care If you already serve data via TDS you can add server-side analysis to your TDS for the cost of installing Ferret, and a few Java classes and configuration files... If you already serve data via TDS you can add server-side analysis to your TDS for the cost of installing Ferret, and a few Java classes and configuration files... and what every cycles get used by the analysis... and what every cycles get used by the analysis... which may be cheaper than the data transfer which may be cheaper than the data transfer

Weathertop Consulting, LLC F-TDS and Server-Side Analysis A DataSource handler can also be plugged in to TDS which allows custom handling of OPeNDAP requests based on the contents of the URL. A DataSource handler can also be plugged in to TDS which allows custom handling of OPeNDAP requests based on the contents of the URL. We built such a DataSource handler which recognizes URLs of the form: We built such a DataSource handler which recognizes URLs of the form: _expr_{dataset1,dataset2,...} {expression1;expression2;...}.URLsuffix?projection

Weathertop Consulting, LLC LAS Example

Weathertop Consulting, LLC F-TDS Specific Example _expr_ { sst_2_regrid=sst[d=2,gxy=temp[d=1]]}.dods?SST_2_REGRID[0:1:0][67:1:131][0:1:14] /thredds/dodsC/data/levitus.dods?TEMP[0:1:0][67:1:131][0:1:14] /coads.dods?SST[0:1:0][33:1:66][124:1:179]

Weathertop Consulting, LLC Community Action Can we define a general syntax for server-side analysis requests? Can we define a general syntax for server-side analysis requests? Some common operations (ave, diff, linear interp.) with standard syntax Some common operations (ave, diff, linear interp.) with standard syntax Still allow server-specific (native) operations Still allow server-specific (native) operations Can we define a general syntax for geo- referenced projections? Can we define a general syntax for geo- referenced projections? ERDDAP’s (180),10,(360) syntax ERDDAP’s (180),10,(360) syntax TDS’s north=17.3&south=12.088&west=180&east=360.0 netCDF subset syntax TDS’s north=17.3&south=12.088&west=180&east=360.0 netCDF subset syntax

Weathertop Consulting, LLC OPeNDAP “Working” Group Working is in quotes because we have not been doing any work recently Working is in quotes because we have not been doing any work recently See: See:

Weathertop Consulting, LLC Summary Server-side analysis is critical for LAS. Server-side analysis is critical for LAS. NetCDF Java and the THREDDS Data Server is a great platform for implementing this type of analysis with a legacy analysis application (like Ferret). NetCDF Java and the THREDDS Data Server is a great platform for implementing this type of analysis with a legacy analysis application (like Ferret). A community-developed server-side analysis framework would make it easier to get the advantages of server-side analysis from other servers. A community-developed server-side analysis framework would make it easier to get the advantages of server-side analysis from other servers.

Weathertop Consulting, LLC IOSP* *Stolen directly from John Caron with only this measly acknowledgement. Application The low-level part of the NetCDF-Java version 2.2 architecture NetCDF-3 HDF5 I/O service provider GRIB GINI NIDS NetcdfFile NetCDF-4 Nexrad DMSP Java Runtime Ferret GrADS readData open isValid