1. Weathertop Consulting, LLC Server-side OPeNDAP Analysis - A General Approach Utilizing Legacy Applications through TDS Roland Schweitzer Weathertop Consulting, LLC Steve Hankin and Ansley Manke NOAA/PMEL

2. Weathertop Consulting, LLC Highlights Server-side analysis Server-side analysis Motivation Motivation LAS as OPeNDAP Client and Server LAS as OPeNDAP Client and Server Evolution of the server implementation Evolution of the server implementation Community call to action Community call to action Summary Summary

3. Weathertop Consulting, LLC Server-side Analysis In general server-side analysis is a computation made by an OPeNDAP server at the request of a client. In general server-side analysis is a computation made by an OPeNDAP server at the request of a client. The specification of the computation is transmitted to the server via the OPeNDAP URL. The specification of the computation is transmitted to the server via the OPeNDAP URL.

4. Weathertop Consulting, LLC Motivation We are interested in server-side analysis for use with the Live Access Server (LAS). We are interested in server-side analysis for use with the Live Access Server (LAS). Primarily as a way to implement comparisons between data defined on different grids Primarily as a way to implement comparisons between data defined on different grids We want our implementation to leverage the analysis capabilities of legacy applications like Ferret and GrADS. We want our implementation to leverage the analysis capabilities of legacy applications like Ferret and GrADS. We want to use our experience running legacy applications (like Ferret) from within a Java runtime environment. We want to use our experience running legacy applications (like Ferret) from within a Java runtime environment.

5. Weathertop Consulting, LLC A highly configurable Web server designed to provide flexible access to geo-referenced scientific data The Live Access Server (LAS)

6. Weathertop Consulting, LLC LAS Architecture Product Server SQL Backend Service DRDS Backend Service Metadata (XML) Local RDBMS LAS Product Server client product metadata product request XML (REST) back end request (SOAP) netCDF data OPeNDAP server Remote RDBMS DRDS server Local netCDF data Ferret Backend Service Ferret

7. Weathertop Consulting, LLC Comparing OPeNDAP datasets metadata product request XML (REST) back end request (SOAP) Product Server SQL Backend Service DRDS Backend Service Metadata (XML) LAS user product netCDF data 2 OPeNDAP server netCDF data 1 OPeNDAP server Suppose the variables are on different grids? Ferret Backend Service Ferret

8. Weathertop Consulting, LLC LAS as an OPeNDAP Server Data on grids which are available via LAS are guaranteed to be geo-referenced and at least COARDS compliant. Data on grids which are available via LAS are guaranteed to be geo-referenced and at least COARDS compliant. We can often repair (including re-gridding) the data and/or metadata by associating a script of Ferret commands with the data source in the LAS configuration. We can often repair (including re-gridding) the data and/or metadata by associating a script of Ferret commands with the data source in the LAS configuration. Wouldnt it be nice to make these repaired data available via OPeNDAP? Wouldnt it be nice to make these repaired data available via OPeNDAP?

9. Weathertop Consulting, LLC The Ferret Data Server FDS made this possible. FDS made this possible. FDS provides an OPeNDAP view of the data being served by LAS and makes any transformations specified by the associated script before serving the data. FDS provides an OPeNDAP view of the data being served by LAS and makes any transformations specified by the associated script before serving the data. FDS also implements server-side analysis (including the ability to pass in external data sources). FDS also implements server-side analysis (including the ability to pass in external data sources).

10. Weathertop Consulting, LLC A GDS Digression The GrADS Data (DODS) Server is the first implementation of this concept. The GrADS Data (DODS) Server is the first implementation of this concept. In fact, FDS used the Anagram framework upon which GDS is built. In fact, FDS used the Anagram framework upon which GDS is built. Both GDS and FDS use the Java Runtime environment to invoke the associated legacy app (GrADS or Ferret) to do the heavy lifting. Both GDS and FDS use the Java Runtime environment to invoke the associated legacy app (GrADS or Ferret) to do the heavy lifting.

11. Weathertop Consulting, LLC FDS Capabilities FDS took advantage of several characteristics of the underlying engine (Ferret). FDS took advantage of several characteristics of the underlying engine (Ferret). New "virtual" data variables can be defined New "virtual" data variables can be defined Can build the metadata (netCDF header described by dimensions, coordinate variables and the structure of data variables) without performing any heavy calculations for both data read from files and virtual data variables Can build the metadata (netCDF header described by dimensions, coordinate variables and the structure of data variables) without performing any heavy calculations for both data read from files and virtual data variables Only performs calculations when the data are requested Only performs calculations when the data are requested Only calculates the minimal set needed to fulfill the current request Only calculates the minimal set needed to fulfill the current request

12. Weathertop Consulting, LLC FDS Evolution Keep these advantages and evolve the implementation. Keep these advantages and evolve the implementation. The Java netCDF library allows new data container formats to be plugged-in by implementing the I/O Service Provider interface. The Java netCDF library allows new data container formats to be plugged-in by implementing the I/O Service Provider interface. Once plugged-in clients using nj22 have access to the data from this container. Once plugged-in clients using nj22 have access to the data from this container. We implemented a Ferret I/O Service provider which can read Ferret command scripts and direct Ferret to perform the calculations as needed to satisfy data requests. We implemented a Ferret I/O Service provider which can read Ferret command scripts and direct Ferret to perform the calculations as needed to satisfy data requests.

13. Weathertop Consulting, LLC Example % more data/simple4.jnl use levitus_climatology let/d=levitus_climatology temp_20 =temp[d=levitus_climatology,z=0:20@sum] set var/title="surface heat content"/units="deg C temp_20[d=levitus_climatology]

14. Weathertop Consulting, LLC Example dncdump -c http://porter.pmel.noaa.gov:8920/thredds/dodsC/mydata/simple4.jnl float TEMP(ZAXLEVITR, YAXLEVITR, XAXLEVITR) ; TEMP:units = "DEG C" ; TEMP:long_name = "TEMPERATURE" ; TEMP:_FillValue = -1.e+10f ; TEMP:missing_value = -1.e+10f ; TEMP:dataset = "levitus_climatology.cdf" ; float TEMP_20(YAXLEVITR, XAXLEVITR) ; TEMP_20:dataset = "levitus_climatology" ; TEMP_20:direction = "IJ" ; TEMP_20:units = "deg C" ; TEMP_20:long_name = "surface heat content" ; TEMP_20:missing_value = "-1.00000E+34" ; TEMP_20:virtual = "true" ;

15. Weathertop Consulting, LLC IOSP* *Stolen directly from John Caron with only this measly acknowledgement. Application The low-level part of the NetCDF-Java version 2.2 architecture NetCDF-3 HDF5 I/O service provider GRIB GINI NIDS NetcdfFile NetCDF-4 Nexrad DMSP Java Runtime Ferret GrADS readData open isValid

16. Weathertop Consulting, LLC The THREDDS Data Server TDS is an OPeNDAP server. TDS is an OPeNDAP server. nj22 I/O Service Providers can be plugged in to TDS. nj22 I/O Service Providers can be plugged in to TDS. The combination of the Ferret I/O Service Provider and TDS (aka F-TDS) serves via OPeNDAP data which are represented by Ferret command scripts (both data read from disk by Ferret and virtual data computed on-the-fly by Ferret). The combination of the Ferret I/O Service Provider and TDS (aka F-TDS) serves via OPeNDAP data which are represented by Ferret command scripts (both data read from disk by Ferret and virtual data computed on-the-fly by Ferret).

17. Weathertop Consulting, LLC F-TDS and Server-Side Analysis A DataSource handler can also be plugged in to TDS which allows custom handling of OPeNDAP requests based on the contents of the HTTPServletRequest Object (and by implication the URL). A DataSource handler can also be plugged in to TDS which allows custom handling of OPeNDAP requests based on the contents of the HTTPServletRequest Object (and by implication the URL). We built such a DataSource handler which recognizes URL with embedded analysis expressions. We built such a DataSource handler which recognizes URL with embedded analysis expressions. The three groups of input to the server-side analysis were copied from GDS. The three groups of input to the server-side analysis were copied from GDS. The three sets are: The three sets are: Data sources (e.g. OPeNDAP URLs). Data sources (e.g. OPeNDAP URLs). Analysis commands which are implementation specific. Analysis commands which are implementation specific. A sub-region. A sub-region.

18. Weathertop Consulting, LLC An Analysis URL http://machine:port/thredds/dodsC/_expr_{dataset1,dataset2,...}{expression1;expression2;...}{region}.URLsuffix?constraint Part of the original GDS specification, not necessary and not often used with FDS

19. Weathertop Consulting, LLC FDS-Specific Example http://host.gov:9090/thredds/dodsC/data/coads _expr_ {http://host.gov:9090/thredds/dodsC/data/levitus} {DIF=SST[d=1]-TEMP[d=2,g=SST[d=1]]}.asc?DIF

20. Weathertop Consulting, LLC Community Action Can we define a implementation-indepedent syntax for server-side analysis requests? Can we define a implementation-indepedent syntax for server-side analysis requests? Some common operations (averaging, differencing, linear interpolation) with standard name and syntax Some common operations (averaging, differencing, linear interpolation) with standard name and syntax Server-specific (native) operations Server-specific (native) operations What mechanism? What mechanism? An encoded XML string? An encoded XML string? We use this technique in LAS with good success We use this technique in LAS with good success A simple command language? A simple command language?

21. Weathertop Consulting, LLC XML Analysis Expression "my_local_dset.nc" "http://remote_dset.nc" DIFF sst sst "my_local_dset.nc" "http://remote_dset.nc" DIFF sst sst

22. Weathertop Consulting, LLC With native Operation my_local_dset.nc http://remote_dset.nc my_local_dset.nc http://remote_dset.nc LET diff = sst[d=2]-sst[d=1] LET diff = sst[d=2]-sst[d=1]

23. Weathertop Consulting, LLC Summary Server-side analysis is critical for the future LAS. Server-side analysis is critical for the future LAS. NetCDF Java 2.2 and the THREDDS Data Server is a great platform for implementing this type of analysis with a legacy analysis application (like Ferret). NetCDF Java 2.2 and the THREDDS Data Server is a great platform for implementing this type of analysis with a legacy analysis application (like Ferret). A community-developed server-side analysis framework would make it easier to get the advantages of server-side analysis from other servers. A community-developed server-side analysis framework would make it easier to get the advantages of server-side analysis from other servers.

24. Weathertop Consulting, LLC Acknowledgments From COLA From COLA Jennifer Adams Jennifer Adams Brian Doty Brian Doty Joe Wielgosz Joe Wielgosz From Unidata From Unidata John Caron John Caron Ethan Davis Ethan Davis Former TMAP Former TMAP Richard Rogers Richard Rogers Yonghua Wei Yonghua Wei