The Server-Side with F-TDS Roland Schweitzer Ansley Manke, Kevin O’Brien, Karl Smith and Steve Hankin

The Ferret-THREDDS Data Server CDM IOSP and plug-in to TDS that allows Ferret to act on data being served. Uses delayed evaluation: Transformation applied virtually over the entire domain Computations done as requested and only over the required sub-domain

Delayed Evaluation Server responds to a OPeNDAP URL in which the transformed variable and its coordinate variable exists with no computations performed. Client requests an area average of SST over all time steps. Global 2-degree with 12 time steps. Grid { ARRAY: Float32 SST[TIME = 12] [COADSY = 90] [COADSX = 180]; MAPS: Float64 TIME[TIME = 12]; Float64 COADSY[COADSY = 90]; Float64 COADSX[COADSX = 180]; } SST; Float32 sst_1_transformed[TIME = 12];

Delayed Evaluation Continued F-TDS computes the area average for the first three time steps. Client receives the requested values and matching coordinates. Client requests value of the first three months.

The Ferret-THREDDS Data Server Two methods of use: Custom URLs Ferret scripts on the server

Custom URL We just saw an example of a client request using a special URL syntax to request a computation. Sadly, it looks like this: http://dunkel.pmel.noaa.gov:8920/thredds/dodsC/las/coads_climatology_cdf/data_coads_climatology.jnl_expr_%7B%7D%7Bletdeq1+sst_1_transformed%3Dsst%5Bd%3D1%2Cx%3D135%3A285%40Average%2Cy%3D-30%3A30%40Average%5D_cr_ATTRCMD+sst+sst_1_transformed%7D.html

Custom URL syntax http://server.gov/thredds/dodsC/ (server) coads.nc (untransformed data set) _expr_{} (magic spell incantation) {letdeq1 sst_1_transformed= sst[d=1,x=135:285@ave,y=-30:30@ave] _cr_ ATTRCMD sst sst_1_transformed} (Ferret)

More about that URL First hard lesson learned: Cramming Ferret syntax into a URL is a near disaster. Syntax elements like the “/” in let/d=1, and the “;” to separate lines are also significant to HTTP clients and servers.

So what to do? These difficulties make it hard for people to spontaneously define transformations to be performed on the server while working in a client. Teach the client to assist...

LAS We’ve already seen a snippet of one client (the web-based Live Access Server) which offers a rich UI for defining transformations.

let/remote Our analysis client can build the custom URL for you if indicate that the OPeNDAP server is F-TDS ready.

upper ocean average temperature Z1 1 ∫ let Tave = Temp dz Z ΔZ use “http://server/dataset.nc” let/remote TaveZ = Temp[z=0:`z1`@ave] fill/T=1-jan-2050 TaveZ plot TaveZ[X=180,Y=0]

average eddy kinetic energy let Ek u2 + v2 = T2 1 ∫ let Ekave = Ek dt T ΔT T2 use “http://server/dataset.nc” let/remote Ek = u^2 + v^2 let/remote EkaveT = Ek[T=`T1`:`T2`@ave] fill EkaveT[Z=0]

Custom scripts on the server Any Ferret command script that defines new variables from an existing data source can be used to create a virtual data set. Variables are evaluated in delayed-mode. Instantly provide access to many types of analysis and transformations of existing variables.

Custom scripts on the server Removes the need for the client to assist. User sees the original data set, plus the transformed virtual variables. Scripting conventions determine which variables in the calculation are hidden and which variables appear in the final data set.

A success story ROMS - Regional Ocean Modeling System The challenge, take the curvilinear, rotated and staggered grid for U and V from the model and serve them via OPeNDAP with it without the rotation and stagger.

The UV Model Grid

Vectors from the original grid Don’t make much sense…

The Final Product U and V on a single grid without the rotation

The original variables

Plus the new rotated U and V

Applying more transformations

Discrete Sampling Geometry Files Difficult to work with over OPeNDAP due to the large number of network requests. What if we could make “delayed-mode” virtual sub-sets?

Discrete Sampling Geometry File F-TDS has extended the concept of a virtual variables to CF-1.7 Discrete Sampling Geometry (DSG) files. Clients request sub-sets via projection style queries and get back fully formed DSG virtual data set.

Example: full dataset > ncdump –h http://SERVER/geoide/dodsC/drifter/DrifterBuoyData_1_5000.nc netcdf DrifterBuoyData_1_5000 { dimensions: TRAJ = 5000 ; OBS = 8332804 ; variables: float DLAT(TRAJ) ; DLAT:long_name = "Deployment latitude" ; float DLON(TRAJ) ; DLON:long_name = "Deployment longitude" ; double TIME(OBS) ; TIME:units = "seconds since 1-jan-1970" ; float LAT(OBS) ; LAT:long_name = "Latitude" ; float LON(OBS) ; LON:long_name = "Longitude" ; float TEMP(OBS) ; TEMP:long_name = "Temperature" ;

Make new virtual data set baseURL + _expr_{}{constraint}{} constraint is a Ferret expression e.g. let mask = lat GT 0 and lat LT 20

Subset dataset > ncdump -h http://dunkel.pmel.noaa.gov:8680/geoide/dodsC/las/DrifterBuoyData_1_5000.nc_expr_%7B/home/data/kobrien/Drifting_buoys.nc_cr_let%20mask%20=%20if%20%20dlat%20GT%200%20and%20dlat%20LT%2010%20then%201%7D%7Bgo%20dg_subset.jnl%7D netcdf Drifting_buoys.nc_cr_let\%20mask\%20\=\%20if\%20\%20dlat\%20GT\%200\%20and\%20dlat\%20LT\%2010\%20then\%201\%7D\%7Bgo\%20dg_subset { dimensions: TRAJ = 1042; OBS = 1481332 ; variables: float DLAT(TRAJ) ; DLAT:long_name = "Deployment latitude" ; float DLON(TRAJ) ; DLON:long_name = "Deployment longitude" ; double TIME(OBS) ; TIME:units = "seconds since 1-jan-1970" ; float LAT(OBS) ; LAT:long_name = "Latitude" ; float LON(OBS) ; LON:long_name = "Longitude" ; float TEMP(OBS) ; TEMP:long_name = "Temperature" ;

Using the full dataset > ferret yes? use "http://dunkel.pmel.noaa.gov:8680/geoide/dodsC/drifter/DrifterBuoyData_1_5000.nc" yes? go basemap yes? plot/vs/overlay dlon, dlat

Using the subset dataset > ferret yes? use "http://dunkel.pmel.noaa.gov:8680/geoide/dodsC/las/DrifterBuoyData_1_5000.nc_expr_%7B/home/data/kobrien/Drifting_buoys.nc_cr_let%20mask%20=%20if%20%20dlat%20GT%200%20and%20dlat%20LT%2010%20then%201%7D%7Bgo%20dg_subset.jnl%7D" yes? go basemap yes? plot/vs/overlay dlon, dlat

F-TDS & the OPeNDAP Community F-TDS is a powerful tool. F-TDS is not widely used. Users need help from the client. Complex transformation are possible with specialized knowledge and best done on the server by data providers.

F-TDS & the OPeNDAP Community Installation, integration, configuration and maintenance are just difficult enough to keep people from using the software. What to do about it?

First Attempt...

Server-side functions WG

Happy to announce the results…

Second attempt… Proposed a set of core transformations: averaging of time, area, depth, and volume computing variance, extrema, integrals and counts of valid/invalid points linear combinations (difference variables, convert units) re-gridding to arbitrary lat/lon rectangular girds

Also proposed Simplified direct implementation to make it deployable without Ferret General syntax (not Ferret script-based)

Community Effort No funding was awarded Any progress will be a community effort

More Information Roland.Schweitzer@noaa.gov http://ferret.pmel.noaa.gov/LAS (search F-TDS)