1. The Server-Side with F-TDS
Roland Schweitzer
Ansley Manke, Kevin O’Brien, Karl Smith and Steve Hankin

2. The Ferret-THREDDS Data Server
CDM IOSP and plug-in to TDS that allows Ferret to act on data being served.
Uses delayed evaluation:
Transformation applied virtually over the entire domain
Computations done as requested and only over the required sub-domain

3. Delayed Evaluation
Server responds to a OPeNDAP URL in which the transformed variable and its
coordinate variable exists with no computations performed.
Client requests an area average of SST over all time steps.
Global 2-degree
with 12 time steps.
Grid { ARRAY: Float32 SST[TIME = 12]
[COADSY = 90]
[COADSX = 180]; MAPS: Float64 TIME[TIME = 12]; Float64 COADSY[COADSY = 90]; Float64 COADSX[COADSX = 180]; } SST;
Float32 sst_1_transformed[TIME = 12];

4. Delayed Evaluation Continued
F-TDS computes the area average for the first three time steps.
Client receives the requested values and matching coordinates.
Client requests value of the first three months.

5. The Ferret-THREDDS Data Server
Two methods of use:
Custom URLs
Ferret scripts on the server

6. Custom URL
We just saw an example of a client request using a special URL syntax to request a computation.
Sadly, it looks like this:

7. Custom URL syntax http://server.gov/thredds/dodsC/ (server)
coads.nc (untransformed data set)
_expr_{} (magic spell incantation)
{letdeq1 sst_1_transformed=
_cr_
ATTRCMD sst sst_1_transformed} (Ferret)

8. More about that URL First hard lesson learned:
Cramming Ferret syntax into a URL is a near disaster.
Syntax elements like the “/” in let/d=1, and the “;” to separate lines are also significant to HTTP clients and servers.

9. So what to do?
These difficulties make it hard for people to spontaneously define transformations to be performed on the server while working in a client.
Teach the client to assist...

10. LAS
We’ve already seen a snippet of one client (the web-based Live Access Server) which offers a rich UI for defining transformations.

11. let/remote
Our analysis client can build the custom URL for you if indicate that the OPeNDAP server is F-TDS ready.

12. upper ocean average temperatureZ1 1 ∫
let Tave =
Temp dz Z ΔZ
use “
let/remote TaveZ =
fill/T=1-jan TaveZ
plot TaveZ[X=180,Y=0]

13. average eddy kinetic energy
let Ek
u2 + v2 = T2 1 ∫
let Ekave =
Ek dt T ΔT T2
use “
let/remote Ek = u^2 + v^2
let/remote EkaveT =
fill EkaveT[Z=0]

14. Custom scripts on the server
Any Ferret command script that defines new variables from an existing data source can be used to create a virtual data set.
Variables are evaluated in delayed-mode.
Instantly provide access to many types of analysis and transformations of existing variables.

15. Custom scripts on the server
Removes the need for the client to assist.
User sees the original data set, plus the transformed virtual variables.
Scripting conventions determine which variables in the calculation are hidden and which variables appear in the final data set.

16. A success story ROMS - Regional Ocean Modeling System
The challenge, take the curvilinear, rotated and staggered grid for U and V from the model and serve them via OPeNDAP with it without the rotation and stagger.

17. The UV Model Grid

18. Vectors from the original grid
Don’t make much sense…

19. The Final Product
U and V on a single grid without the rotation

20. The original variables

21. Plus the new rotated U and V

22. Applying more transformations

23. Discrete Sampling Geometry Files
Difficult to work with over OPeNDAP due to the large number of network requests.
What if we could make “delayed-mode” virtual sub-sets?

24. Discrete Sampling Geometry File
F-TDS has extended the concept of a virtual variables to CF-1.7 Discrete Sampling Geometry (DSG) files.
Clients request sub-sets via projection style queries and get back fully formed DSG virtual data set.

25. Example: full dataset
> ncdump –h
netcdf DrifterBuoyData_1_5000 {
dimensions:
TRAJ = 5000 ;
OBS = ;
variables:
float DLAT(TRAJ) ;
DLAT:long_name = "Deployment latitude" ;
float DLON(TRAJ) ;
DLON:long_name = "Deployment longitude" ;
double TIME(OBS) ;
TIME:units = "seconds since 1-jan-1970" ;
float LAT(OBS) ;
LAT:long_name = "Latitude" ;
float LON(OBS) ;
LON:long_name = "Longitude" ;
float TEMP(OBS) ;
TEMP:long_name = "Temperature" ;

26. Make new virtual data set
baseURL + _expr_{}{constraint}{}
constraint is a Ferret expression
e.g. let mask = lat GT 0 and lat LT 20

27. Subset dataset
> ncdump -h
netcdf Drifting_buoys.nc_cr_let\%20mask\%20\=\%20if\%20\%20dlat\%20GT\%200\%20and\%20dlat\%20LT\%2010\%20then\%201\%7D\%7Bgo\%20dg_subset {
dimensions:
TRAJ = 1042;
OBS = ;
variables:
float DLAT(TRAJ) ;
DLAT:long_name = "Deployment latitude" ;
float DLON(TRAJ) ;
DLON:long_name = "Deployment longitude" ;
double TIME(OBS) ;
TIME:units = "seconds since 1-jan-1970" ;
float LAT(OBS) ;
LAT:long_name = "Latitude" ;
float LON(OBS) ;
LON:long_name = "Longitude" ;
float TEMP(OBS) ;
TEMP:long_name = "Temperature" ;

28. Using the full dataset > ferret
yes? use "
yes? go basemap
yes? plot/vs/overlay dlon, dlat

29. Using the subset dataset
> ferret
yes? use "
yes? go basemap
yes? plot/vs/overlay dlon, dlat

30. F-TDS & the OPeNDAP Community
F-TDS is a powerful tool.
F-TDS is not widely used.
Users need help from the client.
Complex transformation are possible with specialized knowledge and best done on the server by data providers.

31. F-TDS & the OPeNDAP Community
Installation, integration, configuration and maintenance are just difficult enough to keep people from using the software.
What to do about it?

32. First Attempt...

33. Server-side functions WG

34. Happy to announce the results…

35. Second attempt… Proposed a set of core transformations:
averaging of time, area, depth, and volume
computing variance, extrema, integrals and counts of valid/invalid points
linear combinations (difference variables, convert units)
re-gridding to arbitrary lat/lon rectangular girds

36. Also proposed
Simplified direct implementation to make it deployable without Ferret
General syntax (not Ferret script-based)

37. Community Effort No funding was awarded
Any progress will be a community effort

38. More Information Roland.Schweitzer@noaa.gov
(search F-TDS)