1. Practical techniques for Distributed Climate Analysis using GrADS and the GDS
Jennifer Adams, Joe Wielgosz, Brian Doty, and Jim Kinter
Center for Ocean-Land-Atmosphere Studies
(COLA)
AMS Annual Meeting 10 January 2005
We're usually preaching to the choir in the IIPS sessions -- developers talking to each other about all the cool stuff our software can do … This year I submitted my abstract to a joint session hoping to reach a wider audience that includes not only the developers but also the users of the tools we work so hard to refine. I'd like to thank my session chairpersons for giving me such an excellent time slot for accomplishing this goal. Before I get started talking about all the cool stuff our software can do, I'd like to take a moment to acknowledge my colleagues and co-authors -- Joe Wielgosz, Brian Doty,and Jim Kinter.

2. Outline Brief description of GrADS and the GDS
New features in upcoming releases
Metadata search engine
Examples of analysis techniques
Because there may be a few in the audience who don't know much about GrADS or the GrADS-Data Server, I’ll take a moment to describe them, and then I'll touch on some of the new features that will be included in the next releases of both packages. Next on my agenda is an unadvertised topic -- I want to tell you about a metadata search engine we've set up at COLA that has been the motivation behind most of the new development work we've done lately. Finally, I’ll get to the advertised ‘practical techniques’ and show you how easy it is to do climate analyses with GDS data sets.

3. GrADS is …
Freely available software for the display and analysis of a variety of data types and formats
Gridded and Station (in situ) data
Binary, GRIB , BUFR, NetCDF, HDF-SDS
DODS / OPeNDAP - enabled for accessing   distributed data
Handles a large variety of data formats, for both gridded and in situ data.
GrADS may also be used as a client to access a variety of DODS/OPeNDAP servers, which distribute data over the internet.

4. GrADS Data Server is …
Freely available software that provides access to scientific data for the internet community
Open a URL instead of a local disk file
Data are accessible by a variety of clients: GrADS, Ferret, IDV, IDL, Matlab, ncBrowse, et al.
Any GrADS-readable data may be served
Metadata and data are presented in unified framework
Server-side analysis capability Users create data sets that don't yet exist
GDS is one flavor of DODS/OPenDAP server that has a lot of unique features:
The underlying principle is that you open a URL instead of a local disk file
GDS data are accessible by many other clients besides GrADS -- here are some, please forgive me if I've left out your favorite
Any GrADS data set may be served -- all those formats I mentioned previously -- but none of that is apparent to the users
Formats and data types behind the server are transformed into a unifying NetCDF framework.
Best of all, the GDS allows server-side analysis, whereby users can create data sets that don’t yet exist.
This is the feature I'm going to show you how to exploit later on in my talk.

5. Outline Brief description of GrADS and the GDS
New features in upcoming releases
Metadata Handling
Metadata search engine
Examples of analysis techniques
Let's move on to what's new in these two packages -- Well, there's a lot that's new, but what I'm going to focus on today is the handling of metadata.
Metadata is data about your data: grid dimensions, variable name, units, model version number… anything that provides extra information about numerical data is metadata. Without good metadata, your data is essentially useless.
We've been working on ways for users to access and supplement their data's metadata and strategies for the GDS to capture that metadata as pass it on as it serves data on the internet.

6. What’s New In GrADS 1.9b4
Additional metadata may be added to the GrADS descriptor file as attribute u String units v String units lat Float32 minimum lat Float32 maximum global String comment model version 3.2.3
Attribute comments may appear anywhere in the descriptor file
The 'query attr' command returns all file attributes, distinguishing between those in the descriptor file and those native to the data file (e.g. NetCDF)CAT
In GrADS, metadata can be added to the GrADS descriptor file as attribute comments.
These comments begin with sign, and contain variable name (or global when appropriate), attribute type, attribute name, and attribute value.
These attribute comments may appear anywhere in the descriptor file, and are retreived at the command line with the 'q attr' command.

7. What’s New in GDS 1.3 THREDDS 1.0 catalog Improved metadata handling
Improved metadata handling
Captures all descriptor file attributes
Allows attribute override
Java-DODS 1.1.7
GDS captures descriptor file attributes and passes them through, but it also allows administrator override of some attributes for additional control.
Joe has also added the latest versoin of the THREDDS catalog, and upgraded the java-dods code as well.

8. Outline Brief description of GrADS and the GDS
New features in upcoming releases
Metadata search engine
How it has been implemented at COLA
Examples of analysis techniques
Now we move on to the metadata search engine we've set up at COLA that is the motivation for and application of
All the metadata enhancements we've put into GrADS and the GDs.

9. Basic Components of COLA's Metadata Search Engine
Suite of disk servers -- total disk volume > 15 Tb
Each disk server is running GDS
Disk crawler does nightly search on every disk
Disk crawler finds all viable GrADS descriptor files
Disk crawler writes an XML configuration file for the GDS
Result: Suite of GDSs describes all COLA data sets
Jakarta Lucene search engine indexes each GDS in suite
Search engine uses THREDDS generate the catalog and OPeNDAP to populate all metadata fields
Off-site GDSs are also indexed
Users search using a browser interface or unix command line
Text and numerical searches
Result: A searchable metadata catalog for all of COLA's data
COLA is probably a lot like all the other research shops out there in that we have an assortment of enormous disk servers of varying ages and sizes that get filled up almost as quickly as they are allocated. But there isn't much cohesive organization to the data, and it can be be hard to find stuff you haven't looked at lately or something your colleagues generated that you want to use. This is what we've set up to try to fix that problem.
<read and explain>
Next I'll show you a few screen shots to illustrate how the search engine works.

10. Here's the search home page -- when I took this screenshot we were serving over 28,000 data sets.
You can do a basic text search with the same syntax you might use on Google or Ebay
But you can also search on data set owner, disk path, data set attributes, even numerical searching on geographical location or temporal coverage.

11. Here's the results of a simple text search on 'reanalysis' -- a familiar layout that includes data set name, title, and the commands to open the data set in GrADS using GDS or the local disk file
The 'reanalysis' search generated 743 hits --
The links for data set name take you to the info page on the GDS

12. Here's all the metadata for a particular data set -- grid size and domain, variable names, global attributes, and off screen are the long list of variable attributes. The 'modified' and 'owner' attributes were added by the disk crawler when it found this file on disk.

13. Outline Brief description of GrADS and the GDS
New features in upcoming releases
Metadata search engine
Examples of analysis techniques
Spatial Averaging
Temporal Averaging & Anomaly Calculation
Intercomparison of data types
Working with Ensembles

14. Spatial Averaging
Calculate a 50-year time series of monthly global mean 2-meter temperature using NCEP/NCAR Reanalysis data

15. Script Example server = 'http://cola8.iges.org:9191/dods/'
dset = 'rean_2d'
lon = '0:0'
lat = '0:0' lev = '0:0'
time = 'jan1950:jan2000'
expr = ' tloop ( aave ( t2m, global ) ) ' 'sdfopen 'server'_expr_{'dset'} {'expr'} {'lon', 'lat', 'lev', 'time'}'
'display result.1'
Here is a plot showing SST observations from a randomly-selected data set on the EPIC DODS Server. They are color-coded according to sea surface temperature. You can see how nicely the ship tracks are illustrated.
All I did to draw this plot was the following:

16. Global Mean 2m Temperature

17. Temporal Averaging & Anomaly Calculation
Calculate the annual cycle for our 50-year time period
Remove the annual cycle to see the anomaly time series
Here is a plot showing SST observations from a randomly-selected data set on the EPIC DODS Server. They are color-coded according to sea surface temperature. You can see how nicely the ship tracks are illustrated.
All I did to draw this plot was the following:

18. Script Example server = 'http://cola8.iges.org:9191/dods/'
dset = 'rean_2d'
lon = '0:0'
lat = '0:0' lev = '0:0'
time = 'jan1950 : jan2000'
expr = ' tloop ( aave ( t2m, global ) ) ' 'sdfopen 'server'_expr_{'dset'} {'expr'} {'lon', 'lat', 'lev', 'time'}'
'display result.1'
cache = '_exprcache_ '
time = 'jan1950 : dec1950'
expr = ' tloop ( ave ( result.1, t+0, t=600, 12 ) ) '
'sdfopen 'server'_expr_{'cache'} {'expr'} {'lon', 'lat', 'lev', 'time'}'
'set time jan1950 dec1950'
'define annualcycle = result.2'
'modify annualcycle seasonal'
'display result.1-annualcycle'

19. Global Mean 2m Temperature Anomaly
Time series which is much easier to interpret.

20. Synthesis of In-Situ and Gridded Data
GOES Sounder measurements of surface temperature
GOES swath data read into GrADS as station data
1-hr RUC forecast of 2-meter temperature
GrADS gr2stn function applied to model grid
GrADS oacres function applied to differences at pixel locations

21. Script Example
server1 = ' server2 = '
'open 'server1'/stn/goeseast/sndr/conus/ '
'sdfopen 'server2'/NCDC_NOAAPort_RUC/ / /ruc2_236_ _2300_fff''
stn = 'ts.1' grid = 'tmp2m.2'
'display 'stn
'display 'grid
'display oacres ( 'grid', gr2stn ( 'grid', 'stn' ) - 'stn' )'

22. GOES Sounder Surface Temperature

23. GOES Sounder with RUC Forecast
Here is a plot showing SST observations from a randomly-selected data set on the EPIC DODS Server. They are color-coded according to sea surface temperature. You can see how nicely the ship tracks are illustrated.
All I did to draw this plot was the following:

24. Difference (RUC-GOES) at Pixel Locations Interpolated to 0
Difference (RUC-GOES) at Pixel Locations Interpolated to 0.5-degree Grid
Here is a plot showing SST observations from a randomly-selected data set on the EPIC DODS Server. They are color-coded according to sea surface temperature. You can see how nicely the ship tracks are illustrated.
All I did to draw this plot was the following:

25. NCEP Ensemble Spaghetti Plots
Conventional spaghetti plot illustrates how the forecast model ensemble members evolve and diverge throughout the forecast period
Animation shows the following:
500 mb Geopotential Height Analysis (just the 540 dm contour (RED)
500 mb Geopotential Height Forecast (just the 540 dm contour) from each of 10 ensemble members at 6-hr intervals for a 9-day forecast period (YELLOW )
Background shading of ensemble mean variance (GRAYSCALE)
Most of you have probably seen a spaghetti plot before … <read slide>

26. As the spaghetti strands begin to separate, that's when you might begin to lose confidence in the forecast …
The emerging clouds of variance also show where the ensembles begin to diverge -- note how the clouds propagate around the
Hemisphere with the synoptic features.

27. A New Spaghetti Recipe
A different kind of spaghetti plot illustrates how the forecast model ensemble members arrived at an agreement as lead time diminished.
Animation shows the following:
500 mb Geopotential Height Analysis (just the 540 dm contour (RED)
500 mb Geopotential Height Forecast (just the 540 dm contour) from each of 10 ensemble members at 6-hr intervals BUT each forecast is valid at the analysis time beginning with a 9-day forecast and ending with a 6-hr forecast (YELLOW )
Background shading of ensemble mean variance (GRAYSCALE)
Slices through for forecast ensembles in a different way … It's more of a hindcast or forecast in retrospect, where we're looking at the same ensemble spaghetti and the same analysis time, but instead of animating foreward over forecast hour, we animate backward over forecast lead time. We slice backward in time, looking at all the forecasts that were valid at the target analysis time Starting with a 9-day forecast and ending with a 6-hr forecast

28. Here we start with tangled spaghetti and as the lead time diminishes you'll see the ensembles come together and the variance clouds dissipate. One advantage to this perspective is that the underlying height field doesn't evolve through the animation, which makes it a little easier to isolate the ensemble variance.

29. Summary
GrADS gives you a common interface for gathering and analyzing all kinds of gridded or station data
GrADS Data Server (GDS) provides a unified NetCDF framework for serving a large variety of data sets to the community
Use these tools to enjoy data access and analysis without data management
Open a URL and start working!
Data providers can use GDS to serve almost any kind of data, with the formatting and data types completely invisible to the users. It all appears to the world in a unified NetCDF framework.
Using GrADS as a client, the users have a common interface for gathering and analyzing all kinds of gridded or station data sets from a variety of remote servers. Just open a URL and start displaying data!