Advances with the DDS David J. S. Poulter, British Oceanographic Data Centre, National Oceanography Centre, UK

Outline of this talk This talk is split into two sections Introduction to the HR-DDS (for those of you who don’t use it) What is the HR-DDS Time series analysis Spatial inter-comparison Anomaly analysis Data access and more New features of the DDS: Software changes Global DDS (G-DDS) / GMPE integration Please see me later if you have any questions.

What is the HR-DDS? Not all of you will have used the HR-DDS, so here is a very brief introduction: The HR-DDS is an interactive, web-based analysis system which allows you to perform a quick but detailed inter-comparison of GHRRST products, at approximately 250 globally distributed sites (which are shown on the map below).

What is the HR-DDS? At each of these sites, we produce a common format netCDF3 file for each input file, where there are valid SSTs. These files are called HR-DDS granules. For every granule, we store a representation of the data in a fast relational database. There are now nearly a billion individual statistics in this database. This data base can then be used to produce interactive diagnostic plots of the data.

Time series analysis Time series analysis can reveal anomalies in data, for example when a erratic buoy measurement creates a cold bias in and analysis product:

Time series analysis These plots are configurable, interactively: Apply/ignore bias estimates before plotting Display/hide error bar plotting Time range of data (default 2 weeks – can go up-to 5 years) Activate/deactive in-situ observing systems Select which parameter to plot (SST, wind speed, sea ice fraction, wave height, etc.) Select which statistic to plot (mean, median, stdev, kurtosis, etc.) Select max / min range to plot (or select ‘auto’ for automatic range selection) Minimum granule SST area coverage required for inclusion). Users may also select to plot a X-Y plot instead of a TIME-Y plot

Quick look imagery Clicking on any plot in the plot opens a quick look image of the data in question, to observe any possible cause of errors. From here users may select to view this data against all the other images from that site that day.

Spatial inter-comparison and analysis Individual observations may be interactively compared to identify any problems with that file.

Spatial inter-comparison and analysis Trends and such like are easily established.

New software aspects RPyC Process Server: The existing HR-DDS system had a simple database and web server configuration. The data base on one machine and the web server on another. Now every process call made goes through a process server which can distribute the tasks onto difference machines over a network. RPyC M1 M2 Request

New software aspects Memcache Server: Each process started by the RPyC server stores its result in a Memcache repository. This is the fastest available storage medium on modern computers. The the process then returns a key for that data to the RPyC server. Memcache can also be distributed across networks. Individual methods can be cached Memcache Data Memcache M1 RPyC Key

New software aspects Proxy Server: The RPyC and Memcache systems can exist on separate mechiness behind a corporate Firewall That way none of the data OR code exists in the machine open to the internet. Static files can be serverd through any web server, whilst dynamic requests are passed on.

New software aspects Django Server: A Django application framework server now makes the requests to the RPyC server. Django was develped on news rooms to help the delivery of dyanmic web content (Washington Post for example). Features advances web based configurations, authentications, registrations etc.

New software aspects NetCDF4: All internal files and products are stored in netCDF4. netCDF4 allows for much faster read times (order of magnitude) and significantly better compression than the existing bz2 netCDF3 systems. Example: A UK Met Office Wave Forecast run in bz2 netCDF3: Uncompressed size: 730MB Compressed size: 86.8MB Time to read: 102 seconds Time to read 1 var: 30 seconds And in netCDF4: Compressed size: 68 MB Time to read: 37 seconds Time to read 1 var: 2 seconds

Pixies. Automatic reporting has been prototyped (Type 3 Pixies) in the GlobWave project. These are automatically produced reports with detailed analysis. We automatically produce 30 page reports for the Met Office detailing their wave forecast performance over the last month. We will prototype simple SST reports her in the IC-TAG

G(MPE)DDS.

G(MPE)DDS.

G(MPE)DDS.

G(MPE)DDS.

G(MPE)DDS.

G(MPE)DDS.

G(MPE)DDS.

G(MPE)DDS.

G(MPE)DDS.

G(MPE)DDS.

G(MPE)DDS.

Others (talk to me!) Google Earth ™ for every global map produced. Dynamic regions (for example, wihin 100km of ice, or where DV has been mapped) Rgeional DDS (Wasparc or SABIA for Met.No and DMI) NetCDF4 to NetCDF3 on the fly The DDS as an installable executable for easy porting