THREDDS Status John Caron Unidata 5/7/2013

Outline Release schedule Aggregations -> featureCollections / NCSS GRIB refactor Discrete Sampling Geometry (point data) ncstream/cdmRemote/cdmrFeature

THREDDS/CDM personnel John Caron (1.0) – head cook and bottle washer Ethan Davis (.25) – Architecture, standards, catalogs Marcos Hermida (1.0) – NCSS, WMS, maven, spring, javascript Lansing Madry (1.0) – support, testing, domain expertise Sean Arms (.25) – IDV/CDM interface, NCEP models, rosetta, python, domain expertise Dennis Heimbigner (0.5) – OpenDAP, HTTPClient Julius Chastang (0.0): – IDV, python interface Yuan Ho (0.0) – IDV, radar IOSPs

KillCat Release (1 June 2013) Last 4.3 Feature Release motherlode.ucar.edu -> thredds.ucar.edu All Unidata servers upgrade to Major Features – GRIB complete rewrite, FeatureCollection scaling – netCDF-4 writing with netCDF C library (JNA) – CF 1.6 Discrete Sampling Convention – NetCDF Subset Service and WMS improvements – Software Engineering: GitHub, Maven

BlackCat Release 4.4 (31 Oct 2013) Improvements for ESG – millions of catalogs Refactor/harmonize NCSS, CdmRemote, and RadarServer APIs Extend NCSS for use in WRF initialization Release THREDDS Data Manager (TDM) for outside use Migrate HttpClient from 3.x (EOL) to 4.x Possible – OpenID authentication – WaterML from NCSS Grid as Point (where does extra metadata come from?) – Experiments with Async

SantaCat Release 4.5 (25 Dec 2013) DAP4 server and client Grid Feature Collection, replace FMRC GRIB FeatureCollection: Constant Forecast Offset/Hour options CdmRemoteFeature service implemented for all CF-1.6 DSG feature types

SchrödingersCat Release 5.x Require Java 7 (nio2) and Tomcat 7 – Java 6 reached end of life Feb 2013 API changes allowed TDS configuration refactor Refactor GridDatatype to Coverage – Swath/Image – Cross-seam lat/lon data requests – Unstructured Grid? – Time-dependent coordinate system? – Better dataset classification Refactor Catalog reading/writing package Improved metadata harvesting support

SchrödingersCat Release 5.x Search/discovery service ? Asyncronous requests – client and server ? TDS-lite ? – on demand trusted local server – Access from C, python

Aggregation -> FeatureCollections Aggregation is associated with virtual datasets defined with NcML NcML is seriously overloaded with semantics – Originally a client-side configuration, done on-the-fly – Adapted to play seamlessly with TDS configuration catalogs – Server side aggregations more complicated – Hard to do all 3 at once: Very large, updating, performance – has new set of configuration elements that make it easier for both users and implementors Performance comes from storing the result dataset info (“.ncx” files) Aggregation is being phased out in favor of “feature collections” – GRIB, POINT collections are the first real implementation – GRID, FMRC will be refactored in 4.5 NcML can still be used to modify the datasets s/FeatureCollections.html s/FeatureCollections.html

NetCDF Subset Service REST web service for coordinate based subsetting on GRID datasets On our thredds.ucar.edu server, datasets are NCEP model runs as GRIB collections TDS 4.3: Much improved interface and reliability – needs more performance Can return netCDF-3 or netCDF-4 files Net effect is a subset / transformation service

GRIB – CDM/TDS 4.3 Complete rewrite of GRIB1, GRIB2 IOSPs – Table Handling – Multifile collections – eliminate user configuration Automatically figure out coord systems User Groups for multiple horizontal domains – Indexing (.gbx9) and cache metadata (.ncx) – User configuration passed into the IOSP TDS featureCollection=GRIB – Time Partitions (performance) – User Configuration for changing datasets – Motivated by NCDC/NOMADS issues and $

netCDF storage

GRIB storage

GRIB Rectilyzationologicment Turn unordered collection of 2D slices into 3-6D multidimensional array Each GRIB record (2D slice) is independent There is no overall schema to describe what its supposed to be  there is, but not able to be encoded in GRIB

GRIB collection indexing Index file name.gbx9 GRIB file … Index file name.gbx9 GRIB file Index file name.gbx9 GRIB file 1000x smaller Create TDS Collection Index collectionName.ncx 1000x smaller CDM metadata

GRIB time partitioning TDS gbx9 GRIB file … gbx9 GRIB file gbx9 GRIB file ncx gbx9 GRIB file … gbx9 GRIB file gbx9 GRIB file ncx … Partition index Collection.ncx

NCEP GFS half degree All data for one run in one file 3.65 Gbytes/run, 4 runs/day, 22 days Total 321 Gbytes, 88 files Partition by day (mostly for testing) Index files – Gbx9: 2.67 Mbytes each – Ncx: 240 Kbytes each – Daily partition indexes : 260K each – Overall index is about 50K (CDM metadata) – Index overhead = grib file sizes / 1000

CFSR timeseries data at NCDC Climate Forecast Series Reanalysis (31 years, 372 months) Total 5.6 Tbytes, 56K files analyze one month (198909) – 151 files, approx 15Gb. 15Mb gbx9 indexes. – 101 variables, time steps – records duplicates (15%) – 1.1M collection index, 60K needs to be read by TDS when opening.

Big Data cfsr-hpr-ts9 9 month (275~ day run) 4x / day at every 5 day intervals. run since 1982 to present! ~22 million files

GRIB - summary Fast indexing allows you to find the subsets that you want in under a second – Time partitioning should scale up as long as your data is time partitioned – No pixie dust: still have to read the data! – GRIB2 stores compressed horizontal slices, must decompress entire slice to get one value – Experimenting with storing in netcdf-4 – Chunk to get timeseries data at a single point Still getting the bugs out on changing/updating datasets (4.3.17) featureCollections will (eventually) solve many of the problems of Aggregations

Discrete Sampling Geometries (aka Point Data) Conventions added to CF 1.6 CDM 4.3 has complete implementation – ucar.nc2.ft.point package TDS 4.3 featureCollection – type = POINT, STATION – Creates a cdmrFeature web service

Discrete Sample Feature Types point: a collection of data points with no connection in time and space timeSeries: a series of data points at the same location, with varying time trajectory: a series of data points along a curve in time and space profile: a set of data points along a vertical line timeSeriesProfile: a series of profiles at the same location, with varying time trajectoryProfile: a set of profiles which originate from points along a trajectory

ucar.nc2.ft.point Subset by lat/lon box, time range Iterate over rows (result set) Not arrays (netCDF classic data model) Scales to large collections Allows streaming Similar to/compatible with RDBMS Nested tables – hierarchical data model TODO: arbitrary predicates (filter)

ncstream NetCDF files (almost always) have to be written, then copied to network – Assumes random access, not stream – “read optimized” : data layout is known ncstream explores what “streaming netcdf” might look like – “write-optimized”: append only – Efficient conversion to netCDF files on the client Ncstream data model == CDM data model Binary encoding using Google's Protobuf Protobuf – Binary object serialization, cross language, transport nuetral, extensible – Very fast: some tests show >10x OPeNDAP Have experimental versions in CDM and TDS since 4.1

cdmRemote Replacement for OPeNDAP 2.0 that can handle the full CDM data model In 4.3, CDM/TDS uses cdmRemote in preference to OPeNDAP, for remote access to CDM datasets – User can configure this Index based – just like netCDF Not currently promoting outside of CDM/TDS stack

cdmrFeature TDS 4.3 web service for coordinate based subsetting REST API Harmonize / merge with NCSS (4.4 Oct 2013) Extend to all CF-DSG feature types (4.5 Dec 2013) Intended to be used on collections of DSG (point) data – Needs time partition date in the filename Output – netCDF-3/CF – XML, CSV – ncstream fro CDM clients Clients – HTML form, like NCSS – CDM / ToolsUI – Python scripts

featureCollection configuration <featureCollection name="Metar Station Data" featureType="Station" path="urlpath/station/data">

Why not OPeNDAP? Reasonable data model using sequences Incomplete coordinate system data model – cant make requests in lat/lon or time in a standard way Server side processing not standardized – Client cant discover whats possible – Semantics non standardized Waiting for DAP4 – See what we have when that’s ready