Recent developments with the THREDDS Data Server (TDS) and related Tools: covering TDS, NCML, WCS, forecast aggregation and not including stuff covered in GALEON results or discussion of access control John Caron UCAR/Unidata

A few Good Things to know about the THREDDS Data Server John Caron UCAR/Unidata

THREDDS Data Server 100% Java Runs within Tomcat Data Server, not a Data Portal –no pictures –html interface Production status –Early adopters - Cutting but not bleeding edge –Starting to look at performance issues

TDS – 5 things to know 1.Reads CDM files 2.Serves OPeNDAP, WCS, NetCDF 3.Configured with THREDDS Catalogs 4.Integrated with NcML Augment metadata Virtual Datasets / Aggregation 5.Connects to Digital Libraries through OAI

HTTP Tomcat Server THREDDS Data Server Datasets Catalog.xml otherhost.gov THREDDS Server Application NetCDF-Java (CDM) library OPeNDAP HTTPServer WCS OPeNDAP Server hostname.edu OAI Harvester DL Records OAI Provider

1. Reads CDM files NetCDF-3 HDF5 (most)HDF5 OPeNDAP client/server protocolOPeNDAP GRIB version 1 and version 2 (ongoing work)version 1version 2 BUFR (limited - ongoing work)BUFR GINI (GOES Ingest and NOAAPORT Interface ) image formatGINI DMSP (Defense Meteorological Satellite Program)DMSP NEXRAD Radar level 2 and level 3.NEXRADlevel 2level 3. ATD's DORADE radar file formatDORADEformat ADDE/McIDAS client/server protocol (limited)ADDE/McIDAS

Coordinate Systems Application Scientific Datatypes CDM Architecture OPeNDAP THREDDS Catalog.xml NetCDF-3 HDF5 I/O service provider GRIB GINI BUFR Data Access NetCDF-4 … NEXRAD DSMP CoordSystem Builder Datatype Adapter ADDE

2. Serves OPeNDAP, WCS, NetCDF OPeNDAP –remote data access to CDM datasets WCS – gridded data only –CDM must understand geo coord system –Data transferred as GeoTIFF, netCDF-CF –Serve raw data – no re-projections NetCDF subsetting service (experimental ) –request lat/lon/time bounding box –return netCDF-CF files HTTP bulk file transfer

3. Client sees THREDDS Catalogs Plain old XML documents - no search or other capabilities Logical directory tree of on-line resources Place to add metadata

<datasetScan name=“NEXRAD” serviceName=“dods” path=“radars" location=“/data/radar/” /> 3. TDS Configured with Catalogs

4. Integrated with NcML <datasetScan name=“models” serviceName=“dods” path=“models/ocean" location=“/data/models/” />

4. Integrated with NcML

5. Connects to Digital Library through OAI Enhanced Catalogs – enough metadata to create Digital Library Records Crawl catalog and create records –Metadata is in the public catalogs – so anyone can harvest Open Archives Initiative (OAI) harvesting protocol (DLESE) DIF (GCMD) and ADN (DLESE) records –Seems like ISO worth doing also

What’s Next? DataType Aggregation Automatic extraction of metadata from datasets LAS, OPeNDAP, CDP, NOMADS, NGDC Spring(?)

DataType Aggregation Work at the CDM DataType level, know (some) data semantics Forecast Model Collections –Combine multiple model forecasts into single dataset with two time dimensions –With NOAA/IOOS (Steve Hankin) Point/Station/Trajectory/Profile Data –Allow space/time queries, return nested sequences –Start from / standardize “Dapper conventions”

Forecast Model Collections

TDS/CDM training workshop Advanced uses of netCDF and TDS Oct 30 – Nov 2, 2006 –NetCDF C library (1 day) –NetCDF Java library (1 day) –TDS (2 days) Boulder, Colorado Free, limited space Video will be available

TDS Access Control <dataset name=“GOES West 4km Visible" urlPath="satellite/WEST-CONUS_4km“ resourceControl=“groupName” > Don’t know how to trigger authorization challenge from servlet

TDS resource control Track user’s resource consumption –Session cookies –Track by IP Customer categories –Preferred: top priority –Registered: we know who you are –Normal : IP only, use spare cycles, with limits –Abusers : ignore, penalty box, etc.

Data Access Standards 1.Data Subsetting –What queries are allowed? –What queries can be answered efficiently? 2.Data Encoding –What does the result of the query look like? On-the-wire Through the API (language specific)

Standardization 1.Start by designing an abstract model. –OGC/ISO, GML. –Comprehensive –Danger: Premature standardization, wrong tradeoffs 2.Start from current usage –CDM : netCDF, HDF5, OPeNDAP; CF-1.0 –20% covers 80% of what’s needed –Danger: ad-hoc, fragmentation