Experiences of a Earth Science Data User Confessions of a Data Hoarder Rob Carver, The Weather Company

–Andrew S. Tanenbaum “Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.”

Open Data and The Weather Company ❖ Our business model is taking open data and using it to tell interesting stories that engage our users. ❖ Over the years, we’ve archived over 100 Tb of data ❖ GRIB1, GRIB2, NIDS, shapefiles, netCDF, HDF5, ❖ NWS/NCEP, NCDC, FEMA, Census Bureau, NASA DAAC’s

Locating Data 1.Google and literature searches 2.??? 3.Data!

100+ Tb of Weather Models ❖ Most data arrives through Unidata’s LDM and FTP pull scripts. ECMWF pushes data to our FTP site. (All GRIB2/1) ❖ Ingested into the forecast system, and GRADS handles the model visualization ❖ Archived to local disk arrays and Amazon S3

Level-III NIDS Archive ❖ NCDC maintains an archive of the WSR-88D radar network’s products from 1995 to present (>10 Tb) ❖ Order datasets from a tape-based archive ❖ Two years to acquire it using a set of PHP scripts ❖ Easier to acquire the entire archive than figuring out what subset to acquire ❖ Already had a NIDS parser for visualization

FEMA Flood Maps ❖ Data Acquisition Method: DVD for each state ❖ Format: ESRI Shapefiles (1 shapefile of a feature class per state) ❖ Data Display: Split state shapefiles by county and then pre-render tiles for moderate to coarse zoom levels on a map mashup.

Suggestions ❖ Data in a difficult/proprietary format just waste disk space ❖ Please use data formats that are well-supported by open-source software packages (i.e. OGR/GDAL) ❖ netCDF, TIFF, ESRI shapefiles, HDF5, geoJSON ❖ Instead of complex CSV or fixed-width text files, use self-describing formats (JSON,XML,SQLITE)

Suggestions (cont.) ❖ Data/Navigation files should use the same naming conventions/sequences ❖ Don’t use overly large archive files ❖ Data pools/ftp servers attached to large disk arrays are awesome data providers (as long as limits are in place) ❖ For really large, static datasets (>10Gb), Bittorrent would be really useful

Questions/Comments/Answer s? ❖