U.S. Department of the Interior U.S. Geological Survey Management of Oceanographic time-series data at the Woods Hole Coastal and Marine Science Center Ellyn Montgomery USGS NERACOOS/NECOSP Data Management Workshop, Sept. 26, 2012

NECODP 2012  This presentation applies only to Oceanographic measurements acquired by Woods Hole researchers and colleagues to address scientific questions  The data is mostly used for validation of sediment transport and coastal process models  Management of the data is part of the larger Data Life-cycle that includes everything from planning to archival and publication USGS Coastal and Marine Geology Program

NECODP 2012  Our holdings describe conditions at specific locations for limited durations. Except for a 25+ year program in Massachusetts Bay, we don’t do long term monitoring programs.  Once the data files collected during an experiment are processed and QC’d that data is published (Federal requirement)  Updates are made only if a problem is found in the data to correct that problem About our data

NECODP 2012  The data are stored in netCDF format files using EPIC conventions  NcML is used to convert to CF conventions  Files are delivered on-line via  file download  OPeNDAP using THREDDS  FGDC metadata for each experiment is being created and harvested to ocean.data.gov  Each file has ISO metadata (ncISO) About our data II

NECODP 2012  Our raw data is included in the WHSC field activity tracking system  We store 3 levels of data on local disks that are backed up nightly, with periodic backups made and stored off-site. These levels are :  Raw data as off-loaded from the instruments  Processed data (and stages leading to it)  Published data  As EPIC  As CF How we manage our data - 1

NECODP 2012  Our data may be published directly to the website after approval or with an Open File Report that documents the context of the measurements.  We periodically check the website for broken links or non-responsiveness  We add new interfaces as available (for example making KML files to use with GoogleEarth) Publication and access

NECODP 2012 Web access

NECODP 2012 Web access

NECODP 2012 THREDDS

NECODP 2012  NetCDF allows metadata to be stored as attributes with the data in the file  We use standards and conventions to enable interoperability  We have GCMD DIF, FGDC and ISO19115 metadata in as many catalogs and registries as possible  We enhance the metadata as new standards emerge (added Unidata Dataset Discovery terms in 2011) Metadata

NECODP 2012  Community Standards development has evolved to enable LOT of great functionality  We are happy with how THREDDS allows remote clients to access data for analysis in real-time, without having to down-load files  Clients like Matlab, IDV, ncBrowse and ncView allow access and visualization  Registering our THREDDS catalogs in larger repositories of ocean measurements enhances discoverability Data Management Successes

NECODP 2012  Getting links to our holdings into the catalogs of the regional observatories  Since our data is historical, it hasn’t really fit the observatory mission, but including links to historical measurements in the area would be useful for enriching potential analyses  Identifying which of the emerging technologies will persist into the future  Finding ways to let potential users know the data is available Data Management Challenges

NECODP 2012  Use standards to enhance discoverability and enable use of community developed tools  Enable web-based access so the users can access what they need without assistance  Tailor the process to what your clients need to accomplish their missions  Adapt your data and services to take advantage of new techniques as they evolve Take home ideas

NECODP 2012  The human-friendly interface to our experimental data, (EPIC conventions) is at:  The computer-friendly (CF conventions) version of our data holdings, served by THREDDS at: alog.html alog.html  Contact me at:  (508) Web access