1 Winter 2009 ESIP Federation Meeting NOAA Data Users Workshop Data Stewardship at the NOAA Data Centers Data Stewardship at the NOAA Data Centers

2 Scientific Data Stewardship (SDS) Value Added Products –Putting all our data sources together, to understand today’s environment in the context of the past Data Quality Monitoring for Long Term Applications –Critically examining data quality from our unique, integrated, and long-term perspective Long Term Preservation of Data and Information –Ensuring information survives and remains understandable to our children and beyond Data Archaeology and Rescue –Finding, saving, ingesting, and restoring data from all over the world Data and Information Access - Tomorrow’s Session –Providing information to users and ensuring the feedback loop contributes to its continual improvement Increasing Level of Stewardship

3 Scientific Data Stewardship (SDS) Your input is critical! Increasing Level of Stewardship

4 Winter 2009 ESIP Federation Meeting NOAA Data Users Workshop Data Stewardship at the NOAA Data Centers Data Archaeology and Rescue

5 Data Archaeology and Rescue Data centers routinely seek to acquire and preserve information that is in danger of being lost forever, from causes as diverse as deteriorating storage media to retiring principal investigators who never submitted their data to an archive. Participants are encouraged to discuss with the Data Centers their needs, areas of focus, and new technologies to support these rescue efforts. What are the key gaps the Data Centers should focus on?

6 Global Ocean Data Archaeology and Rescue (GODAR) Project GODAR – begun in 1993 with endorsement from the Intergovernmental Oceanographic Commission (IOC) Goals –Increase the volume of historical oceanographic data available –Locate and digitize ocean profile and plankton data sets not yet in digital form –Rescue data on electronic media that are at risk of loss due to media degradation –Submit these data to a national data center Accomplishments –> 3.5 million temperature profiles, >100,000 chlorophyll profiles, and >600,000 plankton observations added to data center holdings –Increased awareness of at-risk data collections –World Ocean Database project established to stimulate international exchange of modern oceanographic data

7 Climate Data Modernization Program CDMP - a multi-year partnership begun in 2000 between NCDC and private industry Goals –scan, digitize, preserve and provide access to diverse types of climate data and information. Accomplishments –>52 million weather and environmental records digitized Digital capture (images and data) of 19 th century surface weather observations Environmental proxy records, e.g., Common murre nesting site images and notes Digitize 7,200 negatives of killer, minke whales and other mammals, scan 15,000 pages of notes, keypunch 3,000,000 characters

8 Over 10 terabytes of Climate data now only a mouse click away Fifty two million weather & environmental images online Hundreds of million of records digitized & now online International data access and rescue activities 1842 Hourly Weather Data from Washington, DC Imaged and Digitized through the CDMP Program North Carolina Cooperative Observer Images -Online Missing images were made available in 2007 African Records Rescue Project Climate Data Modernization Program

9 03/13/20069 Forecast Warning Analysis NWS Coop Observers Global Synoptic Reports NCEP Charts Ship, Buoy Reports Rocketsonde Radiosonde Storm Data Doppler Radar (GOES, POES, NPOESS, many other) Satellites Aircraft Profiler ASOS USCRN Data Received From Many Sources

10 Geosat Tape Rescue From March 1985 until January 1990, the U.S. Navy satellite Geosat generated a new data set with unprecedented spatial and temporal coverage of the global oceans –In the first 18 months, Geosat’s classified Geodetic Mission (GM) succeeded in mapping the marine geoid –From November 1986 until December 1989, Geosat conducted its Exact Repeat Mission (ERM) to collect observations of sea level, wave height, and wind speed Goals –Recover original Sensor Data Records (SDRs) from deteriorating tape media before data were lost Accomplishments –Recovered original Sensor Data Records (SDRs) from the GM –NODC partnering with NOAA’s Lab for Satellite Altimetry and a tape recovery specialist to recover original SDRs from the ERM from deteriorating tape media To date, 670 of track tapes have been rescued An amazingly small loss-rate has been achieved - only 95 seconds of data from 670 days of data

11 Geosat Tape Rescue An amazingly small loss-rate has been achieved - only 95 seconds of data from 670 days of data!!!! Example tape damage

12 Metadata Archaeology Many historic data collections are poorly documented, which limits the usefulness of the data –Data formats, codes used –Methods, instruments, calibrations, precisions Challenge is to discover, document, and connect robust, relevant metadata in gray literature, published works, or PI manuscripts and notes to existing observation data

13 Discussion Questions What are the priority types of data in need of archaeological recovery, rescue, and preservation? –Physical, chemical, biological variables in need of data rescue –Other types of observations in need of rescue Still images Video transects and time series observations Model outputs, source code or other elements What resources should data centers provide or obtain to develop better data rescue partnerships? Other questions…