1. NOAA Science Advisory Board’s Data Archiving and Access Requirements Working Group (DAARWG) May 24-25, 2007 Chicago, IL Tom Karl & Larry Tyminski Co-Chairs: NOAA Data Management Committee Overview of NOAA Archive Policy

2. Data Archive & Access Requirements Working Group Chicago, IL 2 May 24-25, 2007 Briefing Outline NOAA Archive Policy –Overview: NOAA Decision Processes What to archive and how long –NOAA Archive Policies Data Centers & Centers of Data Five NOAA Line Office Policies NOAA archive issues and what is expected in the future –Data versions: how many to keep? –Exponential data volume increase –CLASS data storage for all NOAA data?

3. Data Archive & Access Requirements Working Group Chicago, IL 3 May 24-25, 2007 NOAA Data to Archive Three Archive Considerations 1)Legal Requirements* Data required by law to be archived* 2)Practical Requirements** Data/products which require long processing times to reproduce** Non NOAA Observations that support NOAA’s mission** 3)Policy Requirements* Data for Regulatory purposes or used in Scientific Assessments* * NOAA responsibility **DAARWG recommendations (Note: Multiple versions of data/ products exist in all three areas, (especially 2) due to reanalyses, QC levels, etc. What versions to keep are open for DAARWG recommendations)

4. Data Archive & Access Requirements Working Group Chicago, IL 4 May 24-25, 2007 Which NOAA Data to Archive? Consideration 1: Legal Requirements NOAA Data Archive Legal Requirement Examples –Weather/Climate Original In-situ observations, weather forecasts, satellite, radar –Oceans Original Sub surface observations, Water levels and tides Ecosystem observations (Coral reefs) –Solid Earth Seismic, Volcanic, Magnetic, Bathymetric, Glacial, Nautical Charts, Shoreline –Space Solar, Terrestrial Physics U.S. National Archives & Records Administration (NARA) Policy - “…preserving records that contain adequate and proper documentation of the organization, functions, policies, decisions, procedures, and essential transactions of the agency…” NARA policy requires that a Records Disposition Schedule list the types of records maintained by each department of an organization, and specify a period of time records are to be retained for operational, legal, fiscal, historical, or other purposes.

5. Data Archive & Access Requirements Working Group Chicago, IL 5 May 24-25, 2007 Which NOAA Data to Archive? Consideration 2: Practical Requirements Certain data products require large processing times to reproduce –Model output: Products produced from original observations Examples: Weather and climate models, Climate Model based Reanalyses –Satellite: Products produced from geo-located observations Examples: Vegetation Index, blended sea winds, blended sea surface temperatures, nighttime lights, ocean color, etc. –Radar: Level-III NEXRAD products Examples: Base reflectivity, Echo tops, One-hour precipitation, etc. Non-NOAA Observations that support NOAA’s mission –Satellite: NASA, international –Other International data via WMO, World Data Centers, etc. NOAA Archive Practical Requirement Examples Multiple versions of data and products exist in this area

6. Data Archive & Access Requirements Working Group Chicago, IL 6 May 24-25, 2007 Which NOAA Data to Archive? Consideration 3: Policy Requirements Examples of Policy Made by Government Agencies using NOAA Data –Fisheries data used in policies on fish size and bag limits –US Army Corp Engineers use of revised hurricane statistics for rebuilding New Orleans’ levees –Climate Normals used by Utilities in setting power rates Data used in Scientific Assessment must be archived –U.S. Climate Change Science Program Assessments State of the Climate Reports –International International Panel on Climate Change (IPCC) Data for Regulatory Purposes and Reproduce Scientific Assessments CCSP 1.1 Assessment Released March 31, 2006

7. Data Archive & Access Requirements Working Group Chicago, IL 7 May 24-25, 2007 NOAA Archive Policies

8. Data Archive & Access Requirements Working Group Chicago, IL 8 May 24-25, 2007 NOAA Archive Structure Data Centers & Centers of Data NOAA Data Centers: Archive ~ 95% of NOAA’s data –NCDC, NODC, NGDC and CLASS (data storage) –Similar standards used for data/metadata –NARA policies are implemented NOAA Centers of Data: Archive ~ 5% of NOAA’s data –Many (dozens) NOAA offices are “official” Centers of Data (e.g., National Snow and Ice Data Center) –Almost all NOAA offices (100s) contain smaller data sets critical to their mission Exception: high volume archives of climate and weather models –“Some” standards for data / metadata –NARA policies may not be implemented (e.g., no back-ups, non standard formats and media)

9. Data Archive & Access Requirements Working Group Chicago, IL 9 May 24-25, 2007 NOAA Line Office Archive Policies NESDIS and NWS National Environmental Satellite Data & Information Service –Archive a core mission via 3 data centers: NCDC, NODC, NGDC –NESDIS data centers archive some data for all NOAA Line offices –CLASS plans: archival storage for all NOAA data starting with large array National Weather Service –NCDC performs almost all NWS archive functions for data which may be needed at a later time for accident investigations and litigation Original transmitted observations taken by NWS, FAA, or DOD Analysis and Forecast products officially transmitted by NWS In the event of litigation, NWS offices maintain misc. records (e.g. non transmitted records, station duty manuals); Offices also save weather cases for training –For large array data sets NCDC (stewardship) and/or CLASS (storage) provide archival services for radar observations (NEXRAD), some NCEP weather models, Satellites (GOES, POES)

10. Data Archive & Access Requirements Working Group Chicago, IL 10 May 24-25, 2007 NOAA Line Office Archive Policies NMFS, NOS, and NOAA Research National Marine Fisheries Service –Six NMFS Centers of Data manage small volume, high content data –NESDIS (NODC) provides archive and access for some NMFS data –Restricted data access due to privacy issues of data provider and regulatory nature of data National Ocean Service –Two NOS Centers of Data manage geodetic and water level data –Relies on many archived observations provided by other NOAA Line Offices –Manages data via NOS Centers of Data and NESDIS data centers (NODC, NGDC) NOAA Research –Maintains large volume modeling data bases at Centers of Data (NCEP,GFDL, ARL) –NCDC archives of some climate and weather model outputs –Trace Gas archives managed by DOE’s World Data Center for Trace Gases

11. Data Archive & Access Requirements Working Group Chicago, IL 11 May 24-25, 2007 NOAA archive issues and what is expected in the future

12. Data Archive & Access Requirements Working Group Chicago, IL 12 May 24-25, 2007 NOAA Archive Issues Example Data Set Evolution: NOAA’s Microwave Sounder Unit Issue: How many data set versions to keep? Example 1 –History of data set adjustments based upon scientific discoveries –Data set improvements occur when data are regularly used & more thoroughly scrutinized

13. Data Archive & Access Requirements Working Group Chicago, IL 13 May 24-25, 2007 NOAA Archive Issues Freedom of Information Act (FOIA) Requests: Retention of NOAA information used in scientific research or policy regulations Issue: How many data set versions to keep? Example 2 2007 FOIA Request Data requested from 1990 climate research on urbanization effects Data requested by international researcher

14. Data Archive & Access Requirements Working Group Chicago, IL 14 May 24-25, 2007 NOAA Archive Issues Broad Issues –Annual CLASS data storage budget is in the low million $ range Will the decreases expected in data storage cost keep pace with exponential increase in CLASS data volumes? –NOAA has some capability to provide data stewardship for existing data Will NOAA be able to provide data stewardship to produce new Climate Data Records from planned satellite missions? –GEO IDE to provide standards and protocols for interoperability is unfunded Will data access and inter-disciplinary analysis be feasible without GEO IDE? Exponential increase in NOAA data volume projected from systems such as NPP, NPOESS, and new model simulations NOAA’s Environmental Data Archive

15. Data Archive & Access Requirements Working Group Chicago, IL 15 May 24-25, 2007 NOAA Archive Issues NOAA has identified CLASS as the primary system to meet NOAA’s data storage requirements. CLASS Present: Provide data storage for large array data systems (currently GOES/POES Satellites, NODC smaller data sets as a prototype) CLASS Future: Provide storage for future large array data. Also, provide storage for NOAA’s smaller data sets (e.g., biological & physical). Specific language: “The current CLASS program supports the archiving of the specific and limited set of large array data sets identified above. Other major observing campaigns not listed above that have significant identified archiving requirements are directed to plan to use CLASS to meet their requirements and to provide additional resources to the CLASS program for archiving services. Alternately, they can use the CLASS enterprise approach to develop data management systems, but exemptions to this policy must be granted by the NOSC and the CIO Council.” Issue: Is this NOAA policy reasonable for 1) large array data sets and 2) smaller multi-discipline data sets?

16. Data Archive & Access Requirements Working Group Chicago, IL 16 May 24-25, 2007 Questions