June 23-24, 2011

Development of a NOAA Policy on the Use of External Data Sources 2 DAARWG MeetingJune 23-24, 2011 Motivation for Data Sharing Partnerships Proposed Climate Service Interagency Climate Portal National Climate Assessment Data Management Challenges of External Data Example: CoCoRaHS Example: EarthNetworks GHG Network Existing NOAA Policy Use Case: Argo Ocean Floats

Development of a NOAA Policy on the Use of External Data Sources 3 DAARWG MeetingJune 23-24, 2011 “The Climate Service Line Office at NOAA … would work with the broader climate service enterprise … to provide businesses, communities, and resource managers with services and information for decision-making. The proposed Climate Service Line Office at NOAA would improve NOAA’s organization, such that the agency can be a more accessible, transparent, and collaborative partner… NOAA’s reorganization would also support economic innovation and entrepreneurship. This includes supporting development of the private sector climate services industry.” — Written Statement to the House Committee on Science, Space and Technology Under Secretary Jane Lubchenco, June 22, 2011

Development of a NOAA Policy on the Use of External Data Sources 4 DAARWG MeetingJune 23-24, 2011 Information Delivery and Decision Support NOAA uses its national and regional infrastructure to deliver climate services today Assessments of Climate Change and Impacts NOAA is a leader in national and regional climate impact assessments Over 70% of Federal IPCC AR4 WG1 authors were from NOAA Climate Change Research and Modeling International award winning models of the global climate Climate Observations and Monitoring NOAA operates over 90 observation and monitoring systems NOAA is mandated to monitor and provide access to climate data and information Security Forestry Water Health Infrastructure Land Management Oceans Energy Federal Response to the nation’s climate challenges NOAA’s Assets Partnerships & Collaboration Vision and Strategic Framework Vision and Strategic Framework

Development of a NOAA Policy on the Use of External Data Sources 5 DAARWG MeetingJune 23-24, 2011 National Climate Adaptation Summit, called for by John Holdren, Director of the White House Office of Science and Technology Policy, issued “seven priorities for near-term action” in 2010, including “Creating a federal climate information portal. This would provide single-point access to data from all relevant federal agencies and programs and would evolve over time into a more ‘national’ portal with information about relevant non-federal efforts.” Also, Executive Office of the President, Committee on Environment, Natural Resources, and Sustainability (CENRS) Roundtable on Climate Information Services called for an interagency climate information portal plan, focusing on National Climate Assessment

Development of a NOAA Policy on the Use of External Data Sources 6 DAARWG MeetingJune 23-24, 2011 A National Climate Portal as envisioned is an immense task Use the National Climate Assessment (NCA) to focus the initial scope The NCA can be… A starting point, extensible to interagency data A driver of requirements for Data Management Planning: transparency, access, usability, data quality, and business rules A test case for connecting robust and federated data structures with appropriate user experiences

Development of a NOAA Policy on the Use of External Data Sources 7 DAARWG MeetingJune 23-24, 2011 A product AND a sustained process More stakeholder engagement and contributions More transparency, traceability, accessibility and usability More data, more authors, more users A suite of Indicators (discrete and composite, physical and societal) Next report due in 2013, draft due in 2012, Federal Advisory Committee just formed Snapshots plus frequent updates More frequent interim reports, updated indicators and fresh web-content are expected… BUT An assessment is necessarily a snapshot in time and should be reproducible

Development of a NOAA Policy on the Use of External Data Sources 8 DAARWG MeetingJune 23-24, 2011 External Data Clear need to move beyond government data and the peer reviewed scientific literature as information source, but who guarantees access and a continuous archive? Quality Information Quality Act requirements (highest level) Peer review requirements (highest level)

Development of a NOAA Policy on the Use of External Data Sources 9 DAARWG MeetingJune 23-24, 2011 NCA will likely not ‘host’ basic data, but will provide central access to federated system (critical questions remain for external data) Highly-processed, derived data used in the report may be centrally hosted Must accommodate huge data sets Too big to move, too big to host all data in one location Implies distributed data architecture, federated storage Metadata must adhere to common standards Uncertainty must be documented Quality and original source of data must be clear and peer reviewed Data system must be interoperable Use of data.gov must have clear guidelines Narrative assessment and graphics must link to source data and information

Development of a NOAA Policy on the Use of External Data Sources 10 DAARWG MeetingJune 23-24, 2011 Data Documentation and Quality Are the metadata sufficient and standardized? Does the data come from a trusted source (NASA vs. Joe’s Data Shack) and what is the process to certify data sources as “trusted”? How robust are the science algorithms and quality control procedures? Data Access and Usage Are the data accessible via standardized Web services? Is the data source operationally reliable? Are there data usage or cost restrictions? Data Retention What are the long-term archive prospects for the data? Must NOAA ensure the data are archived? Can NOAA depend on the data whenever necessary?

Development of a NOAA Policy on the Use of External Data Sources 11 DAARWG MeetingJune 23-24, 2011 Unique, non-profit, community-based network of volunteers working together to measure and map precipitation (rain, hail and snow) Using low-cost measurement tools, stressing training and education, and utilizing an interactive Web-site, aim is to provide the highest quality data for natural resource, education and research applications NWS routinely uses CoCoRaHS data and NCDC includes in snow archives Non-standard procedures along with data quality and continuity issues have affected the climate record FEMA relies on NOAA-provided snow totals to assess natural disasters Snow Workshop, May 25-27, 2011organized to address these and other issues

Development of a NOAA Policy on the Use of External Data Sources 12 DAARWG MeetingJune 23-24, 2011 Over the next five years, Earth Networks will invest $25 million to build and deploy a state-of-the-art greenhouse gas emissions observation and measurement network By the middle of 2012, Earth Networks will deploy 100 greenhouse gas measuring instruments 50 in the United States 25 throughout Europe 25 elsewhere around globe Collaborating with Scripps Institution of Oceanography Potential contributor to NOAA carbon observing network?

Development of a NOAA Policy on the Use of External Data Sources 13 DAARWG MeetingJune 23-24, 2011 Much of data NOAA receives from external sources is in the form of raw observations, or “original data” IQA requires that original data be managed using documented processes with routine QC/QA ml ml

Development of a NOAA Policy on the Use of External Data Sources 14 DAARWG MeetingJune 23-24, 2011 Section …NOAA will take advantage of existing capabilities and services of commercial and academic sectors to support efficient performance of NOAA's mission and avoid duplication and competition in areas not related to the NOAA mission… …NOAA will give due consideration to these abilities and consider the effects of its decisions on the activities of these entities, in accordance with its responsibilities as an agency of the U.S. Government, to serve the public interest and advance the nation's environmental information enterprise as a whole. hap_216/ pdf hap_216/ pdf

Development of a NOAA Policy on the Use of External Data Sources 15 DAARWG MeetingJune 23-24, 2011 Primarily directs NOAA’s provision of data to the public. Data layers provided by sources other than federal agencies…and displayed on NOAA Web pages must be used judiciously, because these situations present special challenges with respect to maintaining agency Information Quality (IQ) Act standards. Such data layers must meet NOAA IQ requirements for NOAA use of third-party information… In addition, any external data from third-party providers must meet the following criteria: The data must be must be necessary for, and material to, the presentation of agency information or the delivery of agency services, and the map must credit the contributing source of the data or provide a direct link back to the third-party source data provider. The data must be relevant and timely, and complete steps must have been taken to ensure that data layers are actively updated to achieve the highest level of quality possible. ations_policy.html ations_policy.html

Development of a NOAA Policy on the Use of External Data Sources 16 DAARWG MeetingJune 23-24, 2011 Provider: International - Multiple Nations Trusted Source(s): Yes (Global Ocean Observing System) Purpose: Long-term climate monitoring of sea surface temperature and salinity Science Quality: Excellent (e.g., Scripps Institution of Oceanography) Metadata: Excellent Data Access: Excellent with multiple access points Restrictions on the use of the data: None Long-term Archive Dependability: Excellent

Development of a NOAA Policy on the Use of External Data Sources 17 DAARWG MeetingJune 23-24, 2011 NOAA is committed to partnering with other data producers, even more so in dealing with the challenges of the National Climate Assessment NOAA has been using external data sources for a long time for a number of applications, and this will continue An overall NOAA policy documenting this usage (taking into account existing policies across the agency) is needed to inform acquisition decisions NOAA has an opportunity to lead the way in defining data management practices more broadly The DAARWG is well positioned to advice NOAA on the content of such a policy

Development of a NOAA Policy on the Use of External Data Sources 18 DAARWG MeetingJune 23-24, 2011 Task is to develop a NOAA policy that defines what data should be acquired through non-NOAA sources and what standards should be applied to such acquisitions Agenda National Weather Service Existing Policy (Ed Johnson) IOOS and Data Provider Certification (Zdenka Willis) National Climate Assessment Consideration (Anne Waple)

Scott Hausman Deputy Director NOAA’s National Climatic Data Center (NCDC) 151 Patton Avenue, Room 557 Asheville, NC    Thank you!