Scott Hausman Acting Director, NOAA National Climatic Data Center (NCDC), Asheville, North Carolina June 3, 2010

Overview Leader in Environmental Data ManagementLeader in Environmental Data Management Partner with National Research CouncilPartner with National Research Council Transforming NOAA Data ManagementTransforming NOAA Data Management Strengthening Policies and DirectivesStrengthening Policies and Directives Investing in Enterprise IT InfrastructureInvesting in Enterprise IT Infrastructure Leveraging Universal StandardsLeveraging Universal Standards Expanding Data Discovery and AccessExpanding Data Discovery and Access Redefining Scientific Data StewardshipRedefining Scientific Data Stewardship Developing a Data Management WorkforceDeveloping a Data Management Workforce The way forward and how BRDI can helpThe way forward and how BRDI can help 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting

Leader in Environmental Data Management Improving data stewardship is among NOAA’s top priorities! 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting Mission: To understand and predict changes in Earth’s environment and conserve and manage coastal and marine resources to meet our Nation’s economic, social, and environmental needs Vision: An informed society that uses a comprehensive understanding of the role of the oceans, coasts, and atmosphere in the global ecosystem to make the best social and economic decisions

Leader in Environmental Data Management Broadest Scope of any Agency for Environmental Data Stewardship 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting Space Observations Ocean Observations Land Surface Observations Atmospheric Observations ~150 Research & Operational Observing Systems ( ~4-5 Petabytes of data/year (~15 Pb total) Data Management Challenges are Changing No longer about data volume Data discovery and integration Data stewardship and information

Partner with National Research Council Principles for Effective Environmental Data Management 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting 1.Data should be archived and accessible 2.Adequate resources for end-to-end management 3.Management activities should involve users 4.Interagency and international partnerships 5.Metadata are essential 6.Expert stewards required for management 7.Process to decide what data to archive 8.Archive must support discovery, access, and integration 9.Effective management requires a formal, ongoing planning process National Research Council Committee on Archiving and Accessing Environmental and Geospatial Data at NOAA

Strengthening Policies and DirectivesStrengthening Policies and Directives Investing in Enterprise IT InfrastructureInvesting in Enterprise IT Infrastructure Leveraging Universal StandardsLeveraging Universal Standards Expanding Data Discovery and AccessExpanding Data Discovery and Access Redefining Scientific Data StewardshipRedefining Scientific Data Stewardship Developing a Data Management WorkforceDeveloping a Data Management Workforce Transforming NOAA Data Management 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting

Coordinates the development of NOAA’s environmental data management strategy, and policy, and provides guidance to ensure consistent implementation across NOAA, on behalf of the NOSC and CIO CouncilCoordinates the development of NOAA’s environmental data management strategy, and policy, and provides guidance to ensure consistent implementation across NOAA, on behalf of the NOSC and CIO Council Environmental data management is an end-to-end process that includes acquisition, quality control, validation, reprocessing, storage, retrieval, dissemination, and long-term preservation activitiesEnvironmental data management is an end-to-end process that includes acquisition, quality control, validation, reprocessing, storage, retrieval, dissemination, and long-term preservation activities The goal of the EDMC is to enable NOAA to maximize the value of its environmental data assets through sound and coordinated data management practicesThe goal of the EDMC is to enable NOAA to maximize the value of its environmental data assets through sound and coordinated data management practices Leadership: Chair and Deputy Chair appointed by NOSC and CIO CouncilLeadership: Chair and Deputy Chair appointed by NOSC and CIO Council MembershipMembership Line Office RepresentativesLine Office Representatives NOAA Chief Enterprise ArchitectNOAA Chief Enterprise Architect NOAA Data Management ArchitectNOAA Data Management Architect Ex-officio or AdvisoryEx-officio or Advisory NOAA National Data Center DirectorsNOAA National Data Center Directors Designated Mission-Goal & Sub-Goal Team RepresentativesDesignated Mission-Goal & Sub-Goal Team Representatives NOAA liaisons to key Federal and International initiatives concerning environmental data managementNOAA liaisons to key Federal and International initiatives concerning environmental data management Strengthening Policies and Directives 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting Environmental Data Management Committee (EDMC) Established in Fall of 2009 Helen Wood current Chair NOAA EDMC Environmental Data Management Committee EDMC Environmental Data Management Committee CIOC Chief Information Officer Council CIOC Chief Information Officer Council NOSC NOAA Observing System Council NOSC NOAA Observing System Council DMIT Data Management Integration Team DMIT Data Management Integration Team

Strengthening Policies and Directives NOAA Environmental Data Management Framework 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting Overarching all Aspects of the Data Management Lifecycle Governance, Requirements Management, Architecture Management Developing and maintaining rich metadata to accompany the data Establishing mechanisms that allow for user requirements and feedback Governance, Requirements Management, Architecture Management Developing and maintaining rich metadata to accompany the data Establishing mechanisms that allow for user requirements and feedback Observing Operations Actual observation Transmission/proces sing QA Integration with other data to create products (e.g., models) Dissemination to real-time subscribers Delivery to archive Observing Operations Actual observation Transmission/proces sing QA Integration with other data to create products (e.g., models) Dissemination to real-time subscribers Delivery to archive Archive Ingest (Receipt) Archival storage Data management (populating catalogs, registries, metadata) Preservation planning (migration to new technologies) Archive Ingest (Receipt) Archival storage Data management (populating catalogs, registries, metadata) Preservation planning (migration to new technologies) Access Discovery (catalogs, registries, metadata) Dissemination to users (web services, legacy systems, standard formats) Access Discovery (catalogs, registries, metadata) Dissemination to users (web services, legacy systems, standard formats) Use Integration with other information (NOAA, others) Assimilation into models Product creation Make decisions (policy, emergency, others) Scientific discovery Feedback to NOAA Use Integration with other information (NOAA, others) Assimilation into models Product creation Make decisions (policy, emergency, others) Scientific discovery Feedback to NOAA Planning of New Observing or Data Management Systems Requirements definition Analysis of alternatives Systems design Integration with observing systems (NOAA, interagency, state, international) Determining what to archive and associated funding Buy/build Planning of New Observing or Data Management Systems Requirements definition Analysis of alternatives Systems design Integration with observing systems (NOAA, interagency, state, international) Determining what to archive and associated funding Buy/build Stewardship Overarches Observing Operations, Archive, Access, Use All ongoing, iterative processes that improve: 1) data and metadata content (include reprocessing data) and 2) access and user understanding Stewardship Overarches Observing Operations, Archive, Access, Use All ongoing, iterative processes that improve: 1) data and metadata content (include reprocessing data) and 2) access and user understanding

Maintains NOAA’s policy of “full and open access” to environmental dataMaintains NOAA’s policy of “full and open access” to environmental data Provides mechanism for EDMC to develop procedural directives for more detailed guidance (e.g., the NOAA Procedure for Scientific Records Appraisal and Archive Approval)Provides mechanism for EDMC to develop procedural directives for more detailed guidance (e.g., the NOAA Procedure for Scientific Records Appraisal and Archive Approval) Presents an end-to-end lifecycle framework for the management of environmental dataPresents an end-to-end lifecycle framework for the management of environmental data Signed by NOAA CIO; awaiting clearance from Chief Administrative Officer and General CouncilSigned by NOAA CIO; awaiting clearance from Chief Administrative Officer and General Council Strengthening Policies and Directives 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting Revision of NOAA Administrative Order (NAO) Establishes a NOAA policy for acquiring, integrating, managing, disseminating, and archiving environmental and geospatial data and information obtained from worldwide sources to support NOAA's mission. “What to Archive”

Investing in Enterprise IT Infrastructure NOAA’s primary enterprise IT system for archive and accessNOAA’s primary enterprise IT system for archive and access Employs OAIS-RMEmploys OAIS-RM Enterprise benefits include:Enterprise benefits include: Economy of ScaleEconomy of Scale High Quality of ServiceHigh Quality of Service A System Evolution ApproachA System Evolution Approach Location of “Nodes”Location of “Nodes” Operational: Asheville, NC (NCDC); Boulder, CO (NGDC)Operational: Asheville, NC (NCDC); Boulder, CO (NGDC) Development and Test: Suitland, MD; Fairmont, WVDevelopment and Test: Suitland, MD; Fairmont, WV Environmental Data HoldingsEnvironmental Data Holdings Current: POES, DMSP, GOES, CFSR (Model Reanalysis)Current: POES, DMSP, GOES, CFSR (Model Reanalysis) Development: MetOp, EOS MODIS, NPPDevelopment: MetOp, EOS MODIS, NPP Comprehensive Large Array-data Stewardship System (CLASS) 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting

Leveraging Universal Standards Reference model is to be applicable to all digital archives, and their Producers and ConsumersReference model is to be applicable to all digital archives, and their Producers and Consumers Identifies a minimum set of responsibilities for an archive to claim it is an OAISIdentifies a minimum set of responsibilities for an archive to claim it is an OAIS Establishes common terms and concepts for comparing implementations, but does not specify an implementationEstablishes common terms and concepts for comparing implementations, but does not specify an implementation Provides detailed models of both archival functions and archival informationProvides detailed models of both archival functions and archival information Discusses OAIS information migration and interoperability among OAISsDiscusses OAIS information migration and interoperability among OAISs Open Archival Information System Reference Model (OAIS-RM) 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting

Leveraging Universal Standards Global Earth Observation-Integrated Data Environment (GEO-IDE) Scope – NOAA-wide architecture development to integrate legacy systems and guide development of future NOAA environmental data management systemsScope – NOAA-wide architecture development to integrate legacy systems and guide development of future NOAA environmental data management systems Vision – NOAA’s GEO-IDE is envisioned as a “system of systems” – a framework that provides effective and efficient integration of NOAA’s many quasi- independent systemsVision – NOAA’s GEO-IDE is envisioned as a “system of systems” – a framework that provides effective and efficient integration of NOAA’s many quasi- independent systems Foundation – built upon agreed standards, principles and guidelinesFoundation – built upon agreed standards, principles and guidelines Approach – evolution of existing systems into a service-oriented architectureApproach – evolution of existing systems into a service-oriented architecture Result – a single system of systems (user perspective) to access the data sets needed to address significant societal questionsResult – a single system of systems (user perspective) to access the data sets needed to address significant societal questions 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting Unified Access Framework for Gridded Data (UAF Grid) Integrated Ocean Observing System Data Integration Framework (IOOS DIF)

Expanding Data Discovery and Access 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting NOAA National Data Centers NOAA National Data Centers NOAA Centers of Data NOAA Other Data Sources External Data Sources Portals Cloud Resources Search Engines Portals Cloud Resources Search Engines Online Requests Subscription Services Online Requests Subscription Services Web Services M2M Interfaces Web Services M2M Interfaces Source Agnostic Interface / Federated Data Sources for Transparent Access Metadata Catalogs for Data Discovery Source Agnostic Interface / Federated Data Sources for Transparent Access Metadata Catalogs for Data Discovery Tiers of Access (Customer Sophistication)

Redefining Scientific Data Stewardship New approach for real time data management and production of climate data records 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting Climate Quality Data Records & Products Observing System MonitoringClimate Data Records Observing System Operators Reprocessing & Reanalyses Feedbacks Operational Product Processing Random & Time Dependent Error Checks Original Observations & Metadata Sentinel Scientific Stewardship Teams Scientific Stewardship Teams Metadata Archives Random & Time Dependent Error Checks Intercomparison and Analysis Rapid feedback to observing system Scientist/Analysts involved with observations early on Enable and facilitate future research Safeguard interests of future generations End-to-end accountability of data  Spatial and temporal sampling  Time dependent biases  Metadata  Reprocessing for CDRs

NOAA/NESDIS Top PriorityNOAA/NESDIS Top Priority Partnering with Earth Science Information Partners (ESIP) FederationPartnering with Earth Science Information Partners (ESIP) Federation One day with afternoon practicumOne day with afternoon practicum Focus on graduate students and junior scientistsFocus on graduate students and junior scientists Target Fall AGU MeetingTarget Fall AGU Meeting Developing a Data Management Workforce 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting

Conclusion The Way ForwardThe Way Forward Translate NAO in NOAA DirectivesTranslate NAO in NOAA Directives Finalize a NOAA-wide CONOPS for ArchiveFinalize a NOAA-wide CONOPS for Archive Prototype federated architecturePrototype federated architecture How BRDI can helpHow BRDI can help Defining archival standards for research/small data setsDefining archival standards for research/small data sets Improving interdisciplinary integration of dataImproving interdisciplinary integration of data Increasing transparency and discovery to enhance data reuse and avoid redundant researchIncreasing transparency and discovery to enhance data reuse and avoid redundant research 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting

Scott Hausman Acting Director NOAA’s National Climatic Data Center (NCDC) 151 Patton Avenue, Room 557 Asheville, NC    /3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting

Background Material 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting

1.Commitment and leadership: Information is a strategic asset and information management must be a key component of every environmental data and information program. This ethic must be reflected in a corporate culture, embraced throughout the organization, that recognizes data as a corporate resource. 2.Stewardship: People who take observations or produce data and information are stewards of these data, not owners. These data must be collected, produced, documented, transmitted and maintained with the accuracy, timeliness and reliability needed to meet the needs of all users. 3.Long-term preservation: Irreplaceable observations, data products of lasting value, and associated metadata must be preserved. This information must be well-documented and maintained so that it is available to and independently understandable by users, now and in the future. 4.Requirements-driven: It is essential that providers and users of data and products play an active role in defining the constantly evolving requirements that drive the development and evolution of data management systems. 5.Discovery and access: Freedom of access, mechanisms that facilitate discovery, timely delivery, use and interpretation of data and products (directories, browse capabilities, metadata, mapping, visualization, etc.) are essential, recognizing relevant policies and regulations. 6.Standards and practices: Appropriate use of information technologies, widely shared standards, and integration approaches are vital to facilitate collection, management, discovery, dissemination, and access services for environmental data and products. This will ensure interoperability among providers, systems, and users. Effective application of standards and best practices contribute to the development of systems that are interoperable, efficient, reliable, scalable, and adaptable. 7.Quality: Data, products and information should be of quality sufficient to meet the requirements of society and to support sound decision making. 8.Cooperation and coordination: Environmental and scientific data management is a task of global scope – a whole that should be much bigger than the sum of its parts. It is only by participating in a global community of integrated data management that each organization can realize the potential of its data to the betterment of humankind. 9.Security: Data, information, and products must be preserved and protected from unintended or malicious modification, unauthorized use, or inadvertent disclosure. NOAA Data Management Principles 6/3/2010 NOAA Data Stewardship, Third NRC/BRDI Meeting