EDMC Archive Architecture Team (AAT) Prepared for the Data Archiving and Access Requirements Working Group (DAARWG) Ken McDonald December 9, 2010

Prepared by Archive Architecture Team for DAARWG2 Outline  Introduction  Task Evolution Phase I - Data Center/CLASS Focus Phase II – “Centers of Data”  Current Activity  Summary of Observations  Way Forward  Discussion

Broad Scope for Environmental Data Stewardship ~150 Research & Operational Observing Systems ~4-5 Petabytes of data/year (~15 Pb total) Prepared by Archive Architecture Team for DAARWG3 Data Management Challenges are Changing No longer just about data volume Data discovery and integration Data stewardship and information NOAA’s Environmental Information Management Challenges

Prepared by Archive Architecture Team for DAARWG4 “NOAA is, at its foundation, an environmental information generating organization.” - From the NGSP, 2010 Origin of Archive Architecture Team At early DAARWG meetings NOAA reported on Data Centers, CLASS, GEO-IDE Preservation and stewardship of data and information was a general area of interest DAARWG recognized some lack of coordination across initiatives Archive Architecture Team formed in response to concerns NOAA EDMC Environmental Data Management Committee EDMC Environmental Data Management Committee CIO Council Chief Information Officer Council CIO Council Chief Information Officer Council NOSC NOAA Observing System Council NOSC NOAA Observing System Council DMIT Data Management Integration Team DMIT Data Management Integration Team AAT Archive Architecture Team AAT Archive Architecture Team

Prepared by Archive Architecture Team for DAARWG5 Archive Architecture Team Members Steve DelGreco/NESDIS-NCDC Scott Cross/NESDIS-NODC/NCDDC Dan Kowal/NESDIS-NGDC Brad Nunn/NESDIS-NODC/NCDDC Ken Casey/NESDIS-NODC Rick Vizbulis/NESDIS-OSD-CLASS Doug Zirkle/NESDIS-OSD-CLASS Tina Chang/NMFS Jim Sargent/NMFS Maureen Kenny/NOS Justin Cooke/NWS Bob Lipschutz/OAR-ESRL Derrick Snowden/OAR Richard Bouchard/NWS-NDBC Lewis McCulloch/NESDIS-OSD Ken McDonald/NESDIS-OSD Adam Steckel/NESDIS-OSD

Prepared by Archive Architecture Team for DAARWG6 Phase I Study – Data Centers and CLASS Open Archival Information System Reference Model (OAIS-RM) Functions Data Centers and CLASS following the International Standard for information preservation (OAIS-RM) This standard identifies the functions required to provide long-term preservation Study looked at details of the RM and extensions for a multi-center deployment of CLASS Clarified GEO-IDE focus as NOAA-wide data access and integration Raised the question of NOAA data collections at other facilities (“Centers of Data”)

Prepared by Archive Architecture Team for DAARWG7 Phase II Study – Centers of Data Major NOAA Data and Information Repositories Study broadened to consider archives for all NOAA data and information Team members assembled information by line office on major data collections Also characterized line office functions and facilities Cases where data migrates to Data Centers for preservation and stewardship identified as “best practices. “What to archive” procedure was concurrently developed by a separate team

Address need for a NOAA Archive Concept of Operations – Start with review of all relevant documentation – Use observations and lessons from earlier study phases Concept should include “What to Archive” procedure but also “How” and “Where” ConOps will describe overall procedure for making archive decisions and how it fits in the end- to-end environmental data management life-cycle Prepared by Archive Architecture Team for DAARWG8 Current Study Focus A common language to promote integration across the diverse stakeholders in the NOAA environmental data lifecycle

1.Data should be archived and accessible 2.Adequate resources for end-to-end management 3.Management activities should involve users 4.Interagency and international partnerships 5.Metadata are essential 6.Expert stewards required for management 7.Process to decide what data to archive 8.Archive must support discovery, access, and integration 9.Effective management requires a formal, ongoing planning process Prepared by Archive Architecture Team for DAARWG9 National Research Council Committee on Archiving and Accessing Environmental and Geospatial Data at NOAA, 2007 Principles for Effective Environmental Data Management

Specification of end-to-end data management life-cycle components Using NAO Definitions – Data Management – Data Management Services – Data Stewardship Envision “Archive Procedure” coming out of ConOps to specify decision making process Prepared by Archive Architecture Team for DAARWG10 Environmental data will be visible, accessible and independently understandable to users, except where limited by law, regulation, policy or by security requirements. Following the NOAA-wide Policy (NAO)

Prepared by Archive Architecture Team for DAARWG11 Determining what environmental data are required to be preserved for the long term and how preservation will be accomplished Developing and maintaining metadata throughout the environmental data lifecycle that comply with standards Obtaining user requirements and feedback Developing and following data management plans that are coordinated with the appropriate NOAA archive for all observing and data management systems Conducting scientific data stewardship to address data content, access, and user understanding Providing for delivery to the archive and secure storage Providing for data access and dissemination Enabling integration and/or interoperability with other information and products End-to-End Data Management Lifecycle Components As specified in revised NAO

Prepared by Archive Architecture Team for DAARWG12 Phase I and II Study Results Data Centers and CLASS use of the OAIS Reference Model good starting point for an Archive ConOps Identified and described full set of archive functions Provides common terminology Multiple examples of Project/Program collaboration with Data Centers NWS/National Data Buoy Center sends DART buoy data to NGDC and all other collections to NODC NOS sends hydrographic survey data to NGDC OAR has used NCDC as its archive for U.S. Integrated Surface Irradiance (ISIS) Level 2 data (SURFRAD) since collection began in 1995 Key NMFS data sets are transferred to NODC archive

Prepared by Archive Architecture Team for DAARWG13 Archive Function Reflected in NOAA Program Plans More projects developing Data Management Plans to address full data lifecycle Major satellite campaigns and large surface observation programs Coral Reef Program Greater understanding of the role of the National Data Centers Rolling Deck to Repository (R2R)

NRC Principles and Guidelines NAO OAIS-RM “What to Archive” Procedure “How to Archive” CONOPS… A Framework for developing a “Concept of Operations for NOAA’s Archives” Justification and need for stewardship. The “objective vision.” NOAA policy direction. Best practices and common language for discussing archives. The "conceptual framework" for archives and information preservation. Provides decisions regarding the information to preserve without specifying anything regarding the how to archive. Provides overview or vision for how NOAA’s Archives work together. Provides way to shape implementation decisions or at least frames the questions that need to be answered. WHYWHY HOWHOW Archive Architecture Team Report

Terminology important…to a point -Drawing from NAO and OAIS -OAIS very precise but different from common usage -ConOps will have to reconcile definitions Three NOAA Data Centers recognized as enterprise archive centers -Fully aligned with charter and expertise -Good partnering relationships with NOAA programs -Clarified role of CLASS as the IT component -Single solution not most effective -Diversity of collections, resource restrictions, heritage capabilities, etc. may require different approach for different circumstances (e.g “levels of service”) -Archive Con-Ops and procedures should reflect this Focus of ConOps will be “Information Preservation” -Science stewardship linked to preservation but requires its own set of procedures -ConOps will address relationship to science stewardship and other data life-cycle functions 15 Observations and Conclusions

The Decision Making Process associated with NOAA’s End-to-end Data Life Cycle Concept of Operations for the Preservation and Stewardship of NOAA’s Environmental Information 16 End-to-end Data Life Cycle Decision Making Process Data Identification Stage Resource Verification Stage NOAA Archive Qualification Stage NOAA Procedure for Scientific Records Appraisal and Archival Approval (“What to Archive”) Submission Agreement Stage

NOAA Procedure for Scientific Records Appraisal and Archival Approval (aka the “What to Archive” Process) 17 Detailed View: What to Archive Process

18 Way Forward Plan is to continue development of major decision processes: Identify key questions Leverage best practices to develop options Leverage early usage of “What to Archive” procedure Include issue of access to Archived data Propose procedure(s) to determine answers Develop flow chart for each procedure Use procedures to vet approach with management and stakeholders Envision significant interactions with EDMC and CIO Council Concurrently, develop ConOps to fully document procedures

19 Areas for Discussion “Information Preservation” – Correct focus? “Levels of Service” – Good idea? “Separation of Preservation and Stewardship” – Right approach? “Effective procedures” – How do we provide useful guidance?