NMFS Enterprise Data Management 1 DAARWG Meeting December 8, 2010, 2010 Jim Sargent, NOAA Fisheries Information Architect

NMFS Enterprise Data Management The End in Mind: Take Aways Need to make quality data available to the world It will take a cultural shift - Which is HARD! NMFS EDM: One LO’s approach Understand the comprehensive enterprise-wide process we have been through, and Possibly leverage the fruits of our labors and learn from challenges DAARWG recommendations: –Expand scope beyond Access and Archive Requirements to full Data Management Life Cycle –Promote and support EDMC and LO data management efforts –Leverage similarities while respecting diversity of data –Review and support EDMC’s Procedural Directives –Consider: Adopting a new vision of data management Doing a comprehensive inventory

NMFS Enterprise Data Management Outline A Case for a New Vision for Data and Data Management One LO’s approach: NMFS EDM Recommendations

NMFS Enterprise Data Management

Volume and diversity of data Collected for one-use; potential for multi-use Infrastructure needed to support multiple access mechanisms Releasability of data (organizational vs. technical) Coordination of collection and distribution Comparison and utilization of modeling outputs and observations Overall Need for: Clear and consistent data management policy Better data documentation (metadata) An overarching response plan Need to quickly identify applicable / available data Deep Water Horizon Data Management Challenges Prepared by Environmental Data Management Committee for NOAA Leadership 5 Pre and Post Disaster Immediate Response

NMFS Enterprise Data Management It’s the Next Disaster, Do We Even Know What Data We Have?

NMFS Enterprise Data Management “NOAA is like a library without any card catalog …or even bookshelves” ……..Dr. John A. Knauss, Former NOAA Administrator, 1986

NMFS Enterprise Data Management President's Directive on Open Government –Transparency, participation, and collaboration –Timely publication of quality information It’s the right thing to do! Greater than archive and preservation issues –Not all access is done from archives

NMFS Enterprise Data Management Everyone is Wresting With This Other federal agencies –IWGDD, NSF, ICES, ISO DM maturing as a discipline –DAMA DM Book of Knowledge (DMBOK) –Certified Data Management Professionals H. R April 15, 2010 –Requires Federal agencies to develop public access policies NMFS Science Center Data Management Reviews

NMFS Enterprise Data Management It’s About a Cultural Shift Need to move from the academic paradigm –Publish or Parish  Share or Parish – Science  Science Information –Science Quality  good enuf Changing Culture is Hard –Tipping Point, Blink …. Malcolm Gladwell –Switch, Made to Stick … Phil and Dan Heath

NMFS Enterprise Data Management A New Vision?? NOAA data assets are recognized and managed as a core agency resource, on par with financial and human resources. 11

NMFS Enterprise Data Management

Science Board Leadership 13 Analysis & Research Recommendations: Policies Procedural directives Guidelines Best practices Implementation Marketing Approvals and support Policies; Procedural directives Guidelines Best practices NMFS Staff FIMAC NMFS Staff

NMFS Enterprise Data Management Jan 15, 2009 IA Position Filled FIMC researched NMFS DM and developed EDM Recommendations Recommendations presented to and accepted by LC FIMAC Created Teams Created IMCs Created FIMAC Workshop Teams’ Plans developed Required Resources Identified Policy drafted and vetted Data Documentation Procedure Directive drafted and vetted Data Inventory Initiated Implementation Planning and Development of Teams Research, Analysis, and Recommendations A Brief History of EDM Time Data Stewardship Teams Established NMFS Data Catalog established Policy Enacted

NMFS Enterprise Data Management 15 NMFS EDM MISSION: “To effect a cultural change in which all NMFS data are recognized and managed as a core agency resource, on par with financial and human resources.”

NMFS Enterprise Data Management 16 NMFS EDM Vision NMFS customers can confidently find, access, and use our data

NMFS Enterprise Data Management 17 NMFS EDM Vision NMFS customers can confidently find, access, and use our data Internal and external constituents

NMFS Enterprise Data Management 18 NMFS EDM Vision NMFS customers can confidently find, access, and use our data Confidence in finding and trust in using our data

NMFS Enterprise Data Management 19 NMFS EDM Vision NMFS customers can confidently find, access, and use our data Using various portals, data.gov, geospatial.gov, etc) Browse through an ordered hierarchies or taxonomonies Search using: Discipline specific key words (controlled vocabularies) User tags (folksonomies) Metadata include users’ comments Minimize the number mouse clicks

NMFS Enterprise Data Management 20 NMFS EDM Vision NMFS customers can confidently find, access, and use our data Through confidentiality and security filters while using standard tools and formats

NMFS Enterprise Data Management 21 NMFS EDM Vision NMFS customers can confidently find, access, and use our data Download selected data with sufficient documentation, including quality indicators and warnings to effectively and properly use and understand the data

NMFS Enterprise Data Management 22 NMFS EDM Vision NMFS customers can confidently find, access, and use our data All NMFS enterprise data we choose to share

NMFS Enterprise Data Management 23 NMFS EDM Vision NMFS customers can confidently find, access, and use our data

NMFS Enterprise Data Management 24 NMFS Enterprise Data And Information Management Policy Overview Enacted by Eric Schwaab: June 2010 –Directed RAs, SDs, and ODs to send 2-3 people to NMFS Data Stewardship Workshop General Policy: All data shall: –be visible, accessible and understandable to authorized users; –modeled, named and defined consistently across and within all NMFS programs; –have a standard set of metadata; and –be managed, controlled and shared by data stewards throughout the data management lifecycle. –be publicly available generally within one-year of its collection A high level NMFS Policy Directive implemented by Operational Procedure Directives

NMFS Enterprise Data Management 25 Procedural Directives Principles/Concepts Developing Procedural Directives an iterative process Data Documentation Procedural Directive: FMCs shall: –Develop their own plans for documenting and sharing data assets –Inventory and document data assets and tools in NMFs metadata repository, InPort –Measure metadata quality with a rubric FY11 will be a year of learning and practicing how to document and share our data Develop, use, and refine metadata standards Populate InPort with discovery level metadata Develop practice and refine procedures for sharing

NMFS Enterprise Data Management Data Stewardship Teams 26

NMFS Enterprise Data Management NMFS Data Stewardship Workshop Discuss New Data Management Policies and procedures Develop best practices –How to document our data e.g., quality assessment –Spirals in data documentation –Rubrics and metrics Q&A and Sidebar Communities 27 To develop a shared understanding of data stewardship, empower the data stewardship teams to lead the way for implementing the Data Documentation Procedure Directive, to develop best practice based on the experience during the initial documentation and inventory tasks.

NMFS Enterprise Data Management Next Steps Data Doc. PD approved – Dec 2010 Data Inventory Complete – Dec 2010 Data Doc. Implementation Plans Due- Feb 2011 Data Stewardship Workshop – Apr 2011 First cut Best Practices – Apr 2011 Data Doc. Implementation Plans Finalized – May 2011 Address Preservation and Terminology Socialize, Support, Promote, and Market

NMFS Enterprise Data Management Critical Success Factors Management Commitment Dedicated, Personally Committed Team –FIMAC –Coordination Team –EDM Partners –Steady Hand at the Helm –Executive Sponsorship Drive Up and Then Drive Down –Harness the mavens Socialize, Support, Promote, and Market 29

NMFS Enterprise Data Management Challenges Tipping Point almost reached but not there yet Budget Shortfall –Perception that field not fully bought into it Overworked Teams a high risk

NMFS Enterprise Data Management DAARWG Recommendations Expand scope beyond Access and Archive Requirements to full Data Management life Cycle Support cultural change needed to meet demand for quality data Promote and Support EDMC and LO DM Efforts Review and Support Procedural Directives Leverage similarities while understanding diversity –Establish good data management practices for the whole data management lifecycle Consider Recommending –A new Vision of Data Management –Conducting a comprehensive inventory with defined metadata

NMFS Enterprise Data Management 32

NMFS Enterprise Data Management 33

NMFS Enterprise Data Management Backup Slides 34

NMFS Enterprise Data Management 35

NMFS Enterprise Data Management 36 Data Sharing Data shall be shared in data.gov, as appropriate, as a one- click data asset, a one-click product, or by using a software tool (e.g., FOSS) Sufficient documentation to understand the data being shared must be published in InPort and referred to or provided with the data Data Stewards decide what data is appropriate for sharing Sharing confidential data must conform to Agency policy “Data should be made as widely and freely available as possible while safeguarding the privacy of participants, and protecting confidential and proprietary data”

NMFS Enterprise Data Management 37 Timelines for Sharing Types of Data Does not include provisional, predecisional documents and preliminary analyses leading to final management actions

NMFS Enterprise Data Management 1.Data should be archived and accessible 2.Adequate resources for end-to-end management 3.Management activities should involve users 4.Interagency and international partnerships 5.Metadata are essential 6.Expert stewards required for management 7.Process to decide what data to archive 8.Archive must support discovery, access, and integration 9.Effective management requires a formal, ongoing planning process Prepared by Environmental Data Management Committee for NOAA Leadership 38 National Research Council Committee on Archiving and Accessing Environmental and Geospatial Data at NOAA, 2007 Principles for Effective Environmental Data Management

NMFS Enterprise Data Management 39 Identifying the Problem Top priority Issues –Critical gaps –No authoritative data inventory –Insufficient metadata –Data quality and consistency challenges –Ability to integrate data –Administrative systems Other issues –Data being lost –Data not being archived for perpetuity –Historical data that need rescuing –Communications re: applications and IT The FIMC identified 12 key issues and interviewed their management to determine their priorities Issues Identification Interestingly, the lowest ranking issue was that NMFS did not have buy-in across FMCs for addressing IM return

NMFS Enterprise Data Management 40 Identifying the Problem Top priority Issues –Critical gaps –No authoritative data inventory –Insufficient metadata –Data quality and consistency challenges –Ability to integrate data –Administrative systems Other issues –Data being lost –Data not being archived for perpetuity –Historical data that need rescuing –Communications re: applications and IT The FIMC identified 12 key issues and interviewed their management to determine their priorities Issues Identification Interestingly, the lowest ranking issue was that NMFS did not have buy-in across FMCs for addressing IM return

NMFS Enterprise Data Management NOAA’s Conceptual Framework 41

NMFS Enterprise Data Management 42 DAMA-DMBOK Functional Framework