Data Management Writers Workshop 1/13/2019 Data Management Writers Workshop Provided by Christina Horvat and Teresa Murphy NWS Office of Observations April 23, 2018

Workshop Session Objectives 1/13/2019 Workshop Session Objectives During this workshop-style session participants will have the opportunity to draft a Data Management Plan (DMP) for NOAA intramural data they produce or are responsible for. An introductory briefing will provide an overview of the NOAA DMP Template, after which participants will fill out the template with someone on hand to answer questions.

Data Management Plan (DMP) Information 1/13/2019 Data Management Plan (DMP) Information NOAA Administrative Order 212-15 establishes the Department of Commerce (DOC) National Oceanic and Atmospheric Administration (NOAA) Environmental Data Management Policy. Applies to all NOAA environmental data and to the personnel and organizations that manage the data. Requires environmental data to be visible, accessible and independently understandable to users.

DMP Information (Continued) 1/13/2019 DMP Information (Continued) Requires management environmental data to be based on end-to-end data lifecycle management: Determining what environmental data are required long term and how to preserve it Developing and maintaining metadata throughout the environmental data lifecycle Metadata should comply with standards Developing and following data management plans coordinated with the appropriate NOAA archive Obtaining user requirements and feedback

Environmental Data Recorded and derived observations and measurements. 1/13/2019 Environmental Data Recorded and derived observations and measurements. Includes numerical model outputs. Relevant model outputs should be made publicly accessible even if they are not archived. Includes digital audio or video recordings of environmental phenomena (such as animal sounds or undersea video).

Related Terms On-Line Data: Off-Line Data: Metadata: 1/13/2019 Related Terms On-Line Data: Data accessible over the Internet; either directly or through an ordering mechanism (including near-line robotic tape drives) Off-Line Data: Data recorded on paper, CD-ROM, magnetic tape, disk, or other media not accessible over the Internet. Metadata: Descriptive information, structured according to agreed-upon standards for content and machine-readable encoding, about digital environmental data or research artifacts.

1/13/2019 More on Metadata The NOAA Data Documentation Directive (v2.0) was issued by the Environmental Data Management Committee (EDMC) Requires metadata for on-line and off-line environmental data The directive has 5 core requirements, including: Metadata requirement - Metadata format Metadata content - Revised ISO standard Metadata granularity Does not apply to produced by extramural grants, third party data from external sources, etc. Provided to ensure environmental data are documented with machine-readable metadata to enable discovery, access, and use of these resources.

DMP Template The Data Management Planning Procedural Directive (v.2.0.1), Appendix A, provides the DMP Template https://nosc.noaa.gov/EDMC/PD.DMP.php These NOAA instructions will be used for Critical Systems of Record owned by NWS. Guidance will be provided for each sectional component. General comments before starting: Reference your Observing System Capabilities Summary (OSCS) as you complete the template.  https://nosc.noaa.gov/OSC/OSN/index.php Copy and paste information from OSCS to the DM Plan where applicable. This will ensure consistency in messaging.

Template Section 1 Section 1: General Description of Data to be Managed 1.1. Name of the data, data collection Project, or data-producing Program: (Must match the Exhibit 300 business case, System of Record [SoR] description, or the NOAA FISMA/System Description and System ID.) 1.2. Summary description of the data: (A 500 word or less abstract type description. For SoRs, this should be consistent with the metadata abstract tag.) 1.3. Is this a one-time collection, or an ongoing series of measurements? (A one-time collection should be for an operational and/or grant funded research project. If this is for testing, no DM plan needs to be created.) 1.4. Actual or planned temporal coverage of the data: (Focus the actual or planned temporal coverage of the data on total collection. When did your system start taking observations? Is there a planned date to sunset this observation? What is the frequency of observations collected?)

Template Section 1 (Continued) Section 1: General Description of Data to be Managed 1.5. Actual or planned geographic coverage of the data: (Provide links to maps of coverage area, if available. If not available, provide descriptions of coverage area. Copy geographical information from OSCS.) 1.6. Type(s) of data: (Specify the data formats, e.g., is this observation in GRIB, BUFR, Binary, Text, CSV?) 1.7. Data Collection Method(s): (Specify what data collection methods were used. Reference your “About” section of your website, when appropriate. Do not draft new language to describe your data collection methods.) 1.8. If data are from a NOAA Observing System of Record, enter name of system. 1.8.1. If data are from another observing system, please specify:

Template Section 2 Section 2:  Point of Contact for this Data Management Plan (author or maintainer) 2.1. Enter author’s name 2.2. Enter author’s title 2.3. Enter the author’s affiliation or facility 2.4. Enter the author’s e-mail address 2.5. Enter the author’s phone number

Template Section 3 Section 3: Responsible Party for Data Management 3.1. Program manager’s name, or their designee, who is responsible for assuring the proper management of the data produced by their program 3.2. Program manager’s title 3.3. Name of current position holder

If data is archived at NCEI, these costs should be included. 1/13/2019 Template Section 4 Section 4:  Resources Programs must identify resources within their own budget for managing the data they produce. 4.1. Have resources for management of these data been identified? 4.2. Approximate percentage of the budget for these data devoted to data management.  This includes public data access, metadata and long term data preservation costs. If data is archived at NCEI, these costs should be included.

Template Section 5 Section 5: Data Lineage and Quality 1/13/2019 Template Section 5 Section 5:  Data Lineage and Quality Context for this section:  Think of this part of the DM Plan on justifying funds for a program, or IT security related concerns, which include confidentiality, integrity and availability.   Focus on a high level data flow overview of the steps required to generate your product. Provide a direct link to a metadata record when available.  

Template Section 5 (Continued) 1/13/2019 Template Section 5 (Continued) 5.1. Processing workflow of the data from collection or acquisition to making it publicly accessible (Describe or provide URL of description) 5.1.1. If data at different stages of the workflow or products derived from these data are subject to a separate DMP, provide reference to the other plan (Example model data, raw sensor data, radar and gauge data) 5.2. Quality control procedures employed (Describe the quality control measures used in generating your product.)

Template Section 6 Section 6: Data Documentation Please review the EDMC Data Documentation Procedural Directive (linked) before completing this section. https://nosc.noaa.gov/EDMC/PD.DD.php 6.1. Does metadata comply with Data Documentation Directive? 6.1.1. If metadata are non-existent or non-compliant, please explain: 6.2. Name of organization or facility providing metadata hosting: 6.2.1. If service is needed for metadata hosting, please indicate: 6.3. URL of metadata folder or data catalog, if known:

Template Section 6 (Continued) 6.4. Process for producing and maintaining metadata (Describe or provide URL of description) To address this question: Describe any metadata tools used to generate the metadata record. List the tools used to verify/validate a metadata record. Indicate if the NOAA Rubric tool was used to show compliance with ISO 19115/19139 Standards and completeness of metadata record. Indicate if any metadata tools are used to maintain the metadata records.

Template Section 7 Section 7: Data Access 1/13/2019 Template Section 7 Section 7:  Data Access Please review the EDMC Data Access Procedural Directive (linked) before completing this section. https://nosc.noaa.gov/EDMC/PD.DA.php 7.1. Do these data comply with the Data Access Directive? 7.1.1. If the data are not to be made available to the public at all, or with limitations, has a Waiver (found in Appendix A) been filed? 7.1.2. If there are limitations to public data access, describe how data are protected from unauthorized access or disclosure:

Template Section 7 (Continued) 7.2. Name of organization or facility providing data access: (If there are “live” and “archive” data access links, provide both locations.) 7.2.1. If data hosting service is needed, please indicate: 7.2.2. URL of data access service, if known: 7.3. Data access methods or services offered: (Address whether the web services feature a human user interface with clickable interactions vs. machine to machine communications (e.g. WCS, WMS, WFS)). 7.4. Approximate delay between data collection and dissemination: 7.4.1. If delay is longer than latency of automated processing, indicate under what authority data access is delayed:

Template Section 8 Section 8: Data Preservation and Protection Please review the NOAA Procedure for Scientific Records Appraisal and Archive Approval (linked) before completing this section. https://nosc.noaa.gov/EDMC/documents/NOAA_Procedure_document_final_12-16-1.pdf 8.1. Actual or planned long-term data archive location: (Specify NODC, NCEI, NGDC, World Data Center (WDC) facility, Other, To Be Determined, Unable to Archive, or No Archiving Intended) 8.1.1. If World Data Center or Other, specify: 8.1.2. If To Be Determined, Unable to Archive or No Archiving Intended, explain:

Template Section 8 (Continued) 8.2. Data storage facility prior to being sent to an archive facility: (Answer should address whether or not a data provider originally stored the data, or a “live” web service is serving up data.) 8.3. Approximate delay between data collection and submission to an archive facility: 8.4. How will the data be protected from accidental or malicious modification or deletion prior to receipt by the archive? Discuss data back-up, disaster recovery/contingency planning, and off-site data storage relevant to the data collection: (Answer should indicate NOAA data providers follow all applicable NIST 800-53, Department of Commerce and NOAA IT Security Policies and Procedures.)