1 NOAA Environmental Data Management Committee (EDMC) Report to DAARWG Jeff de La Beaujardière, PhD EDMC Chair

2 Discoverable Accessible Documented Preserved Vision All NOAA data will be: for all types of users and applications.

3 Past: EDMC Milestones o 4 years ago:  Predecessor committee in existence (DMC)  Focus on data preservation. Archive Procedure issued  No comprehensive framework for Data Mgmt  Reporting to CIO Council only o 2 years ago:  NOAA Admin. Order (NAO) issued (2010 Nov 04)  EDMC established, reporting to CIO Council & NOAA Obs Syst Council o 1 year ago:  3 procedural directives issued (2011 Oct) Data Management Planning Data Documentation Data Sharing by Grantees  Active implementation & monitoring begins o 7 months ago:  Jeff DLB becomes EDMC chair  DM Dashboard concept developed  NOAA EDM Framework begun

4 Present: Status of EDMC Directives  Data Management Planning: 6 plans submitted to DM Plan Repository – Positive feedback regarding usefulness of DM Plan template Working on metrics for scoring plan completeness EDMC prepared to assist ITRB/CIO in reviewing DM Plans  Data Sharing by NOAA Grantees: Grants Mgmnt Div now requiring data sharing language on new FFOs – Recent: Joint Ctr for Satellite Data Assim, IOOS Regional Assoc.  Data Documentation: EDMC directive mentioned favorably in GAO Geospatial Info Report NGDC software analyzing metadata. Gradual improvement seen. NCDDC/NODC metadata training sessions well attended.  Archive Procedure: Issued 2008 but not actively promoted; now including in EDM Framework.  Data Citation: in preparation  Data Access: in preparation

5 Future: EDMC Work Plan for FY 2013  Finalize SAB Actions (due Q2)  Respond to SAB follow-up, if any  Perform EDM Assessment of Observing Systems of Record  Continue monitoring and implementation of approved PDs  Begin assigning persistent identifiers (DOIs) to datasets  Finish and issue Data Citation PD  Finish and issue Data Access PD  Implement prototype EDM Dashboard  Host the 2013 EDM Conference  Other:  Review old NAO “Ocean Data Acquisition” (1990)  Support ITRB review of DM Plans as needed  Augment liaison to EA Cmtee, GIS Cmtee, OSC

6 Response to SAB Action: External Data Usage Best Practice Jeff de La Beaujardière, PhD NOAA Data Management Architect

7 SAB Action 1: Develop policy on use of external data o Action: Respond to SAB report which states, "A timely NOAA policy for the use of external data could improve NOAA’s data activities and serve as a model for wider collaborative adoption by partners." o Status:  Started from DAARWG White Paper (2011)  Draft version 0.1 sent to EDMC in Aug 2012  53 comments received  Draft v0.2 circulated to EDMC and original reviewers in Sept 31 new comments  Draft v0.3 reviewed by NOSC & CIO Council in Oct Minor comments  Next step: NOAA Executive Council (NEC) review  Final version to be presented to SAB in Mar 2013

8 External Data Usage Best Practice Summary o Rationale  If NOAA uses external data that are inaccurate, unreliable, or not within NOAA's legal right to use, then NOAA’s credibility or reliability may be damaged o Scope  Use of environmental or socio-economic data from non-NOAA sources  …especially if: NOAA imprimatur applied May affect life, property, or highly influential scientific assessments o Best Practice  Answer questions in Worksheet  …to satisfaction of responsible NOAA official

9 External Data Usage Best Practice Worksheet o Worksheet: series of questions regarding  Purpose  Quality  Reliability  Terms of Use  Life-Cycle Cost  IT Systems & Security  Metadata  Accessibility  Archiving  Formal Agreements

10 Response to SAB Action: Environmental Data Management Framework Jeff de La Beaujardière, PhD NOAA Data Management Architect

11 SAB Action and Status - NOAA DM Framework o Action: Respond to SAB report regarding the "urgent need to establish a NOAA-wide Environmental Data Management Framework"  An EDM Framework means "the organization and governance structure (i.e., roles and responsibilities, policies, procedures), data management principles and practices, technical standards, etc. needed to manage the life-cycle of environmental data across the NOAA enterprise." o Status:  Draft v0.1 circulated for comment to EDMC in Sept.  86 comments received  Draft v0.2 now cmtee level: EDMC, OSC, EAC, GISC  Will circulate v0.3 to NOSC & CIO Council

12 Data Management Framework Data Lifecycle NOAA Environmental Data Management Framework Principles Governance Standards Architecture Assessment Resources o Purpose: To organize, guide and support NOAA environmental data management activities. o Mandate: SAB recommendation to NOAA.

13 Data Management Framework Data Lifecycle Data Management Framework Principles Governance Standards Architecture Assessment Resources Principles o Full and Open Access  Except in special cases o Data Preservation o Information Quality o Ease of Use

14 Data Management Framework Data Lifecycle Data Management Framework Principles Governance Standards Architecture Assessment Resources Governance o Agency bodies  EDMC, CIO Council, NOSC, SAB, DAARWG, EA, OSC, GIS o NOAA policies & docs  NGSP, NAO , EDMC PDs, etc. o National policies & docs o External coordination o Enforcement

15 CIO Council Chief Information Officer Council CIO Council Chief Information Officer Council NOSC NOAA Observing System Council NOSC NOAA Observing System Council DMIT Data Management Integration Team DMIT Data Management Integration Team GIS Committee GIS Committee Enterprise Architecture Committee Enterprise Architecture Committee DAARWG Data Access & Archiving Requirements WG DAARWG Data Access & Archiving Requirements WG SAB Science Advisory Board SAB Science Advisory Board EDMC Environmental Data Management Committee EDMC Environmental Data Management Committee Observing Systems Committee Observing Systems Committee NOAA DM Governance NEC & NEP NOAA Executive Council & Panel NEC & NEP NOAA Executive Council & Panel

16 EDMC Procedural Directives Archive Procedure What to archive, how to submit to archive. Data Access What on-line services to use so your data can be obtained. Data Citation Use unique identifiers to allow data to be referenced and tracked. Data Sharing by NOAA Grantees State in proposal how you will share data, and share within 2 years. Data Documentation How to apply ISO metadata for discovery, use & understanding. Data Management Planning PD Plan, in advance, how you will preserve, document and distribute your data.

17 Data Management Framework Data Lifecycle Data Management Framework Principles Governance Standards Architecture Assessment Resources Personnel Training, Recognition Budget Project-specific, NOAA-wide Other Resources Facilities, Teams, Software, etc.

18 Data Management Framework Data Lifecycle Data Management Framework Principles Governance Standards Architecture Assessment Resources Standards Services Formats Metadata Vocabularies Data Quality IT Security

19 Data Management Framework Data Lifecycle Data Management Framework Principles Governance Standards Architecture Assessment Resources Architecture Service-based approach Distributed systems approach Infrastructure Observing systems Ground systems Archive centers Legacy systems Cloud computing

20 Service-based, distributed systems approach 20 Visualization Service Thematic Portal Transformation Service internet Observing System or Forecast Model access service Data + Metadata Obs Syst. or Model data provider access service Data + Metadata Community Catalog discovery service Obs Syst. or Model access service Data + Metadata

21 Producers Data Management Services Data Access Services Data Management Architecture - Services Viewpoint Observing Systems RequestSubscriptionAlert Discovery Services Service RegistryData Catalog Utility Services Mapping & VisualizationFormat Conversion Coordinate TransformationProduct Generation Consumers Models Scientific Software Archives Web Browsers Workflows Models External Catalogs Web PortalsData Collection & Processing CentersClient Libraries Archives Dedicated Links Sampling Decision Support Tools GIS Surveys

22 Public Cloud Cloud Deployment Concepts Master copy of Data NOAA security boundary One-way push Access services Discovery services Public users Government Cloud Processing Services NOAA Internal customers

23 Data Management Framework Data Lifecycle Data Management Framework Principles Governance Standards Architecture Assessment Resources Assessment Current state Observing System DM assessment (planned) Progress measurements EDMC Reporting DM Dashboard (planned) Feedback from users & implementers (desired)

24 Observing System of Record EDM Assessment o 100+ observing systems providing input to key NOAA products o For each system (or each dataset):  Archive location?  Metadata completeness & format?  Data access service & format?  Dataset identifier(s) assigned?

25 Mockup: DM Dashboard Last Update: yyyy-mm-dd Accessibility Documentation % Records with Data Access Service LinkData Access Service Types Offered Metadata Dialects UsedMetadata Completeness Scores # WMSNoneWCSDAPEsriWFS % of Records Free Text ISOFGDCOBISDC None Historic Trend T %

26 DM Dashboard Approach Metadata Records WAF (Web-Accessible Folder) NGDC Metadata Analysis Software Analysis Results Database Summary Report and Graph Generator DM Dashboard Detailed HTML Reports Exists Does not exist Other sources of info

27 Data Management Framework Data Lifecycle Data Management Framework Principles Governance Standards Architecture Assessment Resources

28 Data Lifecycle Planning and Production Activities Data Management Activities Usage Activities Data Lifecycle Overview

29 Data Lifecycle Usage Activities Data Management Activities Planning and Production Activities Data Lifecycle Activities Collection Processing Quality Control Documentation Cataloging Dissemination Preservation Stewardship Usage Tracking Final Disposition Requirements Definition Planning Development Deployment Operations Discovery Reception Understanding Analysis New Product Generation User Feedback Citation Tagging Gap Assessment

30 Applicability of EDMC Directives Data Lifecycle Usage Activities Data Management Activities Planning and Production Activities Collection Processing Quality Control Documentation Cataloging Dissemination Preservation Stewardship Usage Tracking Final Disposition Requirements Definition Planning Development Deployment Operations Data Documentation DM Planning Data Sharing by Grantees Archive Procedure Data Citation Data Access Discovery Reception Understanding Analysis New Product Generation User Feedback Citation Tagging Gap Assessment External Data Usage BP

31 Backup Slides

32 Data User Archive ConOps Data Management Concept of Operations DM Plan* Data Documentation* (Metadata) Archive Procedure* Data Access Service* OAIS Reference Model Dashboard Catalog Service ID* Result product forecast paper decision policy response ID generate write perform preserve guide bind publish understand get find register measure create Data Producer cite *Subject of EDMC Procedural Directive publish Archive (Not all activities illustrated) NOAA Leadership assess Tools measure generate Requirements refine guide use

33 NOAA Data Discovery Vision 33 NOAA data cataloging Find NOAA data by: Location Time Phenomenon Other characteristics without knowing NOAA organization Enforcement Marine Mammals ROV/UAVNOAA LabsSurveys Data Sources SatelliteRadarBuoyShipSonarFishGaugeChartModel Client Tools Desktop GIS Scientific Software GeoPlatform Map Viewer GEO data.gov GCMD External Catalogs Data Access Services Local Catalogs & Metadata Folders NCDCNODCIOOSUAFNGDC others...

34 Data Lifecycle Usage Activities Data Management Activities Planning and Production Activities Collection Processing Quality Control Documentation Cataloging Dissemination Preservation Stewardship Usage Tracking Final Disposition Discovery Reception Understanding Analysis New Product Generation User Feedback Citation Tagging Gap Assessment Requirements Definition Planning Development Deployment Operations Feedback and Support Interactions Data Lifecycle Activities

35 Primary NOAA Data Catalog(s) metadata record Tag Database metadata record metadata record metadata record metadata record metadata record metadata record metadata record DWH data.gov GEOSS CORE GEOSS StP next crisis and the next DWH Response data.gov GEOSS Data CORE Topical Catalogs or Portals the next crisis Tags are not inserted into metadata records by data providers. Instead, the Catalog adds tags to indicate datasets relevant to a particular purpose. Datasets with a relevant tag are recorded by external catalogs. Tagging Concept