1 NOAA Environmental Data Management Committee (EDMC) Report to DAARWG Jeff de La Beaujardière, PhD EDMC Chair +1 301-713-7175.

Slides:



Advertisements
Similar presentations
Archive Requirements Working Group A NOAA clearinghouse for requirement planning in support of the science objectives related to archive, access, and reprocessing.
Advertisements

Portfolio Management, according to Office of Management and Budget (OMB) Circular A-16 Supplemental Guidance, is the coordination of Federal geospatial.
Geographic Information United Nations Geographic Information Working Group Our Vision It’s A major natural disaster strikes. UN relief agencies,
Peter Griffith and Megan McGroddy 4 th NACP All Investigators Meeting February 3, 2013 Expectations and Opportunities for NACP Investigators to Share and.
Integrated Ocean Observing System (IOOS) Data Management and Communication (DMAC) Standards Process Julie Bosch NOAA National Coastal Data Development.
AMS Summer Community Meeting Boulder, Colorado – August 2008 The Emerging Interface among the Private, Federal & Academic Sectors in the Archive.
CENDI/NFAIS Quality Workshop: The Importance of Quality and Integrity Kevin Kirby, Enterprise Data Architect US Environmental Protection Agency Office.
NOAA Documentation Improvement Ted Habermann How do we measure and visualize improvements in NOAA Documentation? Record Count Completeness (Rubric Scores)
Open Library Environment Designing technology for the way libraries really work November 19, 2008 ~ ASERL, Atlanta Lynne O’Brien Director, Academic Technology.
NOAA Climate Program – An Update NOAA Science Advisory Board March 19, 2003 NOAA Science Advisory Board March 19, 2003 Mary M. Glackin NOAA Assistant Administrator.
The topics addressed in this briefing include:
Marine Recreational Information Program Data Management Standard (MDMS) NOAA Environment Data Management Conference Data Management Planning Workshop June.
NOAA Metadata Update Ted Habermann. NOAA EDMC Documentation Directive This Procedural Directive establishes 1) a metadata content standard (International.
NOAA Data Management Activities Deirdre Jones, EDMC Chair Jeff de La Beaujardière, DM Architect Prepared for DAARWG
1 Open Library Environment Designing technology for the way libraries really work December 8, 2008 ~ CNI, Washington DC Lynne O’Brien Director, Academic.
1 Semantic Cloud Computing & Open Linked Data Pattern Brand Niemann Invited Expert to the NCIOC SCOPE and Services WGs September 22, 2009.
Metadata (for the data users downstream) RFC GIS Workshop July 2007 NOAA/NESDIS/NGDC Documentation.
FCM Quality of Life Reporting System Metadata By: Acacia Consulting and Research June 2002.
Data Interoperability and Access Activities Prepared for the Data Archiving and Access Requirements Working Group (DAARWG) Ken McDonald, TPIO/GEO-IDE Jeff.
The Preparatory Phase Proposal a first draft to be discussed.
Agenda: DMWG SM policy status ESIP meeting recap Reminder - DM Webinar Series New and updated web pages on DM website Metadata Training Sessions CDI meeting.
Information Architecture MMR Briefing 16 January 2014 Presenter: Dan Whitcher.
ID-01 report 17 September 2014 IDIB meeting, Enschede Michel Schouppe
1 Implementing Portal Technology To Support Data/Information Management - Portal Prototype Dashboard for Fisheries Information System (FIS) November 3,
IODE Ocean Data Portal - technological framework of new IODE system Dr. Sergey Belov, et al. Partnership Centre for the IODE Ocean Data Portal MINCyT,
M u l t I b e a m III W o r k s h o p M u l t I b e a m III W o r k s h o p National Geophysical Data Center / World Data Centers NOAA Slide 1 End-to-End.
NOAA Administrative Order : Management of Environmental and Geospatial Data and Information Jeff Arnfield NOAA’s National Climatic Data Center Version.
HIC Meeting July 30, NWS Climate Services Division and NWS Hydrologists in the Field Summer Hydrologist-in-Charge Meeting July 30, 2008 Ahsha Tribble,
1 NOAA Use of the Open Archival Information System Reference Model (OAIS-RM) Ken McDonald NOAA NESDIS ESIP Federation Meeting July 9, 2009.
U.S. Department of the Interior U.S. Geological Survey USGS Geospatial Liaisons - Supporting NHD Stewardship NHD Workshop April 14, 2009.
FEA DRM Management Strategy Presented by : Mary McCaffery, US EPA.
WGISS Working Group on Information Systems and Services Richard MORENO CNES WGISS report, Agenda Item 14 Tromsø, Norway October 2014.
Shaping a Health Statistics Vision for the 21 st Century 2002 NCHS Data Users Conference 16 July 2002 Daniel J. Friedman, PhD Massachusetts Department.
DRAFT EDMC Procedural Directives NOAA Environmental Data Management Committee 12/3/2015 1
Breakout # 1 – Data Collecting and Making It Available Data definition “ Any information that [environmental] researchers need to accomplish their tasks”
NOAA Data Citation Procedural Directive 8 November 2012 DAARWG.
Archiving microdata Standards and good practices United Nations Statistics Commission New York, February 26, 2009 Olivier Dupriez World Bank, Development.
E-Government Initiative Geospatial Information One-Stop FGDC Coordination Group December 4, 2001 John Moeller.
1 Physical and Social Sciences Research Task Team: Final Report Alexander E. MacDonald NOAA Earth System Research Laboratory July 25, 2006 A Presentation.
Evaluate Phase Pertemuan Matakuliah: A0774/Information Technology Capital Budgeting Tahun: 2009.
Fire Emissions Network Sept. 4, 2002 A white paper for the development of a NSF Digital Government Program proposal Stefan Falke Washington University.
Task XX-0X Task ID-01 GEO Work Plan Symposium April 2014 Task ID-01 “ Advancing GEOSS Data Sharing Principles” Experiences related to data sharing.
Government and Industry IT: one vision, one community Vice Chairs April Meeting Agenda Welcome and Introductions GAPs welcome meeting with ACT Board (John.
NSDI Strategic Plan Update National Geospatial Advisory Committee Meeting December 11, 2013.
The Data Sharing Working Group 24 th meeting of the GEO Executive Committee Geneva, Switzerland March 2012 Report of the Data Sharing Working Group.
The Proliferation of Metadata Standards and the Evolution of NASA’s Global Change Master Directory (GCMD) Standard for Uses in Earth Science Data Discovery.
19 October 2004Enterprise Architecture in WSRP Portal 1 Foreword: Building Enterprise Architecture Through WSRP in Sample EPA Regional Portal FEA Goals:
Presented by Eliot Christian, USGS Accessibility, usability, and preservation of government information (Section 207 of the E-Government Act) April 28,
Providing access to your data: Determining your audience Robert R. Downs, PhD NASA Socioeconomic Data and Applications Center (SEDAC) Center for International.
Summary of June 2013 Meetings: National Geospatial Advisory Committee FGDC Steering Committee FGDC Coordination Group Meeting June 18, 2013.
Application of NASA ESE Data and Tools to Particulate Air Quality Management A proposal to NASA Earth Science REASoN Solicitation CAN-02-OES-01 REASoN:
ISWG / SIF / GEOSS OOSSIW - November, 2008 GEOSS “Interoperability” Steven F. Browdy (ISWG, SIF, SCC)
Chang, Wen-Hsi Division Director National Archives Administration, 2011/3/18/16:15-17: TELDAP International Conference.
ISWG / SIF / GEOSS OOS - August, 2008 GEOSS Interoperability Steven F. Browdy (ISWG, SIF, SCC)
EDMC Archive Architecture Team (AAT) Prepared for the Data Archiving and Access Requirements Working Group (DAARWG) Ken McDonald December 9, 2010.
International Planetary Data Alliance Registry Project Update September 16, 2011.
TeraGrid’s Process for Meeting User Needs. Jay Boisseau, Texas Advanced Computing Center Dennis Gannon, Indiana University Ralph Roskies, University of.
November 2, 2009 NOAA Climate Services Portal Prototype A collaborative, NOAA-wide prototyping effort featuring CPC, CPO, CSC, and NCDC A collaborative,
R2R ↔ NODC Steve Rutz NODC Observing Systems Team Leader May 12, 2011 Presented by L. Pikula, IODE OceanTeacher Course Data Management for Information.
IODE Ocean Data Portal - technological framework of new IODE system Dr. Sergey Belov, et al. Partnership Centre for the IODE Ocean Data Portal.
Capacity Building in: GEO Strategic Plan 2016 – 2025 and Work Programme 2016 Andiswa Mlisa GEO Secretariat Workshop on Capacity Building and Developing.
IPDA Registry Definitions Project Dan Crichton Pedro Osuna Alain Sarkissian.
Sample Fit-Gap Kick-off
Adapted from Presentations by:
NOAA Data Management Perspective & Plans NSF RDMI Workshop
Workplan for Updating the As-built Architecture of the 2007 GEOSS Architecture Implementation Pilot Session 7B, 6 June 2007 GEOSS Architecture Implementation.
Agency Requirements: NOAA Administrative Order Management of environmental and geospatial data and information This training module is part of.
WGISS Connected Data Assets Oct 24, 2018 Yonsook Enloe
A Case Study for Synergistically Implementing the Management of Open Data Robert R. Downs NASA Socioeconomic Data and Applications.
Fundamental Science Practices (FSP) of the U.S. Geological Survey
Presentation transcript:

1 NOAA Environmental Data Management Committee (EDMC) Report to DAARWG Jeff de La Beaujardière, PhD EDMC Chair

2 Discoverable Accessible Documented Preserved Vision All NOAA data will be: for all types of users and applications.

3 Past: EDMC Milestones o 4 years ago:  Predecessor committee in existence (DMC)  Focus on data preservation. Archive Procedure issued  No comprehensive framework for Data Mgmt  Reporting to CIO Council only o 2 years ago:  NOAA Admin. Order (NAO) issued (2010 Nov 04)  EDMC established, reporting to CIO Council & NOAA Obs Syst Council o 1 year ago:  3 procedural directives issued (2011 Oct) Data Management Planning Data Documentation Data Sharing by Grantees  Active implementation & monitoring begins o 7 months ago:  Jeff DLB becomes EDMC chair  DM Dashboard concept developed  NOAA EDM Framework begun

4 Present: Status of EDMC Directives  Data Management Planning: 6 plans submitted to DM Plan Repository – Positive feedback regarding usefulness of DM Plan template Working on metrics for scoring plan completeness EDMC prepared to assist ITRB/CIO in reviewing DM Plans  Data Sharing by NOAA Grantees: Grants Mgmnt Div now requiring data sharing language on new FFOs – Recent: Joint Ctr for Satellite Data Assim, IOOS Regional Assoc.  Data Documentation: EDMC directive mentioned favorably in GAO Geospatial Info Report NGDC software analyzing metadata. Gradual improvement seen. NCDDC/NODC metadata training sessions well attended.  Archive Procedure: Issued 2008 but not actively promoted; now including in EDM Framework.  Data Citation: in preparation  Data Access: in preparation

5 Future: EDMC Work Plan for FY 2013  Finalize SAB Actions (due Q2)  Respond to SAB follow-up, if any  Perform EDM Assessment of Observing Systems of Record  Continue monitoring and implementation of approved PDs  Begin assigning persistent identifiers (DOIs) to datasets  Finish and issue Data Citation PD  Finish and issue Data Access PD  Implement prototype EDM Dashboard  Host the 2013 EDM Conference  Other:  Review old NAO “Ocean Data Acquisition” (1990)  Support ITRB review of DM Plans as needed  Augment liaison to EA Cmtee, GIS Cmtee, OSC

6 Response to SAB Action: External Data Usage Best Practice Jeff de La Beaujardière, PhD NOAA Data Management Architect

7 SAB Action 1: Develop policy on use of external data o Action: Respond to SAB report which states, "A timely NOAA policy for the use of external data could improve NOAA’s data activities and serve as a model for wider collaborative adoption by partners." o Status:  Started from DAARWG White Paper (2011)  Draft version 0.1 sent to EDMC in Aug 2012  53 comments received  Draft v0.2 circulated to EDMC and original reviewers in Sept 31 new comments  Draft v0.3 reviewed by NOSC & CIO Council in Oct Minor comments  Next step: NOAA Executive Council (NEC) review  Final version to be presented to SAB in Mar 2013

8 External Data Usage Best Practice Summary o Rationale  If NOAA uses external data that are inaccurate, unreliable, or not within NOAA's legal right to use, then NOAA’s credibility or reliability may be damaged o Scope  Use of environmental or socio-economic data from non-NOAA sources  …especially if: NOAA imprimatur applied May affect life, property, or highly influential scientific assessments o Best Practice  Answer questions in Worksheet  …to satisfaction of responsible NOAA official

9 External Data Usage Best Practice Worksheet o Worksheet: series of questions regarding  Purpose  Quality  Reliability  Terms of Use  Life-Cycle Cost  IT Systems & Security  Metadata  Accessibility  Archiving  Formal Agreements

10 Response to SAB Action: Environmental Data Management Framework Jeff de La Beaujardière, PhD NOAA Data Management Architect

11 SAB Action and Status - NOAA DM Framework o Action: Respond to SAB report regarding the "urgent need to establish a NOAA-wide Environmental Data Management Framework"  An EDM Framework means "the organization and governance structure (i.e., roles and responsibilities, policies, procedures), data management principles and practices, technical standards, etc. needed to manage the life-cycle of environmental data across the NOAA enterprise." o Status:  Draft v0.1 circulated for comment to EDMC in Sept.  86 comments received  Draft v0.2 now cmtee level: EDMC, OSC, EAC, GISC  Will circulate v0.3 to NOSC & CIO Council

12 Data Management Framework Data Lifecycle NOAA Environmental Data Management Framework Principles Governance Standards Architecture Assessment Resources o Purpose: To organize, guide and support NOAA environmental data management activities. o Mandate: SAB recommendation to NOAA.

13 Data Management Framework Data Lifecycle Data Management Framework Principles Governance Standards Architecture Assessment Resources Principles o Full and Open Access  Except in special cases o Data Preservation o Information Quality o Ease of Use

14 Data Management Framework Data Lifecycle Data Management Framework Principles Governance Standards Architecture Assessment Resources Governance o Agency bodies  EDMC, CIO Council, NOSC, SAB, DAARWG, EA, OSC, GIS o NOAA policies & docs  NGSP, NAO , EDMC PDs, etc. o National policies & docs o External coordination o Enforcement

15 CIO Council Chief Information Officer Council CIO Council Chief Information Officer Council NOSC NOAA Observing System Council NOSC NOAA Observing System Council DMIT Data Management Integration Team DMIT Data Management Integration Team GIS Committee GIS Committee Enterprise Architecture Committee Enterprise Architecture Committee DAARWG Data Access & Archiving Requirements WG DAARWG Data Access & Archiving Requirements WG SAB Science Advisory Board SAB Science Advisory Board EDMC Environmental Data Management Committee EDMC Environmental Data Management Committee Observing Systems Committee Observing Systems Committee NOAA DM Governance NEC & NEP NOAA Executive Council & Panel NEC & NEP NOAA Executive Council & Panel

16 EDMC Procedural Directives Archive Procedure What to archive, how to submit to archive. Data Access What on-line services to use so your data can be obtained. Data Citation Use unique identifiers to allow data to be referenced and tracked. Data Sharing by NOAA Grantees State in proposal how you will share data, and share within 2 years. Data Documentation How to apply ISO metadata for discovery, use & understanding. Data Management Planning PD Plan, in advance, how you will preserve, document and distribute your data.

17 Data Management Framework Data Lifecycle Data Management Framework Principles Governance Standards Architecture Assessment Resources Personnel Training, Recognition Budget Project-specific, NOAA-wide Other Resources Facilities, Teams, Software, etc.

18 Data Management Framework Data Lifecycle Data Management Framework Principles Governance Standards Architecture Assessment Resources Standards Services Formats Metadata Vocabularies Data Quality IT Security

19 Data Management Framework Data Lifecycle Data Management Framework Principles Governance Standards Architecture Assessment Resources Architecture Service-based approach Distributed systems approach Infrastructure Observing systems Ground systems Archive centers Legacy systems Cloud computing

20 Service-based, distributed systems approach 20 Visualization Service Thematic Portal Transformation Service internet Observing System or Forecast Model access service Data + Metadata Obs Syst. or Model data provider access service Data + Metadata Community Catalog discovery service Obs Syst. or Model access service Data + Metadata

21 Producers Data Management Services Data Access Services Data Management Architecture - Services Viewpoint Observing Systems RequestSubscriptionAlert Discovery Services Service RegistryData Catalog Utility Services Mapping & VisualizationFormat Conversion Coordinate TransformationProduct Generation Consumers Models Scientific Software Archives Web Browsers Workflows Models External Catalogs Web PortalsData Collection & Processing CentersClient Libraries Archives Dedicated Links Sampling Decision Support Tools GIS Surveys

22 Public Cloud Cloud Deployment Concepts Master copy of Data NOAA security boundary One-way push Access services Discovery services Public users Government Cloud Processing Services NOAA Internal customers

23 Data Management Framework Data Lifecycle Data Management Framework Principles Governance Standards Architecture Assessment Resources Assessment Current state Observing System DM assessment (planned) Progress measurements EDMC Reporting DM Dashboard (planned) Feedback from users & implementers (desired)

24 Observing System of Record EDM Assessment o 100+ observing systems providing input to key NOAA products o For each system (or each dataset):  Archive location?  Metadata completeness & format?  Data access service & format?  Dataset identifier(s) assigned?

25 Mockup: DM Dashboard Last Update: yyyy-mm-dd Accessibility Documentation % Records with Data Access Service LinkData Access Service Types Offered Metadata Dialects UsedMetadata Completeness Scores # WMSNoneWCSDAPEsriWFS % of Records Free Text ISOFGDCOBISDC None Historic Trend T %

26 DM Dashboard Approach Metadata Records WAF (Web-Accessible Folder) NGDC Metadata Analysis Software Analysis Results Database Summary Report and Graph Generator DM Dashboard Detailed HTML Reports Exists Does not exist Other sources of info

27 Data Management Framework Data Lifecycle Data Management Framework Principles Governance Standards Architecture Assessment Resources

28 Data Lifecycle Planning and Production Activities Data Management Activities Usage Activities Data Lifecycle Overview

29 Data Lifecycle Usage Activities Data Management Activities Planning and Production Activities Data Lifecycle Activities Collection Processing Quality Control Documentation Cataloging Dissemination Preservation Stewardship Usage Tracking Final Disposition Requirements Definition Planning Development Deployment Operations Discovery Reception Understanding Analysis New Product Generation User Feedback Citation Tagging Gap Assessment

30 Applicability of EDMC Directives Data Lifecycle Usage Activities Data Management Activities Planning and Production Activities Collection Processing Quality Control Documentation Cataloging Dissemination Preservation Stewardship Usage Tracking Final Disposition Requirements Definition Planning Development Deployment Operations Data Documentation DM Planning Data Sharing by Grantees Archive Procedure Data Citation Data Access Discovery Reception Understanding Analysis New Product Generation User Feedback Citation Tagging Gap Assessment External Data Usage BP

31 Backup Slides

32 Data User Archive ConOps Data Management Concept of Operations DM Plan* Data Documentation* (Metadata) Archive Procedure* Data Access Service* OAIS Reference Model Dashboard Catalog Service ID* Result product forecast paper decision policy response ID generate write perform preserve guide bind publish understand get find register measure create Data Producer cite *Subject of EDMC Procedural Directive publish Archive (Not all activities illustrated) NOAA Leadership assess Tools measure generate Requirements refine guide use

33 NOAA Data Discovery Vision 33 NOAA data cataloging Find NOAA data by: Location Time Phenomenon Other characteristics without knowing NOAA organization Enforcement Marine Mammals ROV/UAVNOAA LabsSurveys Data Sources SatelliteRadarBuoyShipSonarFishGaugeChartModel Client Tools Desktop GIS Scientific Software GeoPlatform Map Viewer GEO data.gov GCMD External Catalogs Data Access Services Local Catalogs & Metadata Folders NCDCNODCIOOSUAFNGDC others...

34 Data Lifecycle Usage Activities Data Management Activities Planning and Production Activities Collection Processing Quality Control Documentation Cataloging Dissemination Preservation Stewardship Usage Tracking Final Disposition Discovery Reception Understanding Analysis New Product Generation User Feedback Citation Tagging Gap Assessment Requirements Definition Planning Development Deployment Operations Feedback and Support Interactions Data Lifecycle Activities

35 Primary NOAA Data Catalog(s) metadata record Tag Database metadata record metadata record metadata record metadata record metadata record metadata record metadata record DWH data.gov GEOSS CORE GEOSS StP next crisis and the next DWH Response data.gov GEOSS Data CORE Topical Catalogs or Portals the next crisis Tags are not inserted into metadata records by data providers. Instead, the Catalog adds tags to indicate datasets relevant to a particular purpose. Datasets with a relevant tag are recorded by external catalogs. Tagging Concept