Data Integrity Management Council F2F Washington, D.C. November 29-30, 2006

Slides:



Advertisements
Similar presentations
Cost Recovery Working Group NANC Report Billing & Collection Working Group (B&C WG) January 19, 2005 Co-Chairs: Rosemary Emmer, Nextel Jim Castagna, Verizon.
Advertisements

Cost Recovery Working Group Billing & Collection Working Group Chairs: Rosemary Emmer, Nextel Jim Castagna, Verizon November 4, 2005.
Identification and Disposition of Official University Records University of Texas at Arlington Records Management.
1 External Review Results November 30, 2011 Reta Beebe.
1 ANSI/EIA A STANDARD Earned Value Management Systems Proposed Changes Discussion January 31, 2007 NDIA Program Management Systems Committee.
FY12 Title I Program Review Preparation Title I Technical Assistance Session School Improvement Grant Programs October 6, 2011.
PDS MC April 2-3, Wash. D. C.1 PDS4 Data Model Working Group Status Report to the PDS Management Council April 2-3, 2008.
Placing Movies in the PDS Archive Engineering Node Pasadena, California Nov 30, 2006
Portal User Group Meeting September 12, Agenda 1.Welcome 2. Updates on the Following: 1.Migration Status 2.Template 3.Disaster Recovery Exercise.
1 April 2010 TX SET Timeline Project Conceptualization 11 weeks Market Requirements 12 weeks ERCOT Requirements 12 weeks Conceptual Design 6 weeks Detail.
System Design/Implementation and Support for Build 2 PDS Management Council Face-to-Face Mountain View, CA Nov 30 - Dec 1, 2011 Sean Hardman.
P l a n e t a r y D a t a S y s t e m 1 Data Integrity Compilation Data Integrity Compilation Presented by: Al Schultz Location: PDS Management Council.
Architecture of information systems Document managment system Peter Záhorák.
Physical Media WG Report Physical Media Working Group August 13, 2007.
Degree and Graduation Seminar Project Management Processes
Planning and Delivery of HND Programmes Janet Strain.
PDS 2010 Project Update Dan Crichton MC Face-to-Face Washington DC March 25-26, 2010.
InTeGrate Assessment David Steer, University of Akron Ellen Iverson, Carleton College May 2012.
IEEE DASC Co-Sponsorship of UPF Working Group Proposal prepared by: Stephen Bailey Chair, Accellera UPF Technical Subcommittee.
Update from the Data Integrity & Tracking WG Management Council F2F UCLA Los Angles, CA August 13-14, 2007
Developing Plans and Procedures
PDS4 Tool Development Strategy PDS Management Council Meeting November 18, 2014 Dan Crichton.
M u l t I b e a m III W o r k s h o p M u l t I b e a m III W o r k s h o p National Geophysical Data Center / World Data Centers NOAA Slide 1 End-to-End.
Label Design Tool Management Council F2F Washington, D.C. November 29-30, 2006
Data Standards and Build 3b Plans Steve Hughes MC Face-to-Face UCLA, Los Angeles, CA November 28-29, 2012.
Information Architecture WG: Report of the Fall 2010 Meeting October 29, 2010 Dan Crichton, Chair Steve Hughes (presenting) NASA/JPL.
NSSDCA Status Dave Williams PDS Management Council Berkeley, CA 19 November 2014.
Geometry Project Ed Guinness MC Face-to-Face Meeting UCLA November 28-29, 2012.
29 Nov 2006PDS MC NSSDC MOU history PDS-NSSDC MOU circa 1994 Reviewed in Jan 2003, June 2004, Oct 2005, Nov 2006 Add words to remove HQ changes Change.
OAIS Rathachai Chawuthai Information Management CSIM / AIT Issued document 1.0.
Validation Tool (VTool) PDS Management Council Meeting Washington, D.C. November 2006
Data Integrity Issues: How to Proceed? Engineering Node Elizabeth Rye August 3, 2006
Operations Report PDS Management Council Meeting Flagstaff, Arizona August 2006
Standing Committee on Improvement Implementation Anne Aikman-Scalese, SCI Chair | ICANN-52 | February 2015.
Development Report Engineering Node August 2006
Page © ASME 2015 Module B – Process B4.Initiating and Terminating Standards Projects Standards and Certification Training.
Cassini Archive Tracking System (CATS) Report Engineering Node November 30, 2006
San Diego Super Computing (SDSC) Testing Summary and Recommendations Physical Media Working Group March 2008.
PDS4 Mission Needs Assessment Reta Beebe Dan Crichton.
Physical Media WG Update November 30, Fall 2006 WG Status Initial technology plan Use Cases Decision Matrix Media Testing NMSU Website Physical.
NSSDC Deep Archive / PDS4/ CDs-DVDs/Hard-Copy Archive 28 August 2012 Dave Williams.
Planetary Data System (PDS) Tom Morgan November 24, 2014.
Section 4.9 Work Group Members Kris Hafner, Chair, Board Member Rob Kondziolka, MAC Chair Maury Galbraith, WIRAB Shelley Longmuir, Governance Committee.
Enhanced Wireless Funding through HB 361 Shawn S. Smith Interim Ohio Coordinator.
Tools Report Engineering Node March 2007
PDS4 Project Report PDS MC F2F UCLA Dan Crichton November 28,
PDS4 Project Report PDS MC F2F University of Maryland Dan Crichton March 27,
IEEE /r5 Submission November 2008 John Notor, Cadence Design Systems, Inc.Slide 1 IEEE IMT-Advanced Review Process Date:
First Tuna Data Workshop (TDW-1) October 2006, Noumea, New Caledonia Oceanic Fisheries Programme (OFP) Secretariat of the Pacific Community (SPC)
1. What is Demand Management? …aims to understand, anticipate, prioritize and influence customer demand for services. Demand Management 2.
CIRTL Data Sharing 10/9/15. New Opportunities The CIRTL Network Commons (CNC) offers the potential for easier access to data for evaluation purposes.
NASA PLANETARY DATA SYSTEM ARCHIVING IPDA Steering Committee Reta Beebe, Dan Crichton Paris, France July
Dr. Gerry Firmansyah CID Business Continuity and Disaster Recovery Planning for IT (W-XI)
GSICS Procedure for Product Acceptance (GPPA) WorkFlow GCC Presented by Fangfang Yu 12/03/
Tools Report Engineering Node August 2007
A Solution for Maintaining File Integrity within an Online Data Archive Dan Scholes PDS Geosciences Node Washington University 1.
8 th Address Policy SIG Report Takashi Arano Yong Wan Ju Kenny Huang Chair/co-Chairs of APNIC Address Policy SIG.
Moderator Training – unitised assessment
Stream 2: Technical research Achievements and future plans
Project Management Essentials Connecting PI to PM
Standards and Certification Training
Records Management & ARIMS
SPR&I Regional Training
Expedited Policy Development Process on the Temporary Specification for gTLD Registration Data Name of Presenter Event Name DD Month 2018.
MODULE B - PROCESS SUBMODULES B1. Organizational Structure
IEEE IMT-Advanced Review Process
TIBC Budget Formulation Improvement Project
IEEE IMT-Advanced Review Process
Ctclink executive leadership committee May 31, 2018
Special Ed. Administrator’s Academy, September 24, 2013
Presentation transcript:

Data Integrity Management Council F2F Washington, D.C. November 29-30,

2 Scope Management Council Action: –Crichton agreed to chair an MC working group on data integrity. New suggested starting with Level 4 requirements before dealing with any implementation issues. The DIWG should have recommendations before the Tech Session tackles the technical issues in data integrity at its proposed face-to-face meeting. The Tech Session can then decide where to go with the existing SCR on checksums, including sub-issues of specificity and process. Members –Dan Crichton, EN, chair –Mitch Gordon, Rings –Ed Guinness, Geosciences –Bill Harris, PPI –Steve Hughes, EN –Al Schultz, GSFC –Mark Showalter, Rings –Tom Stein, Geosciences

3 Scope Cont… A set of use cases and level 4 requirements were developed based on the Level 3 requirements and the end-to-end flow of data across PDS –PDS Data Integrity Use Cases –PDS Data Integrity Level 4 Requirements –Both were reviewed at the PDS Technical F2F on October Scope of use cases/requirements is on file corruption NOTE: Data accountability for files and collections is also a critical function necessary to verify the integrity of collections such as volumes. PDS currently plans to address integrity of collections as part of the end-to-end tracking subsystem engineering.

4 Summary of Use Cases 1. Data Delivery 2. Data Distribution 3. Transfer to Deep Archive 4. Data Node Termination 5. Archive Integrity 6. Media Migration 7. Recover Data from Deep Archive 8. Data Transfer between Nodes

5 Data Integrity L4 Requirements (review at Tech Session) L4.DI.1 - PDS will only accept data files that are not corrupted. (2.5.1 UC-1) L4.DI.2 - PDS will provide procedures for verifying whether a data submission has not been corrupted. (2.5.1, UC-1) L4.DI.3 - PDS will ensure that a PDS Data Node verifies their data submissions are not corrupted prior to submission to a PDS Node. (2.5.2, UC-4) L4.DI.4 - PDS will ensure that archival data to be transferred between PDS Discipline Nodes has not been corrupted prior to transfer. (2.5.2, UC-8) L4.DI.5 - PDS will verify that data submitted from a data provider have not been corrupted. (2.5.2, UC-1) L4.DI.6 - PDS will verify that data submitted from a PDS Data Node have not been corrupted. (2.5.2, UC-4) L4.DI.7 - PDS will verify that archival data transferred between PDS Discipline Nodes have not been corrupted. (2.5.2, UC-8) L4.DI.8 - PDS will notify a data provider, PDS Data Node, or PDS Discipline Node to resubmit data that have been corrupted during transfer. (2.5.3, UC-1, UC-4, UC-8) L4.DI.9 - PDS will ensure that a user receiving data from the PDS can verify that the data have been successfully transferred. (3.2.3, UC-2) L4.DI.10 - PDS will periodically verify that data holdings have not been corrupted based on a schedule determined by the PDS Management Council. (4.1.2, UC-5) L4.DI.11 - PDS will verify that data holdings are not corrupted prior to migrating to another medium. (4.1.3, UC-6) L4.DI.12 - PDS will ensure that data migrated from one medium to another have been successfully migrated and not corrupted during the transfer. (4.1.3, UC-6) L4.DI.13 - PDS will ensure that the archival data holdings, to be submitted to the deep archive (NSSDC) for preservation, are not corrupted. (4.1.5, UC-3) L4.DI.14 - PDS will request the deep archive (NSSDC) to verify that the archival data holdings submitted to the deep archive have been received intact. (4.1.5, UC-3) L4.DI.15 - PDS will ensure that the archival data holdings recovered from the deep archive (NSSDC) are not corrupted. (4.1.5, UC-7)

6 Related Additional Requirements The following requirements were captured during the Data Integrity functions, but were allocated to other requirements areas. –PDS will ensure that they have an accessible backup copy of their archive holdings. (4.1.4, 2.6.1, 2.6.2, UC-5) [Allocated to disaster recovery] –PDS will develop a plan for recovery of missing or corrupted files. (4.1.4, UC-5) [Allocated to disaster recovery] –PDS will ensure that all data holdings are accounted for. (2.2.2) [Allocated to tracking]

7 MC Request Policies/directives related to the following: –Archival Integrity for PDS Develop a specific policy which covers frequency of verification, data integrity of the bits, tracking of the files, and that the files can still be accessed. For example, “Each node is responsible for verifying the integrity of their archival holdings on an annual basis including verification that all files are accounted for, are not corrupted, and can be accessed irregardless of the medium in which they are stored” –Disaster recovery planning for PDS Develop a general set of requirements to guide PDS Develop a specific policy. For example, “Each node is responsible for defining and implementing a disaster recovery plan for their node which covers catastrophic loss of data and/or system functionality” NSSDC/Preservation Planning –Need to align NSSDC/PDS MOU with archival integrity policy –Need to ensure NSSDC is meeting federal requirements for preservation of data (PDS-4.1.5) The WG continue to do the following: –Recommend an implementation for data integrity to the tech staff and then the MC –Develop requirements for Disaster Recovery and Archive Tracking (incl the tracking system)

8 Timeline Present requirements to MC (Nov 2006) Working Group to recommend to MC (Nov 2006) –MC to develop policies for archival integrity –Revisit MOU w/ NSSDC (WG to provide requirements), preservation policy/planning –MC to develop Disaster Recovery Policy, Preservation Policy Data Integrity Engineering Process –Working Group to identify and evaluate options for meeting the requirements (Jan 2007) –Review by Tech Group (Feb 2007) –Recommend solution to MC (Mar 2007) –Develop Implementation Plan upon MC approval of solution (Apr 2007)