Toward a Distributed and Collaborative Framework for Preservation Martin Halbert, UNT Dean of Libraries David Minor, Chronopolis Program Manager Katherine.

Slides:



Advertisements
Similar presentations
Moving Forward With Digital Preservation at the Library of Congress Laura Campbell Associate Librarian for Strategic Initiatives Library of Congress.
Advertisements

A centre of expertise in data curation and preservation DigCCur2007 Symposium, Chapel Hill, N.C., April 18-20, 2007 Co-operation for digital preservation.
Katherine Skinner Executive Director, Educopia Institute Program Manager, MetaArchive Cooperative An Age of Discovery, ARL-CNI Washington D.C. Friday,
The National Digital Stewardship Alliance: Community, Content, Commitment.
Mairéad Martin, Penn State University Commons Solutions Group Storage Workshop May 2010.
Background Chronopolis Goals Data Grid supporting a Long-term Preservation Service Data Migration Data Migration to next generation technologies Trust.
PREMIS in Thought: Data Center for LC Digital Holdings Ardys Kozbial, Arwen Hutt, David Minor February 11, 2008.
Sustainable Preservation Services for Archivists through Distributed Custody Caryn Wojcik State of Michigan Records Management Services.
DCAPE Project Update Richard MarcianoChien-Yi Hou Caryn Wojcik University of University of State of Michigan North Carolina North Carolina Records Management.
Chronopolis: Preserving Our Digital Heritage David Minor UC San Diego San Diego Supercomputer Center.
ADAPT An Approach to Digital Archiving and Preservation Technology Principal Investigator: Joseph JaJa Lead Programmers: Mike Smorul and Mike McGann Graduate.
Tools and Services for the Long Term Preservation and Access of Digital Archives Joseph JaJa, Mike Smorul, and Sangchul Song Institute for Advanced Computer.
Tools and Services for the Long Term Preservation and Access of Digital Archives Joseph JaJa, Mike Smorul, and Sangchul Song Institute for Advanced Computer.
National Digital Information Infrastructure and Preservation Program (NDIIPP) Building a Network of Preservation Partners CNI Spring Task Force Meeting.
Data-PASS Shared Catalog Micah Altman & Jonathan Crabtree 1 Micah Altman Harvard University Archival Director, Henry A. Murray Research Archive Associate.
Mid-Michigan Digital Practitioners, March 14, 2014 The National Digital Stewardship Alliance Agenda Mid-Michigan Digital Practitioners Meeting Abigail.
World Data Center for Human Interactions in the Environment Conducting a Self-Assessment of a Long-Term Archive for Interdisciplinary Scientific Data as.
Trends & Challenges in Digital Object Storage Infrastructure: Notes from the National Digital Stewardship Alliance (NDSA) Infrastructure Working Group.
Tyler Walters Dean, University Libraries and Professor Virginia Tech July 18, 2013 Collaboratively Preserving Our Digital Memory.
Jenn Riley Metadata Librarian Indiana University Digital Library Program.
Katherine Skinner, Executive Director, Educopia Institute Martin Halbert, Dean of Libraries, University of North Texas CNI 2010 Spring Forum, Baltimore.
Digital Preservation through Cooperation: LOCKSS Gail McMillan Digital Library and Archives, University Libraries Virginia Polytechnic Institute and State.
Growing the MetaArchive Cooperative: ETDs (electronic theses and dissertations) Gail McMillan Digital Library and Archives, Virginia Tech July 2008 NDIIPP.
Digital Preservation: Lessons learned through national action Digital Preservation Interoperability Framework Workshop April 2010.
MetaArchive Cooperative Membership Agreements Martin Halbert NDIIPP Partners Meeting Washington, D.C. Wednesday July 9, 2008.
NDIIPP The Next Phase Meg Williams Associate General Counsel The Library of Congress.
Richard MarcianoChien-Yi Hou Caryn Wojcik University of University of State of Michigan North Carolina North Carolina Records Management ServicesSALT DCAPE.
Katherine Skinner Educopia Institute and MetaArchive Cooperative Matt Schultz Educopia Institute and MetaArchive Cooperative NDIIPP Partners Meeting Arlington,
The National Digital Information Infrastructure and Preservation Program Annual Partners Meeting 2008 Since we met last year… Martha Anderson, Director.
Preserving ETDs: NDLTD & MetaArchive Collaboration Gail McMillan Digital Library and Archives, Virginia Tech Newcomers’ USETDA 2012.
Why Archiving and Preserving GIS Data Is Important Maps tell a compelling story of change over time. They document movement, progress, and change to the.
National Digital Information Infrastructure and Preservation Program (NDIIPP) CNI Project Briefing December 5, 2005.
November 2004 NDIIPP: Future Directions and Relevance to Other Countries Beth Dulabahn Office of Strategic Initiatives Library of Congress November 7,
1 Designing Storage Architecture for Digital Collections 2012.
Martin Halbert UNT Dean of Libraries MetaArchive President Monday, April 11, 2011 Newspaper Archive Summit University of Missouri Columbia, MO.
Preserving eScholarship and Digitized Special Collections Distributed Digital Preservation Bill Donovan
Digital Preservation MetaArchive Cooperative.  9:00-9:45 - Session 1: Digital Preservation Overview  9:45-11:00 - Session 2: Policy & Planning Overview.
The Canadian Information Network for Research in the Social Sciences and Humanities Tim Au Yeung and Mary Westell Libraries.
Interoperability within the Grid NDIIPP Partners Meeting Arlington, VA July 9, 2008 Interoperability within the Grid Robert H. McDonald Digital Preservation.
T HE M ETA A RCHIVE M ODEL : D ISTRIBUTED D IGITAL P RESERVATION N ETWORKS Dr. Martin Halbert VIVA/SCHEV LAC Meeting Christopher Newport University Trible.
Katherine Skinner, Executive Director, Educopia Institute ESOPI 2013 Chapel Hill, NC April 19, 2013.
Session 3.  Now you know WHY to make policies and WHAT they should contain…  But HOW do you implement policies?  And then HOW do you implement a program.
Growing the MetaArchive Cooperative ETDs Gail McMillan Digital Library and Archives, Virginia Tech July 2008 NDIIPP Partners Meeting.
Martin Halbert President, MetaArchive Cooperative DigCCurr 2009 Meeting Chapel Hill, NC Friday, April 3, 2009.
Dr. Martin Halbert Dr. Katherine Skinner Digital Preservation: What’s Now, What’s Next. Amigos Online Conference, August 12, 2011.
The Alabama Digital Preservation Network (ADPNet) Aaron Trehub Director of Library Technology Auburn University State Council of Higher Education for Virginia.
What is NDIIPP doing?. July 7 th, Web-At-Risk is opening its archives for public access, having captured nearly 6 TB of data—the entire CA State Government.
Persistent Digital Archives and Library System (PeDALS)
Dr. Katherine Skinner, Executive Director SILS CRADLE Seminar UNC-CH Manning Hall April 25, 2014 Using Collaborative Networks To Support Scholarly Communications.
NCSU Libraries 13 June 2006 JCDL 2006 NDIIPP Preservation Network: Progress, Problems, and Promise Jim Tuttle, Geospatial Data Librarian.
Katherine Skinner, Educopia Institute Emily Gore, Clemson University U.S. Workshop on Roadmap for Digital Preservation Interoperability Framework NIST,
Chronopolis – MetaArchive Improving and Strengthening Inter-Institutional Preservation.
Martin Halbert MetaArchive Cooperative Thursday, June 25, 2009 NDIIPP Annual Meeting Washington, D.C.
APPLYING OAIS TO DISTRIBUTED DIGITAL PRESERVATION Katherine Skinner, Eld Zierau IDCC Workshop, Amsterdam, January 14, 2013.
Distributed Digital Preservation Networks Across a Region, Across a State: Stretching LOCKSS Gail McMillan, Virginia Tech Martin Halbert, Emory Aaron Trehub,
Digital Preservation through Cooperation: LOCKSS Gail McMillan Digital Library and Archives, University Libraries Virginia Polytechnic Institute and State.
Custodians of Culture, Architects of Archives  Martin Halbert (Emory Univ., MetaArchive Cooperative) - Facilitator  Thib Guicherd ‐ Callin (Stanford.
Infrastructure Breakout What capacities should we build now to manage data and migrate it over the future generations of technologies, standards, formats,
The National Digital Stewardship Alliance: Stewardship, Collaboration, Inclusiveness, Exchange.
Katherine Skinner, Martin Halbert & Matt Schultz Educopia Institute and MetaArchive Cooperative NDSA Infrastructure Committee
A Shared Commitment to Digital Preservation and Access.
Digital Preservation MetaArchive Cooperative, Digital Preservation Policy Planning Workshop Boston College, Boston, MA October 26, 2010.
The National Digital Stewardship Alliance: Community, Content, Commitment.
Beyond Technology: Creating and Sustaining the MetaArchive Cooperative Joint Annual Meeting, Society of American Archivists & the Council of State Archivists.
Trustworthiness of Preservation Systems
Joseph JaJa, Mike Smorul, and Sangchul Song
National Digital Stewardship Alliance Web Archiving Survey Update
Gail McMillan Digital Library and Archives, Virginia Tech
CNI Project Briefing December 5, 2005
The MetaArchive Model: Distributed Digital Preservation Networks
Presentation transcript:

Toward a Distributed and Collaborative Framework for Preservation Martin Halbert, UNT Dean of Libraries David Minor, Chronopolis Program Manager Katherine Skinner, Educopia Institute Executive Director Wednesday, July 21, 2010 NDIIPP Partners Meeting 2010, Arlington, VA

1. Context of collaboration in the National Digital Stewardship Alliance 2. Field notes on organizational strategies and collaborative preservation models 3. Need to build on OAIS to create shared models/vocabulary for understanding inter- organizational content stewardship activities Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 2

 A collaborative effort among:  government agencies,  educational institutions,  non-profit organizations, and  business entities  to preserve a distributed national digital collection  for the benefit of citizens now and in the future. Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 3 Source: NDIIPP 2010 Partners Meeting Handout

 Collaborative relationships were core to NDIIPP and will be foundational to the NDSA  Yet, we have barely begun to understand the nature of collaborative digital preservation relationships, especially in a national context  If the NDSA is to succeed, we must begin to model, analyze, and understand such collaborative relationships more systematically Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 4

 Case studies (MetaArchive, Chronopolis)  What organizational strategies work, and what strategies don’t work?  What lessons can we learn from the successes thus far?  What innovations are still needed?  OAIS terminology is often used to describe DDP networks, even though OAIS section 6 interoperability language is limited Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 5

 LOCKSS-based  LOCKSS, CLOCKSS, MetaArchive, DataPASS, PeDALS, COPPUL, LOCKSS-KOPAL, Synergies, ADPNet  iRODS-based  Chronopolis, NARA-TPAP, SHAMAN  CDL microservices  DuraCloud  NetArchiveSuite Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 6

 Established in 2004 (support from NDIIPP and NHPRC), preserving content for 15 members  Uses LOCKSS software to provide peer-to-peer distributed digital preservation infrastructure  Sustainable organizational framework: Membership organization with a 501c3 host (Educopia)  254 TB network capacity (and growing)  Compliant as a Trustworthy Digital Repository (2009 TRAC audit available on our site) Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 7

1. Cultural memory organizations must continue to evolve to maintain their historical role as cultural stewards  Preservation of digital assets as corollary to preserving physical ones  Importance of building in house expertise and knowledge  Value contributed by curators, librarians, and archivists to the digital preservation field Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 8

2. Importance of catalyzing and capitalizing on cultural memory organizations’ proven preservation methodologies  Replication of content  Distribution of content  Partnering to keep costing affordable Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 9

Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010 Current Members Auburn University Boston College Clemson University Florida State University Folger Shakespeare Library Georgia Tech Indiana State University Library of Congress Penn State University PUC Rio de Janeiro Rice University University of Hull University of Louisville University of North Texas University of South Carolina Virginia Tech Current Affiliates NDLTD SDSC Chronopolis Slide 10

 Three node federated data grid at UCSD/SDSC, NCAR and UMIACS with capacity for up to 100 TB of data per node (300 TB total)  Using the Storage Resource Broker (SRB) for data management (moving to iRODS)  Using BagIt file packaging format and SRB tools to ingest and transfer data  Using Auditing Control Environment (ACE) for integrity checking Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010 Slide 11

Spring 2010 Data Providers: Inter-university Consortium of Political and Social Research – preservation copy of collections including 40 years of social science data and Census California Digital Library – political and government web crawls, Web-at-risk collection SIO Explorer – data from 50 years of research voyages NCSU Libraries -- state and local geospatial data Slide 12Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010

 Being conducted by CRL  Doing self-assessment section now  Finishing in early 2011  Really diving back into OAIS  Section 6 “Archive Interoperability” Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 13

 Replicated copies stored in geographically diverse locations have a better chance of survival  Can embed preservation infrastructure and knowledge in cultural memory organizations  Can enable multiple instances to be monitored separately (lessens human error and malicious behavior possibilities)  Emphasizes collaboration and trust Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 14

Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 15

 Technical Levels of Interaction Between OAIS Archives  Independent Archives  Cooperating Archives  Federated Archives  Archives with Shared Functional Areas  Management Issues with Federated Archives Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 16

Skinner, Halbert, & Minor - NDIIPP Partners Meeting

Skinner, Halbert, & Minor - NDIIPP Partners Meeting

Skinner, Halbert, & Minor - NDIIPP Partners Meeting

 “The above examples show that the OAIS model is consistent with federation to accomplish specific objectives.  However, it should also be considered that some of these objectives might be accomplished through voluntary action.  This is an important dimension in the association of systems, including archives, because it establishes the degree of autonomy for each system.  At the heart of the autonomy issue is the ease with which an association may be altered by one of the participants.” Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 20

1. No interactions and therefore no association (complete autonomy, no linkages) 2. Associations that maintain an association member’s autonomy (voluntary participation, members can withdraw at will without penalty, ex. Internet sites) 3. Associations that bind an association member by contract (“The amount of autonomy retained depends on how difficult it is to negotiate the changes. The difficulty may rise as more entities become a party to the contract.”) Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 21

22 Committed Content Custodians Communities of Practice and Information Exchange Services Capacity Building Roles in the Stewardship Network Source: “Since we met last year…” Plenary, Martha Anderson, National Digital Information Infrastructure and Preservation Program Annual Partners Meeting 2008 Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010

 With whom are agreements made?  What happens if a replication site drops out?  Tracking who has curatorial responsibility (is it transferred? To the network or to individual repositories)?  What data management is handled by the Producer and what is handled by the Repository?  When needed, which copy becomes the most appropriate Dissemination Information Package (DIP)? Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 23

 Repository relationships ▪ P2P, hub and spoke, data grid, other  Security ▪ How assess security of each repository? Of the network?  Preservation metadata ▪ Where has a copy lived? Where are others? Are all equal?  Copies, copies, copies ▪ How many copies are enough? What if they don’t match? What if content changes? Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 24

 Analysis / abstracted model for distributed digital preservation:  Peer-to-peer roles  Hub-and-spoke styled preservation relationships  Centrally orchestrated  Ingestion pathways  Contingent elements Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 25

 We need models that build on the OAIS, but focus on inter- organizational Distributed Digital Preservation alliances, systems, and strategies  Such models should abstract the functions and logical potential relationships between entities seeking to work together to further digital preservation aims  These models should inform collaborative efforts between different groups seeking to preserve digital information  They should also provide a common vocabulary for interoperable systems development Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 26

 A cluster of DDP organizations including MetaArchive, Chronopolis, and others are considering a collaborative project to develop OAIS-based DDP models and use cases  Are in conversations with LC and other agencies about hosting an initial planning meeting to study this issue  If you are interested, please contact us Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 27

 Layers/types of organizations to understand:  Varieties of Repositories  Varieties of Content Creators  Varieties of Networks  Types of interactions to understand:  Collection exchange (for preservation or access?)  Collection enhancements/remediation (metadata, content additions, other?) Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 28

Martin Halbert Katherine Skinner David Minor Skinner, Halbert, & Minor - NDIIPP Partners Meeting 2010Slide 29