Digital information lasts forever or five years, whichever comes first. —Jeff Rothenberg

Digital information lasts forever, or five years, whichever comes first. —Jeff Rothenberg The average lifespan of info on the Web? 44 days. —Brewster Kahle, Internet Archive

The Digital Dark Age New generations of software and platforms make old data unreadable –“Migration” required every 10 years –Old data grows in value –Almost no one is working on the problem, because it lives in a slower time frame RESULT: catastrophic extinction of data

Moore’s WallMoore’s Law

©

T i m e s c a t t e r s e v e r y t h i n g.

T i m e s c a t t e r s e v e r y t h

T i m e s c a t t e r s e v e r y t h i n g.

Cuneiform writing circa 6th Century BCE

Hyakumanto Darani CE 1 million copies World’s oldest printed text

ROSETTA STONE Created BCE Found CE Deciphered CE

Library Of Congress

Digital Preservation: A Five-Layered Challenge Curatorial challenge: How to decide what to preserve? Technological challenge: How to deal with obsolescence of hardware and software? Intellectual property challenge: How to provide access while protecting rights of creators? Economic challenge: How to pay for it all? Organizational / institutional challenge: How to enable creation of a distributed system that works without the federal government in control?

Five-Stage Design Process for Phase One 1.Environmental Scan three convening sessions with about 50 leading experts on relevant issues 2.Scenario Development Workshop 30 leading experts explored different solution spaces and how they might be affected by external forces 3.Draft design of High-Level Technical Architecture 4.Pilot Projects Workshop Convening of stakeholders to explore possible experiments 5.Draft NDIIPP Plan with Recommended Path Forward for Iterative Learning

Preservation Architecture Design Principles The NDIIPP digital preservation architecture must: support relationships between institutions, allow questions of preservation to be handled separately from questions of access, be built modularly, using existing technology and efforts where possible, be able to be assembled over time, rather than needing to be built all at once, be upgradable in pieces, without disrupting the whole system, and be specified using broadly acceptable protocols Source: “Plan for the National Digital Information Infrastructure and Preservation Program” (p. 52)

Congressional Approval for Next Phases Source: New York Times, February 13, 2003 (p. E7)

Next Steps for Develop working prototype of the technical architecture 2.Continue to build the preservation network through additional workshops and other outreach 3.Recruit and select pilot projects, applying criteria agreed upon in plan 4.Create strong learning feedback loops between work of the technical team and pilot projects by creating a learning interface and infrastructure

Preliminary Architectural Proposal for Long-term Digital Preservation (via Clay Shirky) System design by Bob Spinrad

Argument to Congress Digital data loss is huge and growing exponentially The problem will not solve itself Solving the problem will be very hard-- but not impossible It must be solved for national continuity

People Above Institutions In Between Bits Below

Design Principles First, do no harm Humility in the face of change – Legal – Economic – Cultural – Technological

Design Principles II Modular approach Minimal requirements at each level Implementation-agnostic The system should never be optimized Survive the first migration Piggyback on existing work – OAIS/W3C/IETF/Digital Libraries

Caveats 30,000 Foot View Iterative Process Collaborative work

Interfaces Collections Gateways Repositories 4 Layers Between People and Bits

Up

Repositories

Repository Characteristics Don't know about patrons, Interfaces, or even Collections Don't know about contents, only numbers Intentionally “stupid”

Gateways

Gateway Characteristics Go-betweens for Collections & Repositories Bridge Collections & Repository Views –"This is a photo" vs. "These are some bits" Hold minimal metadata -- IDs and Authorizations Both Facilitator and Barrier

Collections

Collection Characteristics Semantics, not storage Where judgment & responsibility reside Served by Gateways, not Repositories Serve Interfaces, not patrons

Interfaces

Interface Characteristics Link patrons with digital assets Build on Collections Minimally defined, to allow for innovation

Down

Connection Principles Each layer is potentially opaque Dark archives can be triply opaque Each connection defined as Protocol Minimally defined – Easily audited – Easily debugged – Easily extended – Easily replaced Exit strategies

Patron and Interface Only negative definitions –Interface must not violate the terms of any Collection it accesses Conversations between Interfaces and patrons otherwise out of scope

Interface and Collection Collection presents Interface with: –What material Interface may use –In what fashion Interface presents Collection with: –Requests for material –Authorization credentials, if required OAIS/DIP

Collection and Gateway Collection presents Gateway with: –Requests for operations on bits –Authorization credentials, if required Gateway presents Collection with: –Results of its request, From 'Not Authorized' Through 'Here are the bits’

Gateway to Repository Gateway presents Repository with: Requests for operations on bits –Read, write, return –Authorization credentials, if required Repository presents Gateway with: –Results of request From 'Not Authorized' Through 'Here are the bits"

Connection Overview These matter more than any instantiation Must survive changes of hardware Must survive element design changes

Top to Bottom Comparisons Interface is:Repository is: VolatileStable Frequent useRare use Read/WriteRead-Only

Ingest Principles Intake is done by Collections Easy in/Hard out Follow OAIS SIP model

Ramifications for Ingest Bit storage is cheap Asset preservation is expensive Security is potentially very expensive Responsibility resides with Collections

Sideways

LoC Involvement Left: Owned and operated by the LoC. Right: The National Library of Latvia Center: Certified partners in National Digital Preservation Infrastructure

Principles of Horizontality Interface definitions fit on a floppy –The manual will have to go on a CD System could be built on one box Many inter-layer diagonal connections could exist. No lateral connections are required.

System as a Whole So where's the preservation layer? Preservation arises from institutions, not technology System as a whole provides tools for stewardship

The Library Cares About Everything In Red

Use Cases: System Stories Caddyshack III - The Iron Mountain Janine's American Memory The Weblogs that Changed the World

Certified Preservation Fills three characteristics – In a certified repository – Referenced by a certified agent – Listed in a certified collection

LC Scenario Framework for the Future Evolution of Digital Preservation What is Saved? Who saves? TriageTriage Congress of Libraries Libraries UniversalLibraryUniversalLibrary Library of Congress Everyone Most Important Everything LC takes the lead in collecting the most critical materials of our digital heritage LC facilitates the development of a “peer-to-peer” system of comprehensive preservation LC plays a clearinghouse role in coordinating preservation efforts that are executed mainly by other institutions

Open Architectural Issues One too many layers for any given problem – Right number of layers for all problems Separating semantics from presentation will be very difficult How to handle dynamic materials? – Self-altering content – Time-sensitive content – Volatile databases

Open Organizational Issues Questions from rights holders –Legal –Economic Moral hazard as collector of last resort Faith in the system is a critical lubricant

Conclusions Preservation is a systemic process What you sign up for is stewardship Divorce between preservation and access Elements can change quickly Connection protocols must change slowly Costs are mainly human and ongoing, not technological and upfront

Conclusions II Profound mental shift in what a library is. Architecture provides tools, not a full solution. Hardware stores, institutions preserve.