1. 1 Stanford Archival Repository Project Brian Cooper Arturo Crespo Hector Garcia-Molina Department of Computer Science Stanford University

2. 2 Data does not live forever Much data is stored digitally (perhaps exclusively) –Text –Multimedia (images, sound, etc.) –Scientific data But digital storage is currently unreliable –Magnetic tapes decay, break or lose magnetism –Disks crash –Buildings burn down –Users delete data (accidentally or maliciously)

3. 3 Data does not live forever Digital information already lost: –Early NASA records –U.S. Census Information –Toxic waste records Decay time for common media: –Magnetic Tapes: 10-20 years –CD-ROM:5-50 years –Hard Drive:3-5 years

4. 4 Digital archiving Digital archivists need: –A reliable system to store digital data for long periods without losing it –Convenient tools to add new data and manage data already archived –Methods for finding the “best” configuration »Most reliable »Most cost effective »Etc.

5. 5 Archival Repository Project Goal: Reliably archive digital information for long periods of time (decades or centuries) –Focus on “preserving bits” –Preserving meaning: future work Strategy –Replicate objects –Automatically detect and correct errors Our project –Stanford Archival Vault (SAV) – reliably archives data –InfoMonitor – automatically adds newly created data to the archive –ArchSim – a simulation tool to model archives

6. 6 Architecture Users Filesystem InfoMonitor SAV Archive Archived data Archived data Internet Local archive Remote archive

7. 7 SAV architecture Object Store Reliability LayerRemote SAV Sites Upper Layers User InterfaceData Creation/Import “Core” SAV components Upper layers Application/user level

8. 8 Write-once repository Deletions/modifications disallowed –Any object deleted or modified must have been corrupted, and is replaced Challenges –Constructing structures of objects »Object references constrained to point from new to old objects –Representing modifications »Archive new version of objects = version chain –Finding objects »Indexes

9. 9 Write once repository: Indexes Key to performance –Locate an object quickly using its signature, “Who points to me?” problem, etc. Disposable indexes –Can be rebuilt at any time from SAV objects “Bookmarks” used to find collections of objects using indexed name

10. 10 Write once repository: Indexes Bookmark (with well-known name) SAV

11. 11 Replication: Site networks Sites form “replication agreements” –Agree to replicate data –Specify data to replicate in agreement »May be a subset of all of the data in the archive –Periodically connect and compare data, looking for errors Strongly connected Weakly connected

12. 12 Replication: Data sets SAV replicates different data sets separately –E.g., web pages under agreement A, Usenet articles under agreement B –“Replication sets” should grow without human intervention –Traverse link structure to find objects in set

13. 13 Replication: Data sets Start traversal SAV

14. 14 User interface

15. 15 User interface

16. 16 Object store performance

17. 17 Reliability layer performance

18. 18 The InfoMonitor Goal –Create a convenient, transparent mechanism for getting data from existing stores into the archive Architecture Users Filesystem InfoMonitor SAV Archive

19. 19 Detecting new data Must find and archive new data –Filesystem will not signal data writes –Users should not have to explicitly “check-in” data Scanning –Quick scan: detect changes using timestamps –Slow scan: detect changes using file contents Filtering –Automatically decide what to archive –Use filtering rules

20. 20 User interface

21. 21 User interface

22. 22 InfoMonitor performance

23. 23 Designing Archival Repositories Designer needs to answer questions like: –What is the minimum number of copies of a documents that are needed to ensure its preservation? –What is a more cost efficient, to store the information on one expensive disk with low failure rates or on two inexpensive disks with high failure rate? –Are two sites enough to guarantee preservation? –How often should we scan the repositories for errors? –What’s the MTTF of this design?

24. 24 Contributions A comprehensive model for an Archival Repository A powerful simulation tool: ArchSim, for evaluating Archival Repositories and the available strategies. A detailed case study for an hypothetical TR Repository operated between Stanford and MIT

25. 25 How important is having good disks?

26. 26 Preventive maintenance

27. 27 Current and future work New models for replication agreements and “data trading” Archiving the World Wide Web Modeling cost Managing “meaning” Security Alternative object naming schemes Other “upper layers,” e.g. user access, metadata, etc.

28. 28 Conclusion Digital librarians need tools to preserve data Our project addresses this need –Reliable storage: SAV –Convenient access: InfoMonitor –Finding the best configuration: ArchSim More work must be done to refine these models –More automation –More flexibility –Answer a wider range of design questions

29. 29 For more information http://www-db.stanford.edu/archivalrep Brian Cooper: cooperb@db.stanford.edu Arturo Crespo: crespo@db.stanford.edu Hector Garcia-Molina: hector@db.stanford.edu