1. DIGITAL PRESERVATION IN THE WILD CARLI Digital Preservation Forum – July 21, 2009 Tim Donohue Research Programmer - IDEALS University of Illinois (with many thanks to Sarah Shreeves)

2. It all starts with a small story a dream of sorts

3. IDEALS: the “dream” “Create a reliable and easy to use repository service to preserve, manage, and provide persistent and widespread access to the digital scholarship faculty and students now produce…”

4. IDEALS: the “dream” “Create a reliable and easy to use repository service to preserve, manage, and provide persistent and widespread access to the digital scholarship faculty and students now produce…” Can we preserve everything? What kind of expertise do we need? What kind of resources? What kind of infrastructure? What’s it mean to preserve this stuff? BUT…

5. Not Really Our Server Room! Backup tapes stored next to the server! IDEALS: the initial reality

6. 1. Brought in Preservation Librarian 2. Training and self education 3. Assessment of where we were and where we needed to go

7. The Foundation Image borrowed from the ICPSR Digital Preservation Tutorial: http://www.icpsr.umich.edu/dpm/ http://public.ccsds.org/publications/archive/650x0b1.pdf Open Archival Information System (OAIS) Model

8. TRAC (Trustworthy Repositories Audit & Certification) http://www.crl.edu/PDF/trac.pdf  Organizational Infr.  Digital Object Mgmt  Technical Infr. & Security The Foundation II Documentation Transparency Adequacy Measurability?

9. The Digital Preservation Platform Image borrowed from the ICPSR Digital Preservation Tutorial: http://www.icpsr.umich.edu/dpm/

10. From Dorothea Salo. 2009. Institutional repositories for the digital arts and Humanities. Humanities Digital Curation Institute. Champaign IL. May 2009. http://www.slideshare.net/cavlec/digital-preservation-and-institutional-repositories http://www.slideshare.net/cavlec/digital-preservation-and-institutional-repositories

11.  “Preservation” needs to be unpacked.  Not about the technology.  Explicitness is key.  You don’t have to preserve everything to the fullest extent if you say you aren’t.

12. The 5 Stages of Preservation  Denial / Ignorance  Anger / Fear  Bargaining  Depression  Acceptance & Hope Based on the Kübler-Ross five stages of grief: http://en.wikipedia.org/wiki/K%C3%BCbler-Ross_model

13. Denial / Ignorance backups** ** - This service is entirely fictitious Again, Not Really Our Server Room!

14. Anger / Fear Data Loss Obsolescence

15. Bargaining Please, just let me get this data migrated elsewhere

16. Depression Why even try? How can we ever preserve everything? This is too hard. We don’t have the resources for this.

17. Acceptance & Hope We can take small steps to…  Preserve some things locally  Develop policies (say what you do)  Enact policies via procedures (do what you say)  Work with others on best practices to preserve the rest

18. The Principles of Preservation (1) Say what you do… (2) Do what you say… Based on: Sarah Shreeves. 2009. Saying what we do – Doing what we say: Preservation Issues (Metadata and Otherwise) in Institutional Repositories. American Library Association Conference. Chicago IL. July 2009.

19. IDEALS - Saying what we do  Secured explicit administrative support and commitment for digital preservation management program in IDEALS. http://hdl.handle.net/2142/135 http://hdl.handle.net/2142/135  Developed high level preservation policy: http://hdl.handle.net/2142/2383 http://hdl.handle.net/2142/2383  Developed actionable procedures and policies that can be reassessed and changed as needed  Began next stage of identifying & documenting gaps

20.  Format-based, “Categories of Support” High Confidence Full Support Medium Confidence No migration promised Low Confidence “Bit-level” support only Openly Documented Widely Adopted Widely Supported Uncompressed or Lossless Compression No Embedded Content or DRM Low Confidence (gray area) IDEALS Preservation Support Policy https://services.ideals.uiuc.edu/wiki/bin/view/IDEALS/PreservationSupportPolicy

21.  Compilation of “known” formats  Concentration on textual formats ProprietaryOpen Microsoft OfficeOpenOffice.org, HTML Limited Adoption Widely Adopted OpenOffice.orgMicrosoft Office, HTML Limited Support Widely Supported Microsoft OfficeAdobe PDF, HTML Embedded Content / DRM Nothing Embedded MS Powerpoint (w/ Audio or Video)MS Powerpoint Lossy Compression No/Lossless Compression JPEGTIFF, JPEG 2000 IDEALS Format Support Matrix

22. Textual CSV, Text, PDF/A, XML, Open Document Format RTF, MS Office, PDF, HTML Audio AIFF, WAVE, Ogg Vorbis AAC, MP3, Real, WMA Images TIFF, JPEG 2000 GIF, JPEG, PNG Video AVI, Motion JPEG 2000 MP2, MP4, Quicktime, WMV High Confidence / Preference Medium Confidence / Preference IDEALS Format Recommendations https://services.ideals.uiuc.edu/wiki/bin/view/IDEALS/FormatRecommendations

23.  Basic Activities (All Items: )  Regular Virus Scans, Checksum verification  Nightly off-campus backups  Refresh storage media  Preservation Metadata (minimal) Format, checksum, file size, etc.  Permanent Identifiers (Handles)  Always keep the original document  Monitoring and reassessment of formats IDEALS – Doing what we say

24.  Intermediate Activities ( )  Additional monitoring, more frequent reassessment  When possible, attempt to migrate formats to preserve content and style (hopefully) No promises that functionality will be preserved (e.g.) Powerpoint  PDF (possible functionality loss) (e.g.) PDF 1.4  PDF/A (possible style loss) IDEALS – Doing what we say

25.  Full Support Activities ( )  Additional monitoring, more frequent reassessment  When necessary, migrate document to successive format.  Attempt to preserve content, style and functionality (e.g.) PDF/A  successor to PDF/A IDEALS – Doing what we say

26. Our First Preservation Problem…  Character issues in Word (and PDF)  Found by chance  Consultation with submitter  Caused by conversion to Word (from Wordperfect)  Resubmitted as RTF

27. We Acknowledge our Gaps  Not checking format validity (yet)  Minimal metadata collection  Not checking files for problems (besides viruses)  Not checking every automated conversion

28. Back to that “dream”? “Create a reliable and easy to use repository service to preserve, manage, and provide persistent and widespread access to the digital scholarship faculty and students now produce…” Total Items: 11,500Total Downloads: 870,000+

29.  Slide 2 (Book image): http://www.flickr.com/photos/riot/100006656/http://www.flickr.com/photos/riot/100006656/  Slide 3 (MS office): http://www.flickr.com/photos/niallkennedy/374272762/http://www.flickr.com/photos/niallkennedy/374272762/  Slide 5/13 (Server room): http://www.flickr.com/photos/sylvar/31436961/http://www.flickr.com/photos/sylvar/31436961/  Slide 6 (Get your act…): http://www.flickr.com/photos/dreamsjung/3595425744/http://www.flickr.com/photos/dreamsjung/3595425744/  Slide 11 (I know this…): http://www.flickr.com/photos/ali_kat_xx/1373989245/http://www.flickr.com/photos/ali_kat_xx/1373989245/  Slide 13 (Disk backups): http://www.flickr.com/photos/tonyaustin/2355186770/http://www.flickr.com/photos/tonyaustin/2355186770/  Slide 14 (Disk lost): http://www.flickr.com/photos/mag3737/2415681602/http://www.flickr.com/photos/mag3737/2415681602/  Slide 14 (Broken cd): http://www.flickr.com/photos/rickheath/72041533/http://www.flickr.com/photos/rickheath/72041533/  Slide 15 (VHS to DVD): http://www.flickr.com/photos/28910181@N05/3085532220/ http://www.flickr.com/photos/28910181@N05/3085532220/  Slide 15 (Mac transfer): http://www.flickr.com/photos/cyprien/6173244/http://www.flickr.com/photos/cyprien/6173244/  Slide 16 (Depression): http://www.flickr.com/photos/dbarefoot/2652496167/http://www.flickr.com/photos/dbarefoot/2652496167/  Slide 17 (Hope): http://www.flickr.com/photos/livenature/259458056/http://www.flickr.com/photos/livenature/259458056/  Slide 17 (Dioum Quote): http://www.flickr.com/photos/wallyg/469808717/http://www.flickr.com/photos/wallyg/469808717/  Slide 27 (Gaps): http://www.flickr.com/photos/aduki/2416528101/http://www.flickr.com/photos/aduki/2416528101/ Credits

30. Contact Info Tim Donohue University of Illinois tdonohue@illinois.edu http://www.ideals.uiuc.edu/ http://www.ideals.uiuc.edu/wiki/ This work is licensed under a Creative Commons Attribution- Noncommercial 3.0 United States LicenseCreative Commons Attribution- Noncommercial 3.0 United States License