Tackling concrete digital preservation challenges with SPRUCE Paul Wheatley SPRUCE Project Manager University of Leeds Twitter:

Summary Some digital preservation challenges and solutions –Not exhaustive –Illustrate with some real examples –Summarise with some practical steps for digital preservation Taking a community approach to digital preservation –SPRUCE Project –How to get involved –Where to get help

Keeping the bits

Digital data is fragile Courtesy of State and University Library, Denmark

Media decay Media becomes partially or completely unreadable Media obsolescence Without the respective hardware to read the (hand held) media, it becomes inaccessible Practical issues Inserting lots of discs into a drive is costly Images courtesy of The British Library Digital preservation storage: keeping the bits

Bit storage recommendations Don’t fall for media longevity claims from vendors! They are missing the point! Accept that media decays, media formats will change, and any media will become inaccessible in the medium term Rather than putting your data in a dark archive and trusting it will survive for a long period... Manage it closely, refresh to new media frequently, chose media that is easy to manage –Choose media that is easy to access (server storage, cloud, external hard drives) –Make at least 3 copies of all data, keep copies in different geographical locations –Frequently check the condition of your data

Are any of your digital files missing? Are any of your digital files damaged? Verifiable Manifests (Checksums!) Single most useful digital preservation activity Generate manifests as early as possible Frequently re-check them over time Mend content when necessary LoC Bagit specification and Bagger tool Allow you to easily check the condition of your digital stuff

SOI APP0 JFIF 1.2 APP13 IPTC APP2 ICC DQT SOF0 200x392 DRI DHT SOS ECS0 RST0 ECS1 RST1 ECS2… … Dependence on software

When it goes wrong…

Migration, Emulation and all that... Migrate content from an obsolete format to a more modern usable format Emulate the original computing environment and run the obsolete software originally used Words of caution: –Is software obsolescence a really critical risk for our digital data? –The debate continues... International Council of Archives Congress 2012: Michael Carden, National Archives Australia: NAA migrates all content Oliver Morley, UK National Archives: “digital formats have standardized” Blogged by Inge Angevarre: –The hard part is the quality assurance of the results. Was anything lost or damaged in the process?

Stuff happens! Whenever a digital collection is moved, processed, curated or altered in any way.... things can go wrong! Network dropouts at critical times Disks get full, subsequent data copied there is lost Software bugs lead to unexpected results Human error leads to all sorts of issues Stuff happens a lot more at scale!

Digitisation post processing corruption Images courtesy of The British Library

TIFF to JPEG2000 migration corruption Images courtesy of The British Library

JPEG 2000 Format specification ambiguity and corresponding tool bugs JPEG2000s can be missing vital source resolution Technology can be imperfect! For more on JPEG2000 format and tool risks see: Images courtesy of The British Library

Only process or alter digital content when it is absolutely necessary Double check everything Make no assumptions Assume nothing, validate everything

Prompt check in – have you got what you thought you would receive? –Check expected files are present, open a random selection to verify expected quality –Request replacements from supplier promptly Create a verifiable manifest –Create a top down manifest file that lists each digital object in your collection as a relative filename and a checksum –Library of Congress Bagit specification and tools will also do a good job here Make at least 3 copies. Protect the bits –Keep a copy on easily accessible media –Backup to tape or more disk. Keep copies in different geographical locations to avoid catastrophic disaster. Cloud storage is also an option. Frequently inspect the condition of your data –Revisit the collection, recalculate your manifests and verify content has not been lost –Do a test recovery of your backups to ensure they are working effectively! Record the existence of each of your collections in a digital items register –Record: What it is, who is the responsible owner, where it is, who owns it, and who can access it. Assume nothing, validate everything! –Double check any processes in the lifecycle that move or alter your digital content –Built in checks can be flawed, a second opinion is much more trustworthy First steps in practical digital preservation

JISC funded 2 years in length (until Nov 2013) £250k funding SPRUCE Project Sustainable Preservation Using Community Engagement

Some observations Lack of focus on the real needs of digital preservation practitioners Insufficient collaboration + coordination Duplication of effort

3 day workshop for ~30 people Practitioners bring along digital collections We identify preservation challenges Pair up practitioners with technical experts Apply existing open source tools to solve the problems In doing so, we exchange knowledge about digital preservation Develop a supportive community The SPRUCE Mashup: Identify and Solve concrete problems Glasgow Mashup April 2012

–What is this digital collection? –What risks are associated with this digital collection? –Separate collection content from temporary/other files. –Identify and weed duplicate or similar files. –Is the metadata consistent with the content? –Are all the pages present in each issue? –Are all digitised pages in focus? –Are any files damaged? –Are the files compliant with a particular profile? See the results here: What questions do practitioners want answered?

Work with practitioners to develop a business case for their work Make small funding awards to further develop and embed the work begun in the mashups Make it sustainable York Mashup September 2011

Sharing requirements Sharing experiences: what tools worked well, what approaches should be avoided Building on existing tools, rather than re-inventing the wheel Libraries + Information Science question and answer site: – More recommended collaborative activities: – Online collaboration

Thanks for listening! Any quesions? Paul Wheatley SPRUCE Project Manager University of Leeds