Wrangling DigiTool Data For LOCKSS Brian Meuse - Digital Collections Systems Analyst University Libraries Boston College MetaArchive Cooperative Annual Meeting October 23, 2009

Electronic Theses and Dissertations –Undergraduate Honors Theses –Graduate Level Theses and Dissertations Archive and distribute Provide global Open Access to content –Embargoes when needed –No mandate to publish

What happens next? ProQuest processes students submission ProQuest ftp's back –Thesis (pdf) –Any additional files (3rd party permissions) –Descriptive metadata Once student uploads to ProQuest we get back within a day.

LOCKSS MetaArchive Cooperative –LOCKSS based dark archive –Long Term Digital Preservation

DigiTool Oracle backend –Maintains object relationships –Stores all associated MetaData (XML) –Original filenames File storage –Simple directories on filesystem –Renamed to Unique Identifier (PID)

DigiTool To LOCKSS Export ETD files from DigiTool –Export function –Duplicate data –Current ETD collection is ~1GB –Bobbie Hanvey, ~30,000 photo-negatives ~600GB

DigiTool To LOCKSS Direct URL links –MetaData –Objects (Viewers for different formats) Direct links not persistent –Redirected to URL with session id –Every node is different –Not good for polling.

DigiTool To LOCKSS DigiTool API –SOAP web service –Can query database –Retrieve XML MetaData Links to objects

DigiTool To LOCKSS Wrangling the data –Perl –Web Services –XSLT

DigiTool To LOCKSS createMetaArchiveAU.pl #!/usr/bin/perl -w use strict; use SOAP::Lite; use FileHandle; use Getopt::Long; use LWP::Simple; use Time::localtime; use XML::LibXSLT; use XML::LibXML; ….

DigiTool To LOCKSS Query DigiTool … contains type electronic thesis dissertation after createDate FROM … XML response is list of pid’s

DigiTool To LOCKSS Retrieve digital entity for each PID XML contains –All Metadata for object –PID’s of related objects –Filename and path of file on server

DigiTool To LOCKSS Metadata descriptive etd-ms The Impact of Pension Policy on Older Adults' Life Satisfaction: an Analysis of Longitudinal Mulitlevel Data Calvo, Esteban aging individualization life satisfaction pension policy redistribution subjective well- being Boston College Williamson, John B Electronic Thesis or Dissertation text application/pdf English I hereby allow Boston College to include and preserve my dissertation/thesis in electronic form in the Boston College Institutional Repository, which shall include the right to publicly post my dissertation/thesis on the World Wide Web. I will retain copyright ownership, but I grant to Boston College the non-exclusive right to copy, distribute, and publicly display my dissertation/thesis in any form as may be necessary or convenient in the future as file formats, storage media, and distribution mechanisms evolve. PhD Doctoral Sociology Boston College. Graduate School of Arts & Sciences. ]]>

DigiTool To LOCKSS Related objects manifestation manifestation manifestation

DigiTool To LOCKSS Filename and path Calvo-Esteban.pdf pdf application/pdf /exlibris1/bcd03storage/2009/08/27/file_1/

DigiTool To LOCKSS Retrieve each related item to get filename and path for those items manifestation manifestation manifestation

DigiTool To LOCKSS Generate script to generate links –Symbolic link for AU –From manifest web directory to object ln -s /exlibris1/bcd03storage/2009/08/27/file_1/ /106484/Calvo-Esteban.pdf When file is harvested, it will be given the original filename.

DigiTool To LOCKSS Manifest Pages –Transform XML to HTML –XSLT

DigiTool To LOCKSS manifestation manifestation manifestation

DigiTool To LOCKSS Manifest for Calvo, Esteban 2009 Electronic Theses and Dissertations at Boston College Manifest for Calvo, Esteban 2009 Metadata and Relationships est=true&GET_XML=1&pid= ETD PDF Calvo-Esteban.pdf Permissions/Suppressed file Calvo-Esteban-permission.txt Thumbnail _106484_pdf_thumbnail.jpg

Thank you!