1. Implementing an Integrated DAMS FEDORA / VITAL at the National Library of Wales OR08 3 rd April 2008 Paul Bevan paul.bevan@llgc.org.uk Glen Robson glen.robson@llgc.org.uk

2. Implementing an Integrated DAMS at NLW Outline Introduction Single Point of Access – how and why Objects Structure Content Streams Ingest Architecture Implementation of Single Sign On Lessons Learnt

3. Implementing an Integrated DAMS at NLW Background FEDORA Users since 2005 Early Pilot Project –What do we have? –How can we deal with it? –What should come first? Steep Learning Curve Planning for a repository infrastructure

4. Implementing an Integrated DAMS at NLW 2006: A ‘New System’ For NLW Diverse Collections No Single Metadata Standard/Format Variance in the application of Standards Many Legacy Systems Improved Efficiency Increasing Digital Resources & Assets  VTLS (Virtua & VITAL)

5. Implementing an Integrated DAMS at NLW Single Metadata Repository Existing Data Formats Migrated to MARC21 Integrated Information Management System controls accession, circulation, statistics, preservation, etc. Facilitates: –Combined (& Efficient) Workflows –Reduced Support Requirements –Single Point of Access…

6. Implementing an Integrated DAMS at NLW Single Point of Access Why? –Varied Collections –Compartmentalised Users Why Not? –Information Overload –Data Specific Functionality Solution: Many & One

7. Implementing an Integrated DAMS at NLW Multiple Resource Providers Single Point of Access, Single Metadata Repository, Single Catalogue. Resources from many back-end systems: –Subscribed Electronic Resources –Free Online Resources –Digitised/Digital Resources (VITAL) –Stacks (Virtua)

8. Implementing an Integrated DAMS at NLW Catalogue Full SkineResources Family History FEDORA/ VITAL Subscribed Resources IngentaConnect (etc) Digital Exhibitions OAI-PMH Search Engines ATHENS APIs Handle Resolution

9. Implementing an Integrated DAMS at NLW DAMS Project 18 Month implementation process Development requirements from VTLS Substantial metadata discussions Lots of institutional self-discovery

10. OAIS METS Digital Archive Virtua VITAL Disseminators MARC

11. Implementing an Integrated DAMS at NLW Virtua / Vital Integration NLW Spec, VTLS Development Primary access through catalogue –Less confusion for the user –Retain existing skills –Initially DAMS will hold digitisation images so catalogue record already exists ‘Single Sign On’ –Existing patron information used for authentication –User & Group details passed directly from Catalogue to FEDORA as part of HTTP request

14. Implementing an Integrated DAMS at NLW Would we do the same again? Or… in the age of cross-searching is it worth integrating?

15. Implementing an Integrated DAMS at NLW Object Rules & Content Model Every object will have a MARC record (at some level) within the IMS METS will be our SIP, AIP and DIP METS will be the policy, FEDORA services will enact the policy. –eg. DC Section in METS populates the DC datastream –eg. Structural Map held in METS, structure in RELS-EXT datastream Every object in the repository will have a METS document Every object will have DC for OAI-PMH Object Rules evolve into a content model

16. Implementing an Integrated DAMS at NLW Object (PID/Handle) DS: METS DS: DC (OAI-PMH) DS: RELS-EXT DS: Object (eg. TIFF File) DS: Object (eg. JPEG File) DS: Object (eg. Thumbnail File) DS: Object (eg. Zoomify File) Content Model (AIP): Still Image http://hdl.handle.net/10107/1 http://hdl.handle.net/10107/1-0 http://hdl.handle.net/10107/1-1 http://hdl.handle.net/10107/1-2 http://hdl.handle.net/10107/1-10 http://hdl.handle.net/10107/1-11 http://hdl.handle.net/10107/1-12 http://hdl.handle.net/10107/1-13

17. Implementing an Integrated DAMS at NLW METS Header: –History of METS document Descriptive Meta Data –mods on digital copy –link to dc for original –premis details on original file name –relationship information to other objects in Fedora Administrative Meta Data –mix technical meta data on images 1 –dc legacy technical data –premis admin data including checksums, and relationships between data streams –link to MARC record in Virtua Digital Providence –List of actions that have occurred to the object –List of agents who ran the software / versions of the software used File Sec –List of Data streams linking above sections to particular data streams Structural Map –First one allows display via a METS viewer type application –Subsequent ones are linked to Disseminators Behaviour Section –Links objects to their dissemination

18. Object Structure MARC Catalogue Record isPartOf Digitisation Projects (Object) Geoff Charles Collection (Object) … Wrexham Young Farmers' Club ploughing competition (Object)

19. Implementing an Integrated DAMS at NLW Content Streams The library’s mission is to: “collect, preserve and give access to recorded knowledge, in all documentary forms, with an especial emphasis on the intellectual record of Wales, for the benefit of all engaged in research and learning, or with other information needs” In reality this means a LOT of stuff: –Born digital –Digitised –Physical Media –Digital Deposits –Archived websites –Floppy disks in archival files –And on… All of which needs prioritising

20. Implementing an Integrated DAMS at NLW Digitisation Projects (Still Images) Geoff Charles “Geoff Charles (1909 – 2002) was a newspaper photographer who dedicated 50 years of his life to portraying Wales through the lens of his camera. The contribution by Geoff Charles to Wales is unique and today his archive of 120,000 photographs is one of the treasures of the National Library of Wales.” 2,294 Records 14,038 Images 16,322 Fedora Objects 

21. Implementing an Integrated DAMS at NLW Welsh Journals Online JISC Funded 90 Journals since 1900 Titles in Welsh and English 400,000 scanned pages OCR TEI & Coordinates metadata ‘Complex’ rights issues.

22. Implementing an Integrated DAMS at NLW Physical Digital Deposits Variety of unexpected and unknown deposits as part of archives Large collection of deposited CDs No selection process yet CD Accessioning System (CDAS) –In-House development –Remove data from CD (to ISO) & Checksum –Create METS manifest –Determine filetypes with DROID Selection Interface –Navigation of CDs over web interface –View files to aid in selection

24. Implementing an Integrated DAMS at NLW Offair Recording NSSAW Archive Every Programme (DVB-T & Radio) of Welsh Interest –All S4C –News & Current Affairs –Welsh Productions Proprietary BOB system integrated with Virtua & VITAL No Networked Access

25. Implementing an Integrated DAMS at NLW Wills All Welsh probate records 182, 404 Records 816, 325 Images 998, 729 Fedora Objects 27 Days ingest Concerns over Triplestore Upgrade?

26. Implementing an Integrated DAMS at NLW Ingest All automated Wills data is catalogued in MARC 21 with links to an image 1 Record points to a number of pages

27. Implementing an Integrated DAMS at NLW Process – Step 1 Create SIP MARC Record Catalogue Parent METS Image METS Default METS Marc To METS Jhove Output

28. Implementing an Integrated DAMS at NLW Process – Step 2 Ingest METS FOXML METS To FOXML Fedora Fedora Web Services Ingest XML Reserve PID Time to Ingest 16,322 Objects 10 Hours 40mins

29. Implementing an Integrated DAMS at NLW Architecture: Servers Using Fedora 2.1.1 (No Journaling) Redundancy rather than Performance. Server 1 Server 2 Handle Server Vital Instance File System (FOXML) Oracle Database SAN Connected through iSCUSI (Holds Reference Images) Tape / Optical library (Stores Archive copies)

30. Implementing an Integrated DAMS at NLW DataGuard Allows failover of database Keeps both databases in sync Currently we use rsync to sync fedora's data files Data more important than Database!!

31. Implementing an Integrated DAMS at NLW Disseminators Disseminator for each object that returns METS Use cocoon to turn METS XML into web page With no authentication you can build the view functionality as a disseminator Once you turn on authentication it asks for username and password every time Solution is to create a guest account and a web application which sits in between the user and fedora Could be in the same tomcat instance as Fedora

32. Implementing an Integrated DAMS at NLW System Architecture

33. Implementing an Integrated DAMS at NLW Rights Issues for Wills Free access internally Paid for external access External Group paid access Printing services

34. Implementing an Integrated DAMS at NLW Implementing ‘Single Sign On’ Issue 1 - Authentication Username + Password and group in catalogue Web service on catalogue to authenticate Need to develop servlet filter for Fedora to authenticate against this web service Issue 2 - Authorization Use Fedora XACML

35. Implementing an Integrated DAMS at NLW Implementing ‘Single Sign On’ Issue 3 - Single Sign on Use Case 1 User goes to library catalogue Logs on to catalogue finds an item which is in Vital shouldn't have to login again Use Case 2 User in the library goes to catalogue finds an item which is in Vital Solution to Use Case 1 - Surrogates Solution to Use Case 2 - Surrogates or local username? Fedora sees IP address as catalogue Can we pass a surrogate IP address

36. Implementing an Integrated DAMS at NLW Lessons Learnt Keep your options open 80% Soft, 20% Dev Managing Expectations is Difficult And that’s not even talking about format migration…

37. Implementing an Integrated DAMS at NLW Questions? Paul Bevan – paul.bevan@llgc.org.uk Glen Robson – glen.robson@llgc.org.uk http://dev.llgc.org.uk