IBM Experiences in Digital Collection Building Virtual Libraries

Night Watch by Rembrandt, 1642 A few year ago the painting was damaged by a mentally disturb man with a knife and had to be repaired Digital Collection Building 19-8-2019

Long Term Preservation What is digital preservation? Digital preservation can be defined as: “The act of maintaining information in a correct and independently understandable form, over the long term (OAIS)”. Safe storage and permanent access are both part of digital preservation Connection between the research area of digital preservation and a national digital repository for digitized cultural heritage Digital Collection Building 19-8-2019

Long Term Preservation challenge Sense of urgency to resolve problem is increasing: More and more objects are only available in digital form (“born digital”) There is no mechanism for systematic preservation of information published on the Web national digital repository for digitized cultural heritage No complete integrated solutions exist yet, but: there are many initiatives in the Academic world the topic is on the agenda of the European Commission The DIAS system can be a solution: to structure the complex problem to resolve – already today - a part of the problem to facilitate gradual incorporation of further partial solutions Digital Collection Building 19-8-2019

Technology Preservation Within Long Term Preservation (LTP) three areas of interest can be identified: Medium-, Technology-, and Intellectual Preservation Medium Preservation Medium preservation is the concern for preserving the medium on which information is stored, such as tapes, disks, optical disks, CD-ROMs and the like. Technology Preservation We need to be aware of technology obsolescence as even more of a problem than medium decay, and undertake steps of technology preservation. Rather than simply refreshing, we also need migration and emulation: migrating information forward through technology / format stages as they become available and as the old technologies / formats cease being supported by vendors and the user community. emulating old and obsolete technologies / formats on current technology platforms Intellectual Preservation There remains a third preservation requirement, intellectual preservation, which addresses the integrity and authenticity of the information as originally recorded. Digital Collection Building 19-8-2019

Introduction Koninklijke Bibliotheek – Library of the Netherlands KB was one of the first libraries to develop an electronic deposit with long-term preservation as the key objective Koninklijke Bibliotheek (KB) Medium-sized national library, founded in 1798 Financed by Ministry of Education, Culture and Science Annual budget € 50 million Digital archiving and R&D Permanent Preservation: € 1,1 million structurally allotted to staff, system maintenance € 1,2 million permanently dedicated to research Digital archiving and preservation embedded in organization Departments: e-Depot (electronic publications) Digital Preservation IT Digital Collection Building 19-8-2019

Mission The KB is the National Library of the Netherlands We give researchers and students access to research information  We enable everyone to share in the riches of our cultural heritage We foster the national infrastructure for scientific information We further permanent access to digital information within an international context. Digital Collection Building 19-8-2019

E-Depot – Digital Information Archiving System KB is responsible for all publications appearing in the Netherlands (and international publications from 2002) At the heart of the e-Depot system is a technical component called DIAS (Digital Information and Archiving System) based on the OAIS (Open Archival Information System) Reference Model. This Reference Model establishes a common framework of terms and concepts which comprise an Open Archival Information System (OAIS). It is also a technical recommendation for use in developing a broader consensus on what is required for an archive to provide permanent, or indefinite long-term, preservation of digital information. e-Depot system at the KB is one of the first operational digital archives based on the OAIS, worldwide. Digital Collection Building 19-8-2019

Digital preservation research KB’s research initiatives are geared towards ensuring long-term access - on the digital preservation strategies emulation and migration. Emulation aims to render the digital objects in their original form and to preserve all functionality of the objects. Migration strategy is focused on converting the digital objects to provide access to them in a future computer environment. The digital preservation department is looking into properties of file formats, preservation metadata and the development of a module called the Preservation Manager. The Preservation Manager contains information about how to render digital objects on the file format level. The Preservation Manager will provide the information necessary to be able to plan actions before a specific file type becomes inaccessible. Digital Collection Building 19-8-2019

Scale Current activities are focused on e-publications based on PDF Volume 8 million e-publications currently 10 million e-publications eventually (based on current agreements) Size 1 e-publication equals 1 Mb on average 1 Terabyte for every 1 million publications Capacity 5,000 – 50,000 e-publications ingested per day Digital Collection Building 19-8-2019

Requirements of a national digital repository A function for depositing collections and corresponding metadata A way of searching in deposited collections A request function for specific TIFF files or a complete collection of TIFF files A delivery function for the requested files A security system that should prevent the unwarranted use of collections by other institutions or third parties. The service should be available on a daily basis and independent of the location of the institutions. Development of a data model that facilitates long-term preservation and permanent access to digital images. Digital Collection Building 19-8-2019

Implementation of research results: Upcoming challenges Growth will increase dramatically with the itroduction of new collections Increasing capacity Ingest of digitised master images Ingest of websites 10 TB  500 TB Implementation of research results: Characterisation Operational Preservation Manager Migration module Emulator Digital Collection Building 19-8-2019

Digitization The KB is involved in many national and international digitization projects The Memory of the Netherlands Metamorfoze IMPACT: IMProving ACcess to Text Atlantic World: Dutch-American relationship since the 17th Century Atlas Van der Hagen and Atlas Beudeker Bibliotheca Universalis- Linschoten Bookbindings The Anglo-Dutch heritage Transatlantic Digital Library Digital Collection Building 19-8-2019

Digitization KB has looked into the potential strategies to store digitized material into the their electronic deposit system Each digitized page results in: One high quality master image (TIFF or PNG or JPEG2000) One or more derived lower quality images (e.g. JPG for access) Multiple machine-readable text files (e.g. obtained through OCR) Descriptive metadata Technical metadata about the digitization process Structural metadata describing layout of page (e.g. separate articles) In most cases, each digitized page is part of a larger whole Most books, newspapers are multipage items Requires additional structural metadata to describe images as part of the whole Digital Collection Building 19-8-2019

Digitization Open questions How to link and manage the different representations of the same object One physical representation, one digital master, multiple derived files – only one bibliographic description MPEG-21 DIDL Which fileformat to choose TIFF is very inefficient for storage, JPEG2000 and PNG are more efficient Certainly not JPG (lossy compression) Is lossless compression allowed to reduce storage volume At what level? Compressed file formats, zip/tar packages, storage hardware compression Which files constitute an AIP? A single master image (not a correct representation of a multipage intellectual entity; relations to be resolved outside AIP) Full set of master images for each intellectual entity Including OCR text Are access copies to be considered a new AIP obtained through migration Access copies vs. Preservation copies Storing full set of access copy vs. generating access copies on the fly Digital Collection Building 19-8-2019

Component Business Model Conceptual Approach: A Business Component Map is a is a tabular view of the business components in the scope of interest (Source; Internal 2004) Columns are Business Competencies, defined as large business areas with characteristic skills and capabilities, for example, product development or supply chain. A Business Component is a part of an enterprise that has the potential to operate independently, in the extreme as a separate company, or as part of another company. controlling executing directing Business Planning Business Unit Tracking Sales Management Credit Assessment Reconciliation Compliance Staff Appraisals Relationship Management Sector Management Product Management Production Administration Product Fulfillment Sales Marketing Campaigns Product Directory Credit Administration Customer Accounts General Ledger Document Management Customer Dialogue Contact Routing Staff Administration Business New Business Development Servicing & Sales Product Fulfilment Financial Control and Accounting Sector Planning Portfolio Planning Account Planning Sales Planning Fulfilment Planning An Accountability Level characterizes the scope and intent of activity and decision-making. The three levels used in CBM are Directing, Controlling and Executing. Directing is about strategy, overall direction and policy. Controlling is about monitoring, managing exceptions and tactical decision making Executing is about doing the work Digital Collection Building 19-8-2019

KB´s Component Business Model The competencies are clustered around 5 business areas Service Management includes all the competencies needed to deliver collection objects and associated services to the customers of the KB across the supported channels Acquisitions Management involves the competencies needed to acquire, process and catalogue all publications, both for the research collection and the deposit collection. Preservation Management focuses on the competencies needed to facilitate access to the different collections over-time addressing decay or obsolete technology support associated with a collection. Business Management identifies all the competencies associated with the management of any business IT Management relates to the competencies needed to manage the overall IT infrastructure. KB differs from other organizations because their electronic deposit solution should be operational for hundreds of years without major interruptions Digital Collection Building 19-8-2019

KB´s Component Business Model 51 competencies have been identified in order for the KB to operate Digital Collection Building 19-8-2019

Digital Information Archiving System Architecture Digital Collection Building 19-8-2019

Complexity and data types Web Sites: Dynamic generated pages Volatile external references No general accepted notion of versioning Security hurdles on web sites Needed Web server environment Applications / CDROMS: Dependent on operating system and peripheral devices Require user interaction Potential dependencies with additional software: driver, dlls Static data format Only dependent upon viewer application Format migration will preserve digital object Autonomous digital object Data type Digital Collection Building 19-8-2019

Depending on the main focus different implementation strategies can be identified: Supplier Focus Quality assurance Automatic ingest Security (identification, authentication, authorization) Consumer Focus Delivery channels Content metadata Security (identification, authentication, authorization) Decompose into components AIP composition SIP requirements Self-describing Content (bibliographical) metadata Preservation (technical) metadata Preservation Focus Media preservation Technical metadata Migration tooling Emulation tooling Digital Collection Building 19-8-2019

Descriptive Information about the Package Information Packages are the building blocks of every electronic deposit Submission Information Package (SIP) The Information Package identified by the producer in the submission agreement with the OAIS. Archival Information Package (AIP) Content Information and the associated Preservation Description Information required to preserve the Content Information over the long term. This information includes the related Packaging Information. Dissemination Information Package (DIP) An Information Package that contains part or all of one or more AIPs and that is distributed to the consumer as requested. Information Package Content Information Preservation Description Descriptive Information about the Package Digital Collection Building 19-8-2019

Producer Focus: Infrastructure to support different ingest streams (Batch Builder) Stable SIP and DIP definitions Alignment QA processes between supplier and deposit organization Ingest Preservation Data Management Access Archival Storage Delivery & Capture Packaging Administration Monitoring & Logging Query AIP SIP DIP Digital Collection Building 19-8-2019

Consumer Focus: Solid identification, authentication and authorization infrastructure Support for multi-channel delivery (Packaging & Delivery) Collection building (value-add of deposit organization) Ingest Preservation Data Management Access Archival Storage Delivery & Capture Packaging Administration Monitoring & Logging Query AIP SIP DIP Digital Collection Building 19-8-2019

Preservation Focus: Management of technical metadata Strong focus on providing accessibility (identification, performing technical preservation, building rendering environment) Definition of authenticity criteria Ingest Preservation Data Management Access Archival Storage Delivery & Capture Packaging Administration Monitoring & Logging Query AIP SIP DIP Digital Collection Building 19-8-2019

Typical DIAS implementation scenario Requirements analysis Project starts with requirements analysis, a fit-gap analysis and see how far DIAS-Core fits all requirements Application development Possible new applications components have to be designed, built and tested Infrastructure design and implementation Definition and implementation into the customer’s environment, conforming to the DIAS compliant infrastructure Delivery Through Fast Deploy deliver the DIAS solution in the different environments (DTA) Acceptance Acceptance of the system by the customer Digital Collection Building 19-8-2019

Interoperability Interoperability will be key to the success of the next generation long-term electronic deposit systems Digital Collection Building 19-8-2019

Reference Model for an „Open Archival Information System" – ISO 14721 Digital Collection Building 19-8-2019

IBM´s DIAS is based on some OAIS modifications developed in the Networked European Deposit Library (EU-Projekt 2000 – 2002) Delivery & Capture handles the pre-processing of digital objects to be ingested. It receives or captures digital objects and offers a working space for verification in conformance with the specifications for ingestion into the electronic deposit system. Packaging & Delivery is the output interface of the deposit system. It handles the post-processing of digital objects retrieved from the electronic deposit system. It negotiates access requests, delivers and installs electronic publications along with the appropriate software for viewing or running the electronic publication and handles the metadata for direct access by the requestor. Ingest Preservation Data Management Access Archival Storage Delivery & Capture Packaging Administration Monitoring & Logging Query AIP SIP DIP Digital Collection Building 19-8-2019

DIAS-Core: Technical Components ISIP IDIP Ingest Access Archival Storage Preservation Planning Administration CM Resource Manager TSM Storage Server Loader Retriever Data Management AccessManager CM Library Server Logging Reporting Preservation Manager Monitoring and Control Digital Collection Building 19-8-2019

DIAS-Core: Application Architecture Client Layer Midtier Layer Server Layer Physical Storage Layer Admin Client Admin Access Manager Server Retriever Client Retriever DB2 CM LS CM RM1..n Tape Magnetic Disk SAN Loader TSM Optical Access Manager Logger Persistent Identifier Generator Monitoring & Control Digital Collection Building 19-8-2019

WebSphere Application Server Content Manager together with other members of the DB2, Tivoli and WebSphere product families covers the majority of components in the e-Depot architecture Archival Storage Ingest Preservation Data Management Access Delivery & Capture Packaging Administration External System Monitoring Content Manager DB2 WebSphere Application Server TSM Digital Collection Building 19-8-2019

Delivery&Capture and Package&Delivery Java applications running on Websphere Application Server take care of specialized DIAS functions and customer specific requirements Ingest and Access: Implement SIP and DIP interfaces Translate and validate customer metadata to DIAS metadata Delivery&Capture and Package&Delivery Customer functions to create SIPs and extract DIPs Customer functions for Preservation Manager Implement functions to monitor file formats and support migration of file formats Digital Collection Building 19-8-2019

Reference on long-term preservation Raymond Lorie and Raymond J. van Diessen: Long-Term Preservation of Complex Processes in IS&T Archiving Conference, Washington, DC, April 26-29, 2005. Hoeven, J.R. van der, Diessen, R.J. van en Meer, K. van der, Development of a Universal Virtual Computer (UVC) for long-term preservation of digital objects, Journal of Information Science, vol. 31(3), p. 196-208, 2005. Raymond van Diessen and Raymond Lorie, UVC: A Universal Computer for Long-Term Preservation of Digital Information, RJ 10338, IBM Almaden Research Center, San Jose, CA, 2005 Eric Oltmans, Raymond J. van Diessen, Hilde van Wijngaarden: Preservation Functionality in a Digital Archive in ACM/IEEE Joint Conference on Digital Libraries,Tucson, AZ, June 7-11, 2004 Raymond J. van Diessen and Titia van der Werf - Davelaar: Authenticity in a Digital Environment, in: IBM / KB Long-term Preservation Study Report Series, IBM Global Services Netherlands, 2002, ISBN/ISSN: 90-6259-155-8 Raymond J. van Diessen: Preservation Requirements in a Deposit System, in: IBM / KB Long-term Preservation Study Report Series, IBM Global Services Netherlands, 2002, ISBN/ISSN: 90-6259-156-6 Raymond J. van Diessen and Ben J. van Rijnsoever: Managing Media Migration in a Deposit System, in: IBM / KB Long-term Preservation Study Report Series, IBM Global Services Netherlands, 2002, ISBN/ISSN: 90-6259-158-2 Raymond J. van Diessen and Johan F. Steenbakkers: The Long-Term Preservation Study of the DNEP Project - an Overview of the Results, in: IBM / KB Long-term Preservation Study Report Series, IBM Global Services Netherlands, 2002, ISBN/ISSN: 90-6259-154-X Digital Collection Building 19-8-2019

DIAS Solution: http://www.ibm.com/nl/dias/ Important sites related to DIAS and IBM´s long-term preservation effort: DIAS Solution: http://www.ibm.com/nl/dias/ IBM Alphaworks UVC: http://www.alphaworks.ibm.com/tech/uvc KB: http://www.kb.nl/site/sitemap-en.html Kopal: http://kopal.langzeitarchivierung.de/ Digital Collection Building 19-8-2019