1. Automated Archiving of DVD Content
Esteva, Vega, Nieto, Scott, Gunnels, Kumar, Lamphear, Henriksen, Lee, Martin
TCDL 2013

2. Motivation and perspective
Find a file based preservation solution for video art in DVD media
Create a SIP including files to fulfill museum functions
Availability of high performance distributed storage
Next generation display systems
This project presents the design and implementation of an automated workflow that produces Submission Information Packages (SIP) for archiving video art in DVD only format. The project was elicited by the need of the Blanton Museum of Art at UT Austin to preserve their video art collection and deal with the risk of obsolescence, corruption and security of the media but also to fulfill with exhibition, outreach, publication, loans, research and education functions at a university museum. The project was designed in the context of the availability of high performance distributed storage for UT Austin and UT System faculty and researchers which is provided through the Texas Advanced Computing Center. It is also informed by the center’s experience in visualization technologies and work in research and development of next generation display systems.

3. Considerations
Study the role of the DVD in the works technical history
Options for conversion
Usage of the DVD in the museum
To begin the project we had various issues to consider.
DVDs are an optical storage medium in which video data is encoded in MPEG 2 lossy format [1]. Popular due to their high storage capacity and portability, a little more than a decade after their release DVDs are becoming obsolete. In terms of their life expectancy , and despite different manufacturing qualities, DVDs do not constitute a long-term archival storage solution due to their vulnerability and risk of obsolescence [1] [6] In addition, from a digital assets management perspective, DVDs’ physicality is inconvenient in contrast to the benefits of file-based archival solutions now preferred in cultural heritage and commercial settings for storage, long term preservation and distribution.
All of these considerations justified moving the media-dependent collection to a file based system, but it was the media dependency of the DVDs and their compression which raised a set of questions. Was it reasonable to preserve works that had been already compressed? Some of these videos are already on Yu tube? Couldn’t we find the masters? To what kind of digital files we were going to convert the DVDs? We needed to understand the technical trajectory of the collection, the role of the DVDs and their usage at the museum.

4. Considerations UT Research Storage Infrastructure and Services
Example of next generation display systems
What we did have was the infrastructure required to store the files. The University of Texas at Austin through TACC, designs and maintains hp storage resources that are accessible remotely and can be configured by the user or with aid from the data and collections management team. In this environment users can“build their collections” using different technologies available as such as rdbms, gis, irods, web services and different access protocols, security and 24/7 maintenance and replication facilities. Researchers and curators can create, develop, and manage data and collections throughout their lifecycle and create functioanlities needed for access, research, analysis, and display functions. We decided to develop a workflow to convert and ingest the collection into the repository where it could be shared, documented and move forward.
We also had the example of what next generation display tools may look like in the near future. One of TACC’s key resources is its scientific visualization laboratory (Vislab), whose 2900-square foot space contains several cutting-edge display systems. These include the highest resolution tiled-display in the world (nicknamed Stallion) a 4K projection system a multi touch tiled display which was developed through a research collaboration with NARA as well as stereoscopic displays capable of playing video and rendered graphics in 3D using active shutter glasses. All of the display systems in the lab can be controlled through a single computer or node, and through this “head node” all other machines including storage resources can be accessed.

5. Workflow requirements
Only for DVD-based works
Transcoding should not involve further compression
Fixity to establish authenticity of disk image and production quality/preservation master
Quality metric to assess the quality of the transcoding
Metadata and provenance documentation
Easy to implement
SIP contents should fulfill preservation and access functions
Using the information gathered from the survey and after conducting some transcoding tests we established requirements for the workflow.
The workflow would be used only for DVD based works for which no other high quality media existed.
We definitely did not want to further compress the DVD media. In our transcoding tests we realized that using default settings would reduce the quality of the final product, we wanted to preserve the content and to introduce a quality metric to asses the outcome of the process.
We wanted to provide the museum with a functional best quality copy that they could use for exhibition purposes as well with streaming or other low quality accessible files that they needed to distribute to curators or to teachers if needed.
Having gone through difficulties finding out the technical transitions of the pieces, we wanted to make sure that we were recording our intervention.
Such workflow has many steps and involves different pieces of software, we wanted to automate the process as much as possible, to make it easy for museum staff to undertake it.

6. Preservation roadmap Best practices in video and digital preservation
Few references for DVD preservation
ISO disk images as identical copy of the DVD
Preservation master
Base for metadata extraction and further conversions
Not playable
High quality access file (also preservation file)
Matroska container and ffv1 codec
Quality metric
Structural Similarity Index (SSIM)
Documentation – metadata and processing provenance
In the literature we did not find many recommendations in terms of what to do with DVD based art. To design our workflow we adapted recommendations from outstanding digital preservation and from analogue video reformatting projects. As we move through the workflow implementation and we learn more about the process, we are still considering the components of the Submission Information Packages (SIP), and we have changed our minds since we wrote the paper as to the terminology and roles of the different components. So, this is our roadmap for now.
ISO disk images are open standard, uncompressed files, considered archival because they constitute an identical copy of the content and structure of an optical disc medium. we capture a disk image of the DVD and use it as the source for transcoding to other preservation and access formats. The ISO image is not playable but it can be burn into another DVD which at the museum is important given that some of the exhibit copies are already unreadable in some DVD players and computer drivers.
To generate a highest possible quality preservation and access file, we transcode the ISO image containing all the media streams to a Matroska container with FFV1 codec to create a lossless conversion with respect to the lossy MPEG2. The choice of wrappers and codecs in different institutions is decided by emerging standards and usage needs. We are aware of recommendations to use Material eXchange Format (MXF) wrapper and JPG2000 codec [12]. However, our choice of open source software for transcoding limits our possibilities as it does not support MXF or Jpeg2000 at the moment. Like MXF, the Matroska container allows encoding of audio and any accompanying materials. Matroska is an open standard free container format for general use. We expect to provide the MXF and JPG2000 combination in the future.
Optionally, the workflow can generate online access and streaming files, in which case a lossy codec is acceptable.
Visual quality control is routinely used to determine the differences between the analog original and the digital file in digitization projects. In the same manner, we use a full reference video quality metric, which assesses the quality of a transcoded video using the original video as reference, to assess the quality of the transcoding step. The metric selected is the Structure Similarity (SSIM) index The DVDs are already a highly compressed format, the video quality metric allows us to determine if the conversion was lossy or lossless by providing a measurement of quality between the original and the converted file.
Documentation is produced at every step.

7. Workflow step by step Metadata extraction ISO disk image Checksum
Transcoding
Conversion to other access files
Quality metric
Visual evaluation
To illustrate a workflow we use the case of the video “Winchester” part of the Winchester trilogy” by Jeremy Blake
The artist describes the three works as time-based paintings containing digital animations and mash-ups of retouched images. The master DVDs are signed, dated, and numbered and come with a corresponding viewing disc. The Blanton’s files contain instructions on displaying the piece in 16:9 aspect ratio with a 42-inch plasma screen. The artist also indicated that this aspect ratio should be preserved in any future conversions.
Extract MetaData w/ MediaInfo
Technical metadata is extracted from the disk image and the reformatted files using MediaInfo [17]. This includes the container, frame size, frame rate, overall bit rate, duration, aspect ratio, video and audio format and types and number of streams. Through this step the script can later verify the exact technical characteristics of the content so that these are not changed during the transcoding process. Provenance metadata, containing information about the kinds of conversion and their outcomes as results of the video quality control metrics, are generated by the workflow script as each step begins and ends.
Generate ISO & MD5 checksum
A disk image of the DVD and its checksum are produced.
The disk image contains all the contents of the optical disc and can be used to reproduce a new copy of the master DVD and to convert it to preservation and access files. It does not allow for straightforward content playback.
Its checksum are created.
Generate MKV using FFMPEG
In order to create the Matroska high quality preservation and access file, we use the metadata of the original file to ensure that the technical characteristics of the video are preserved accurately . We had to make sure though that the right information is parsed, as the technical metadata also includes aspect ratios for other video streams such as the top menu. We use the FFV1 codec, a lossless video codec, to allow exact original data to be reconstructed from the compressed DVD. If available, other streams such as audio and subtitles are copied in their original format.
Generate MP4 streaming file
Finally, the metadata of the master file is also extracted.
The streaming is created to give the museum the option to stream the content electronically. The transcoding software, HandBrakeCLI, is used to generate an MP4 container along with a lossy codec such as H264.
Quality Control
The last step is the video quality metric is used to compare the original disk image to the preservation files and optionally to the streaming file.. The output of the SSIM algorithm is recorded as metadata.

8. Quality Control Full reference quality metric
Structural Similarity Index
Aim for result of 1
Algorithm available in different programming languages
Modeled closely to human perception
Visual inspection of the Matroska file
The first table shows the output of the video quality metric for one of the test cases. The table shows the average SSIM index for the production quality file (MKV) and the streaming copy (MP4). A SSIM Index of one means that the two images being compared have the same quality, and thus that the encoding was lossless. In the conversion from disk image to production quality file we aim to for a result of one. A SSIM Index below one indicates a loss of quality. However, in our experience we still have to determine an acceptable threshold for lossy conversion. As seen in the table, the streaming copy has a small loss in quality due to the lossy encoder used to create it. When visually inspected, the loss in quality is imperceptible.

9. Software choices Informed by testing Open source, command line tools
Support technical documentation
Our choice of software was informed by testing, preference for open source well documented command line tools and cross platform pieces that are actively contributed by the community and support varied codecs and containers, can extract metadata, can handle audio files and menus and are highly configurable beyond their default settings. The main software pieces behind the program are mediainfo, ffmpeg, and python

10. Testing and Implementing the Automated Workflow at UT Libraries

11. UT Libraries and Audiovisual Digitization
Audiovisual Digitization Unit
DVD Archiving
Preservation Reformatting
Digital Preservation
Example DVD archiving projects
Benson Latin American Collection
Fine Arts Library Streaming Video Project

12. DVD Archiving Requirements
ISO image of original
Streaming MP4
DVD access copy
2 checksums
Descriptive metadata
Source metadata
Technical metadata

13. Current Workflow SIP BagIt Copy
Create streaming MP4 from main ISO media stream using Handbrake
Quality control of MP4 file
Move and delete file from local workstation to remote server
Compile descriptive metadata as MODS XML from catalog MARC record using MARCedit
Compile source metadata as XML from Digitization record using Oxygen
Compile technical metadata as XML from ISO and MP4 using MediaInfo
Move all XML documents to remote server
Create decrypted ISO disc image using DVDShrink
Quality control of ISO file
Move and delete file from local workstation to remote server
Update Project Record with transfer notes and status
Compile ISO file, MP4 file, and all XML files into a single directory folder
Run BagIt script to contribute verifyvalid data, checksum data, manifest and tagmanifest data, as well as well baginfo data.
Systems confederates XML into single UTVideoMD XML document
Create physical derivatives from ISO
Quality control of physical derivatives
Label physical derivatives
Send physical derivatives and originals to cataloging or return to branch and denote return in Project Record
SIP
BagIt
Copy

14. Partnering with TACC
TACC approached UT Libraries to expand testing and implementation of automated workflow
Benefits of automation for the Libraries:
Streamline and unify current workflow
Less staff-intensive
Reduce opportunities for error

15. Testing and Results Test DVDs Workstation Results
18 single-stream, unencrypted discs
3 multi-stream, encrypted discs
Workstation
Mac Pro running OS Mountain Lion
Results
Encryption errors
MP4 streaming compatibility
Saving files to host directory
SSIM and Python Library limitations
Checksums
Enthusiasm for the next phase!

16. UT Libraries Modifications
Addition of drag and drop functionality
DVD
Project Folder
Addition of web-optimized MP4
Elimination of Matroska file

17. Future Modifications Decryption 2 Checksums SSIM quality measure
Expand metadata collection
UTVideoMD
Integrate BagIt

18. Manual Workflow SIP BagIt Copy
Create streaming MP4 from main ISO media stream using Handbrake
Quality control of MP4 file
Move and delete file from local workstation to remote server
Compile descriptive metadata as MODS XML from catalog MARC record using MARCedit
Compile source metadata as XML from Digitization record using Oxygen
Compile technical metadata as XML from ISO and MP4 using MediaInfo
Move all XML documents to remote server
Create decrypted ISO disc image using DVDShrink
Quality control of ISO file
Move and delete file from local workstation to remote server
Update Project Record with transfer notes and status
Compile ISO file, MP4 file, and all XML files into a single directory folder
Run BagIt script to contribute verifyvalid data, checksum data, manifest and tagmanifest data, as well as well baginfo data.
Systems confederates XML into single UTVideoMD XML document
Create physical derivatives from ISO
Quality control of physical derivatives
Label physical derivatives
Send physical derivatives and originals to cataloging or return to branch and denote return in Project Record
SIP
BagIt
Copy

19. Automated Workflow
BagIt
SIP

20. Thank You
Maria Esteva and Karla Vega Texas Advanced Computing Center Vandy Henriksen, Jennifer Lee, Wendy Martin University of Texas Libraries Sue Ellen Jeffers and Meredith Sutton Blanton Museum of Art Bethany Scott Charlotte Mecklenburg Library Kertana Kumar University of Texas College of Natural Sciences Summer Gunnels University of Texas Cockrell School of Engineering