Summer Institute on Data Curation: Digital Preservation & Standards Jerome McDonough Asst. Professor, GSLIS June 4, 2008

I love standards. There are so many of them to choose from.

Standards & Sustainability  Disclosure: Are complete specifications available? For free?  Adoption: To what extent is the standard already used?  Documentation: Is the specification clear and straightforward? Are there additional resources to assist in understanding the standard?

Standards & Sustainability  External Dependencies: To what extent does use of the standard rely on particular hardware or software? On other standards? On other non-standards?  Impact of Patents: If patents cover some or all of the standard, are licensing issues likely to complicate use of the standard?  Technological Protection Measures: Does the standard rely on technological protection measures which will inhibit your ability to preserve data? Tip of the hat to Library of Congress Sustainability Of Digital Formats Site

Part I: How to Operate an Archive

Open Archival Information System Reference Model  Developed by the Consultative Committee For Space Data Systems  Adopted as ISO 14721:2003  Available at /650x0b1.pdf /650x0b1.pdf  Provides definitions of components of an archive, their relationship to each other, a set of mandatory responsibilities for an archive, and both functional and data models.

OAIS Reference Model: Mandatory Responsibilities  Negotiate for an accept information from producers  Obtain sufficient control of information to ensure long-term preservation (including necessary IP permissions and authority to migrate)  Determine which communities should be the Designated Communities and should be able to understand the information provided  Ensure that the information to be preserved is independently understandable to the designated community (i.e., they can understand it without the assistance of experts who created it ).  Follow documented policies and procedures ensuring information is preserved against reasonable contingencies  Make the information available to the designated community

OAIS Functional Model

OAIS Functional Model: Ingest

OAIS Functional Model: Archival Storage

OAIS Functional Model: Data Management

OAIS Functional Model: Access

OAIS Functional Model: Preservation Planning

OAIS Functional Model: Administration

OAIS Data Model

Part II: How to Create Content for an Archive

Archival Content  A syllogism to ponder:  No digital media can be read without a hardware device designed to read the media format.  It is exceedingly rare for a hardware device intended to read a specific digital media format to be manufactured for more than 30 years, and many have had shorter lifespans.  Therefore, if your content is not device independent, it is not really archival.

Archival Content: Text  Some Issues to Consider When Examining Text Standards  Technical aspects of character encoding  Character Repertoire (Script & Language Support)  Line Break Handling & Line Orientation  Indexing  Formatting  Other processing

Archival Content: Text  A Standard for Characters  Unicode ISO/IEC  Two variable length encodings (UTF-8, UTF-16) and a fixed length encoding (UTF-32). In UTF-8, byte order is not an issue. In UTF-16 and UTF-32, big-endian and little-endian encodings are supported.  Over 100K characters, supporting 75 different scripts and many additional symbols and diacritics, with room for expansion to 1,114,112 characters.  Support for a variety of line breaking mechanisms  Support for different text directionality, including algorithms specifying the appropriate handling of text of mixed directionality

Archival Content: Text  A Standard for Syntax  XML (World Wide Web Consortium)  Standards for Semantics  Chemical Markup Language, Chemical Industry Data Exchange  Astronomical Markup Language, Astronomical Dataset Markup Language, Astronomical Instrument Markup Language  Earth Science Markup Language, Geography Markup Language, NetCDF Markup Language, ArcGIS Markup Language  MathML  Etc., etc., etc….

Archival Content: Images  Some Issues to Consider When Examining Image Standards  Color Depth  Color Space  Color Management  Image Resolution Scalability  Compression

Archival Content: Images  Tagged Image File Format (TIFF) to 64-bit color depth, supports grayscale, RGB, YCbCr, CMYK and CIELab color spaces, supports embedded ICC color profiles, raster format, supports uncompressed as well as lossless and lossy DCT-based compression  JPEG 2000 (ISO/IEC 15444) bits per channel with multiple channels (including alpha & transparency), supports wide array of color spaces with sRGB and sYCC as defaults, supports ICC color profiles, raster format, supports uncompressed as well as lossless and lossy wavelet based compression  Scalable Vector Graphics uses sRGB color spaces, supports ICC Color Profiles, vector format

Archival Content: Audio/Video  Some Issues to Consider when Examining Audio/Video Standards  Audio sampling rate  Audio bit depth  Video frame rate  Video color space/depth  Compression  Good News: Audio/Video is a bit more standardized than text/image world  Bad News: Lossless digital audio is rare; lossless digital video is almost nonexistent.

Archival Content: Audio/Video  Broadcast WAVE Audio (EBU Standard N )  For video, picture is less clear. Proprietary solutions dominate market. Many of these (e.g., QuickTime, WMV) do support lossless image frame and audio data. MXF, a SMPTE standard, is gaining some traction in digital library circles (and the movie industry)

Archival Content: Data  Some disciplinary de facto standards (e.g., Chemical Markup Language). Cover Pages ( is a good source for information on many of the major ones.  No single standard for general use for data encoding, although many contenders

Archival Content: Data  Binary Format Description Language (BDFL) - - XML language based on the Extensible Scientific Interchange Language (XSIL) that supports documentation of binary and ASCII data  eXtensible Data Format (XDF) -- scientific data format supporting hierarchical data structures, N-dimensional arrays, scalar and vector fields, user-defined coordinate systems

Archival Content: Data  Data Format Description Language (DFDL) -- A language for describing the structure or binary and character encoded data to expose their structure, format and metadata so that machine processes can work upon them.  Data Documentation Initiative (DDI) -- An effort by the ICPSR at Univ. of Michigan to develop an XML format for documenting social science data sets. XML files can be used to produce either bibliographic descriptions of data sets or SAS/SPSS/STATA data definition statements.

Archival Content: Data  Hierarchical Data Format (HDF5) -- General purpose file format (with supporting software library) for storing scientific data, developed by NCSA. Uses two fundamental structures, groups and data sets, where a data set is an N-dimensional array of data elements with metadata.

Archival Content: Paper  ANSI/NISO Z , Permanence of Paper for Publications and Documents in Libraries and Archives  ISO , Information and documentation -- Paper for documents -- Requirements for permanence

Part III: How to Create Metadata for an Archive

Metadata: Identifiers  Persistence is important, but…  Clarity on what is being identified may be more important (or, why an OpenURL is not a call number).  Standards proliferate in this space; choice of any identifier may depend on:  Social concerns (for whom am I identifying something?)  Identifier/address resolution (how do I find a copy/item using this identifier?)

Metadata: Structural  Metadata intended to identify the components of an object and their relationship to each other in order to support the object’s navigation and use  Metadata Encoding & Transmission Standard (METS)  MPEG-21 Digital Item Declaration Language  XML Formatted Data Units (XFDU)  OAI-ORE

Metadata: Provenance  Metadata documenting the origins and life- cycle of a digital object  PREMIS Data Dictionary for Preservation Metadata 2.0  Joint project of OCLC & RLG  Defines metadata element set that “supports the viability, renderability, understandability, authenticity and integrity of digital objects in a preservation context.”

Metadata: Provenance The PREMIS Data Model

Metadata: Provenance  PREMIS Object Metadata:  Identifier  Category  Preservation Level  Significant Properties  Characteristics (fixity, size, format, etc.)  Original Name  Storage  Environment  Signature  Relationships to other Objects, Events, Rights

Metadata: Provenance  PREMIS Event Metadata  Identifier  Type  Date & Time  Details  Outcome  Relationship to Agents and Objects

Metadata: Provenance  PREMIS Agent Metadata  Identifier  Name  Type

Metadata: Provenance  PREMIS Rights Metadata  Rights Statement  Rights Basis  Copyright Information  License Information  Statute Information  Rights Granted  Relationship to Objects and Agents

Metadata: Administrative  Technical Metadata  Z39.87 and MIX  Technical Metadata for Text (TextMD)  AES-X098 Administrative Metadata for Audio Objects  SMPTE RP Metadata Dictionary  Rights Metadata  Standards, yes. That you want to use, no.

Metadata: Descriptive  Issues to consider:  Nature of object to be described  Real purpose(s) of description  Community(ies) that will utilize description  Supporting standards of descriptive practice and controlled vocabularies

Metadata: Descriptive  Library/Archives/Museums/Educators  MARC, MODS, Dublin Core  EAD  VRA Core, CDWA  IEEE LOM  Data Repositories  Data Documentation Initiative  Content Standard for Digital Geospatial Metadata  Darwin Core  Access to Biological Collection Data (ABCD)

How to Evaluate an Archive

Evaluating Archives  Trustworthy Repositories Audit & Certification (TRAC) Criteria & Checklist  =58&l3=162&l4=91 =58&l3=162&l4=91  Digital Repository Audit Method Based on Risk Assessment (DRAMBORA) 

Exercise: URLs  Images  yce.tif yce.tif  yce.jp2 yce.jp2

Exercise: URLs  METS Schema, Documentation, Namespace     PREMIS Schema, Documentation, Namespace     MIX Schema, Documentation, Namespace   oject_key=b897b0cf3e2ee526252d9f830207b3cc9f3b6c2c oject_key=b897b0cf3e2ee526252d9f830207b3cc9f3b6c2c 