The Global Digital Format Registry (GDFR) Project CNI Fall Task Force Meeting Washington, DC, December 10-11, 2007 The Global Digital Format Registry (GDFR) Project Stephen Abrams Harvard University Andreas Stanescu OCLC

Digital preservation and format Preservation is concerned with ensuring access to managed digital assets over time Thus, preservation activities are focused on Viability Fixity Authenticity Interpretability Renderability The last two are primarily a function of format

Without format typing, all content is opaque ffd8ffe000104a46494600010201 008300830000ffed0fb050686f74 6f73686f7020332e30003842494d 03e90a5072696e7420496e666f00 0000007800000000004800480000 000002f40240ffeeffee03060252 0347052803fc0002000000480048 0000000002d80228000100000064 000000010003030300000001270f 0001000100000000000000000000 0000600800190190000000000000 0000000000000000000000000000 0000000000000000000000003842 494d03ed0a5265736f6c7574696f 6e0000000010008313a3000200 ...

Without format typing, all content is opaque ffd8ffe000104a46494600010201 008300830000ffed0fb050686f74 6f73686f7020332e30003842494d 03e90a5072696e7420496e666f00 0000007800000000004800480000 000002f40240ffeeffee03060252 0347052803fc0002000000480048 0000000002d80228000100000064 000000010003030300000001270f 0001000100000000000000000000 0000600800190190000000000000 0000000000000000000000000000 0000000000000000000000003842 494d03ed0a5265736f6c7574696f 6e0000000010008313a3000200 ... SOI APP0 JFIF 1.2 APP13 IPTC APP2 ICC DQT SOF0 183x512 DRI DHT SOS ECS0 RST0 ECS1 RST1 ECS2 ...

Without format typing, all content is opaque ffd8ffe000104a46494600010201 008300830000ffed0fb050686f74 6f73686f7020332e30003842494d 03e90a5072696e7420496e666f00 0000007800000000004800480000 000002f40240ffeeffee03060252 0347052803fc0002000000480048 0000000002d80228000100000064 000000010003030300000001270f 0001000100000000000000000000 0000600800190190000000000000 0000000000000000000000000000 0000000000000000000000003842 494d03ed0a5265736f6c7574696f 6e0000000010008313a3000200 ... SOI APP0 JFIF 1.2 APP13 IPTC APP2 ICC DQT SOF0 183x512 DRI DHT SOS ECS0 RST0 ECS1 RST1 ECS2 ... Edward Burne-Jones (British, 1833-1898) The Days of Creation: the First Day, 1870-1876 Watercolor and gouache, 102.2×35.5 cm Fogg Art Museum, Harvard University, 1943.454 Bequest of Grenville L. Winthrop

Global Digital Format Registry “The Global Digital Format Registry (GDFR) will provide sustainable services to collect, review, store, discover, and deliver significant representation information about digital formats.” Centrally-organized collection and review Distributed storage, discovery, and delivery on a network of independent, but cooperating registries

What is a format? “A serialized encoding of an abstract information model” Encompasses the nominal sense of “file format” as well as a range of conceptual entities from the micro to the macro level IEEE 754 floating point number File system In both case, there are well-defined syntactic and semantic rules for mapping from information to bits, and back again

What’s wrong with MIME types?

What’s wrong with MIME types? Non-standardized documentation Intended for human, not machine consumption Coarse granularity image/tiff vs. TIFF 4.0 – 6.0 Baseline Class B, G, P, R Extension Class Y TIFF/EP TIFF/IT with file types CT, LW, HC, MP, BP, BP, BL, FP Exif 2.0 – 2.2 GeoTIFF TIFF/FX DNG

GDFR project Two DLF-sponsored invitational workshops University of Pennsylvania, January 2003 Washington, March 2003 Two independent demonstration projects FRED [John Ockerbloom, University of Pennsylvania] tom.library.upenn.edu/fred/ FOCUS [Joseph JaJa, University of Maryland] www.umiacs.umd.edu/~joseph/focus-archiving06.pdf FRED, Format Registry Demonstrator, TOM (Typed Object Model) FOCUS, Format Curation Service (LDAP)

GDFR project Harvard University Library (HUL) funded for 2 years by the Andrew W. Mellon Foundation Staffing and technical work subcontracted by HUL to OCLC (July 2006)

GDFR project oversight Technical Working Group (TWG) Bibliothèque nationale de France British Library California Digital Library Digital Curation Centre, UK Library of Congress National Archives, UK National Archives and Records Administration National Library of Australia National Library of New Zealand Stanford University University of Pennsylvania

General development goals A generalized registry framework, specialized for the distributed GDFR application Based on well-known products and protocols Human and machine interfaces Full information content expressible in XML form, and can be re-instantiated from that expression Platform independence Globally fault tolerant Open source

GDFR data model Consistent with PRONOM registry

Identifiers Canonical, GDFR-assigned identifier “info” URI info:rfa/gdfr1/Formats/1 Other well-known identifiers Common name “TIFF”, “Tagged Image File Format” MIME type image/tiff PRONOM identifier info:pronom/fmt/7 Library of Congress Format Description Document (FDD) identifier fdd000022

Classification scheme Eight facets Genre (required) text, still-image, sound, aggregate, … Role (required) family, file-format, encoding, serialization Composition unitary, container-bundle, container-wrapper Form binary, text Constraint structured, unstructured Basis sampled, symbolic Domain astronomy, cad-cam, gis, web-archive, … Transform compression, encryption, message-digest, …

Classification scheme Examples TIFF (Tagged Image File Format) genre:still-image role:family composition:container-wrapper form:binary basis:sampled LZW (Liv-Zempel-Welch) genre:still-image role:encoding transform:compression SVG (Scalable Vector Graphics) genre:still-image role:file-format form:text basis:symbolic

Signatures External signatures Internal signatures File extension Mac OS type Mac OS X Uniform Type Identifiers (UTI) Internal signatures “Magic numbers” Required vs. optional Fixed vs. restricted vs. unrestricted

Grammar Formal description of the syntactic grammar underlying a format, expressed in some formal typed notation BNF Backus-Naur Form BSDL MPEG-21 Bitstream Syntax Description Language DFDL Data Format Description Language EAST CCSDS 644.0-B-2 XCEL Extensible Characterisation Extraction Language

Assessment Assessment of a format, expressed in some formal typed notation Cornell Virtual Remote Control (VRC) DTSC PANIC Library of Congress Sustainability, Quality, Function (SQF) National Library of Australia AONS OCLC INFORM

Documentation Specification documents (and software files) can be managed and distributed in the network Applicable only in cases of public domain resources or if explicit permission is granted by rights holders Other documents (and software) will be referenced by full citation, including actionable links where possible Mechanism for individuals or institutions to register locally-held copies, with terms of use

Software Format role Input, output Process type Characterize, create, edit, identify, … Enables discovery of transformative processing chains

Relationships Modification BWF → WAVE Definition NITF → XML DTD Extension DNG → TIFF 6.0 Restriction PDF/A → PDF 1.4 Definition NITF → XML DTD Requisite XML → Relax NG Containment ZIP → * Equivalence DXF (ASCII) → DXF (binary) Version Word 97 → Word 6.0 Affinity SPIFF → JPEG

GDFR node Based on the OCLC IWSA / RFA framework

GDFR node Java, Apache/Tomcat, Berkeley DB XML GNU LGPL license Including pre-existing OCLC technology and technology newly-developed for the project Release schedule v0.1 (alpha) March 23, 2007 v0.1 (beta) June 14, 2007 v1.0 June 30, 2007 v1.1 August 12, 2007 v1.3 September 17, 2007 v1.3.1 October 26, 2007

GDFR node

GDFR node

GDFR node

GDFR network Peer-to-peer network of independent, but cooperating registries communicating over a common protocol

GDFR network Public notification of the availability of new data RSS feed available at well-known public address to which remote nodes can subscribe Remote harvesting of local data OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting) Initially, a single source (root node) for all new data

Project status Extensive internal testing of GDFR software in a stand-alone mode Current project activities are focused on Implementing the distribution and synchronization functions Building the network Data acquisition Succession planning

Initial population Manual addition is possible, but time consuming Automated update using Atom What sources are available for bulk population? PRONOM registry www.nationalarchives.gov.uk/pronom Library of Congress Format Description Documents (FDD) www.digitalpreservation.gov/formats/fdd/descriptions.shtml Unix / Linux magic(4) database

Subsequent population RFC 2026, Internet Standards Process www.ietf.org/rfc/rfc2026.txt “Iterations of review by the ... community and revision based upon experience” Draft distribution and public discussion Approval by “area” editors Release to the network for distribution

Sustainability The technological solution is the (relatively) easy part, but… The technology is expendable The important point is for the data to survive, evolve, and expand

Governance and succession Mellon funding was for technical work only At the end of the two year project… Harvard will continue maintenance for up to two years Library of Congress has agreed to be a care-taker agency until a permanent body is identified NARA ERA, Ken Thibodeau, Robert Chadduck, Richard Steinbacher

Governance and succession NARA GDFR governance investigation Part of the Electronic Records Archives (ERA) initiative GDFR Governance Workshop, November 2007 Bibliothèque et Archives, Canada • NARA Corp. for National Research Initiatives • NASA Digital Curation Centre, UK • NIST Digital Library Federation • National Library of Australia General Services Administration • National Library of New Zealand Georgia Institute of Technology • San Diego Supercomputer Center Government Printing Office • Stanford University Harvard University • Statens Archiv, Sweden IBM Watson Research Center • Tessalla Support Services Koninklijke Bibliotheek, Netherlands • University of Pennsylvania Library of Congress MIT

Administrative considerations Policy Who (and how many) can join the network? What are the eligibility requirements? What are the rights and obligations of membership? Technical Who will maintain and enhance the data model? Who will maintain, enhance, distribute, and support the software?

Administrative considerations Data Who will contribute data? Who will vouch for data authenticity? Who will ensure data integrity? Financial What are the real human and system costs associated with GDFR operation? Who pays, and how?

Summary The GDFR is an enabling technology that will support digital repository and preservation activities Supports the strong typing of digital assets at an appropriate level of granularity Enables the future recovery of the syntax and semantics associated with typed digital assets A means to pool and redistribute the expertise of the international digital preservation community

For more information… www.formatregistry.org stephen_abrams@harvard.edu andreas_stanescu@oclc.org