A METS Application Profile for Historical Newspapers Morgan Cundiff Network Development and MARC Standards Office Library of Congress

Outline XML and Standards Definition of METS Definition of METS Profiles Use of MODS relatedItem element Draft METS Profile for Historical Newspapers Parting Thoughts

XML “XML has become the de-facto standard for representing metadata descriptions of resources on the Internet.” Jane Hunter Working towards MetaUtopia - A Survey of Current Metadata Research

The Importance of Standards “In moving from dispersed digital collections to interoperable digital libraries, the most important activity we need to focus on is standards… most important is the wide variety of metadata standards [including] descriptive metadata… administrative metadata…, structural metadata, and terms and conditions metadata…” Howard Besser The Next Stage: Moving from Isolated Digital Collections to Interoperable Digital Libraries

What is METS? METS is an XML Schema designed for the purpose of creating XML document instances that express the hierarchical structure of digital library objects, the names and locations of the files that comprise those objects, and the associated metadata. METS can, therefore, be used as a tool for modeling real world objects, such as particular document types.

What are the 7 Sections of a METS Document? <metsHdr/> <dmdSec/> <amdSec/> <fileSec/> <structMap/> <structLink/> <behaviorSec/> </mets>

The Descriptive Metadata Section with mdWrap <mets> <dmdSec> <mdWrap> <xmlData> <!-- insert data from different namespace here --> </xmlData> </mdWrap> </dmdSec> <fileSec></fileSec> <structMap></structMap> </mets>

The Descriptive Metadata Section with MODS as extension schema <mets:mets> <mets:dmdSec> <mets:mdWrap> <mets:xmlData> <mods:mods></mods:mods> </mets:xmlData> </mets:mdWrap> </mets:dmdSec> <mets:fileSec></mets:fileSec> <mets:structMap></mets:structMap> </mets:mets>

The Descriptive Metadata Section with MODS and relatedItem elements <mets:mets> <mets:dmdSec> <mets:mdWrap> <mets:xmlData> <mods:mods> <mods:relatedItem type=“constituent”> <mods:relatedItem type=“constituent”></mods:relatedItem> </mods:relatedItem> </mods:mods> </mets:xmlData> </mets:mdWrap> </mets:dmdSec> <mets:fileSec></mets:fileSec> <mets:structMap></mets:structMap> </mets:mets>

MODS relatedItem element Child element to MODS relatedItem element has same content model as mods (titleInfo, name, subject, physicalDescription, note, etc) The relatedItem element makes it possible to create very rich analytic descriptions for contained works within a MODS records relatedItem element is repeatable and it can be nested recursively (thus making it possible to build a hierarchical tree structure) relatedItem elements make it possible to associate descriptive data with any structural element.

Use of MODS relatedItem element to express logical structure <mods:mods> <mods:titleInfo> <mods:title>Baltimore Sun</mods:title> </mods:titleInfo> <mods:relatedItem type="constituent"> <mods:title>Sports</mods:title> <mods:relatedItem type="constituent"> <mods:title>O’s Split Beantown Twi-niter</mods:title> </mods:relatedItem> <mods:title>Chisox Nip Tribe</mods:title> </mods:mods>

METS document with two hierarchies (logical and physical) <mets:mets> <mets:dmdSec> <mets:mdWrap> <mets:xmlData> <mods:mods> <mods:relatedItem> <mods:relatedItem></mods:relatedItem> </mods:relatedItem> </mods:mods> </mets:xmlData> </mets:mdWrap> </mets:dmdSec> <mets:fileSec></mets:fileSec> <mets:structMap> <mets:div> <mets:div></mets:div> </mets:div> </mets:structMap> </mets:mets>

Linking in METS Documents (XML ID/IDREF links) DescMD mods relatedItem AdminMD techMD sourceMD digiprovMD rightsMD fileGrp file StructMap div fptr

Linking in METS Documents (XML ID/IDREF links) DescMD mods relatedItem AdminMD techMD sourceMD digiprovMD rightsMD fileGrp file StructMap div fptr

Linking in METS Documents (XML ID/IDREF links) DescMD mods relatedItem AdminMD techMD (mix) sourceMD digiprovMD rightsMD fileGrp file StructMap div fptr

Linking in METS Documents (XML ID/IDREF links) DescMD mods relatedItem AdminMD techMD (mix) sourceMD digiprovMD rightsMD fileGrp file StructMap div fptr

Linking in METS Documents (XML ID/IDREF links) DescMD mods relatedItem AdminMD techMD (mix) sourceMD digiprovMD rightsMD fileGrp file StructMap div fptr

Linking in METS Documents (XML ID/IDREF links) DescMD mods relatedItem AdminMD techMD (mix) sourceMD digiprovMD rightsMD fileGrp file StructMap div fptr

What is a METS Application Profile? “METS Profiles are intended to describe a class of METS documents in sufficient detail to provide both document authors and programmers the guidance they require to create and process METS documents conforming with a particular profile.” A profile is expressed as an XML document. There is a schema for this purpose. The profile expresses the requirements that a METS document must satisfy. A sufficiently explicit METS Profile may be considered a “data standard”. Note: A METS Profile is a human-readable prose document and is not intended to be “machine actionable”.

METS Profile for Historical Newspapers [draft] The METS Profile for Historical Newspapers specifies how METS documents representing digitized historical newspapers should be encoded. Note that the profile is to be used to represent a single issue of a newspaper. The profile uses MODS to express the logical structure of a newspaper issue, and uses the METS structMap to express the physical structure of the newspaper issue. [draft abstract] URL to find Profile and related documents: http://www.loc.gov/standards/mets/test/ndnp/profile_notes.html mcundiff@loc.gov

METS Profile (features) Represents one issue of a newspaper.

METS Profile (features) The Profile presumes the use of alto files (or some equivalent) where the zones on the corresponding digital image (expressed as coordinates) are correlated to the corresponding logical entity (e.g. article or paragraph) and also to the corresponding OCR text.

METS Profile (features) The Profile maintains a strict separation between logical entities and physical entities.

METS Profile (features) The primary logical entities are issue, issue section, article, article section, illustration, and advertisement. The top-level MODS record describes the issue. The other primary logical entities (issue section, article, article section, illustration, and advertisement) are described in a heirarchy of MODS relatedItem elements.

METS Profile (features) Logical structure is represented using MODS in the METS dmdSec. It is necessary to use the latest version (version 3.2) of MODS.

Hierarchy of Logical Entities issue issue section article (or article-like entity) paragraph illustration (photograph, drawing, map, table) article section illustration advertisements article

METS Profile (features) The primary logical entities are expressed as values of the MODS genre element.

Use of MODS relatedItem element to express logical structure <mods:mods> <mods:titleInfo> <mods:title>Baltimore Sun</mods:title> <mods:genre>newspaper</genre> </mods:titleInfo> <mods:relatedItem type="constituent"> <mods:title>Sports</mods:title> <mods:genre>section</genre> <mods:relatedItem type="constituent"> <mods:title>O’s Split Beantown Twi-niter</mods:title> <mods:genre>article</mods:genre> <mods:title>Aparicio puts tag on Jensen to end 7th</mods:title> <mods:genre>photograph</genre> </mods:relatedItem> </mods:mods>

METS Profile (features) The allowable genre values (for Profile compliance) are listed in Newspaper Genre Terms [draft].

METS Profile (features) It is also possible to tag subparts of the primary logical entities. The typical example of this is tagging the paragraph. This is accomplished using the MODS part element.

METS Profile (features) There are only three physical entities. They are: issue, page, and pageRegion.

METS Profile (features) The physical entities are represented in the structMap section of the METS document as div types (div type="news:page"). There is only one structMap.

METS Profile (features) Page regions are correlated to the corresponding logical entity by means of an IDREF link. Note that one or more page regions may correspond to a single logical entity. This makes it possible to make the necessary associations when the logical entity is split into more than one physical entity, e.g. when a paragraph is continued on the next column or an article is continued on a different page.

METS Profile (features) Example document http://memory.loc.gov/cocoon/diglib/loc.news.sr.1002/default.html

Parting Thoughts Agreement on a data standard (such as a METS profile) will facilitate interoperability. Interoperability can be between any two agents (digital library applications, preservation repositories, search and retrieval systems, etc.) Newspaper community has a “quality vs. quantity” dilemma. Large volume of material to be digitized necessitates automatic processing. Automatic processing produces dirty data and less satisfying results. High quality processing (requiring more human intervention) is more expensive but produces far better results and pays dividends far into the future (the data will be used over and over without additional cost).