Search Interoperability, OAI, and Metadata Sarah Shreeves University of Illinois at Urbana-Champaign Basics and Beyond Grainger Engineering Library April 18, 2005

Basics and Beyond Grainger Engineering Library Scenario: A teacher is putting together a lesson plan comparing immigration in the early 20 th century to immigration and wants to include a variety of primary sources

April 18, 2005Basics and Beyond Grainger Engineering Library IMLS funded digital collections with relevant content

April 18, 2005Basics and Beyond Grainger Engineering Library Search interoperability “the ability to perform a search over diverse sets of metadata records and obtain meaningful results.” – Priscilla Caplan Metadata Fundamentals for All Librarians

April 18, 2005Basics and Beyond Grainger Engineering Library Keys to Search Interoperability Communication protocol (Z39.50, OAI, etc.) Organizational commitment Standards And More Standards

April 18, 2005Basics and Beyond Grainger Engineering Library Sharing metadata : Federated search The distributed databases are searched directly. Mill? My resource 04 For Example: Z39.50, SRU/SRW

April 18, 2005Basics and Beyond Grainger Engineering Library Sharing metadata : Data aggregation The user searches a pre-aggregated database of metadata from diverse sources. Mill? My resource 04 For Example: Search engines, union catalogs, OAI

April 18, 2005Basics and Beyond Grainger Engineering Library Open Archives Initiative Protocol for Metadata Harvesting The OAI-PMH is a tool Moves metadata (not content for the most part yet ) from a data provider to a service provider (or harvester) A set of rules that defines the communication between two systems (like FTP and HTTP) Facilitates the aggregation of metadata (like a union catalog)

April 18, 2005Basics and Beyond Grainger Engineering Library Basic OAI-PMH Concepts “Aggregated search” rather than “Federated search” Data providers – support OAI PMH as a means to expose metadata Service providers – ‘harvests’ metadata from data providers via the OAI-PMH OAI-PMH based upon HTTP and XML OAI-PMH requires use of simple Dublin Core BUT supports and encourages use of other metadata schemas Unique and Persistent Identifiers and a Datestamp for each OAI record

April 18, 2005Basics and Beyond Grainger Engineering Library OAIster: CIC Metadata Portal:

April 18, 2005Basics and Beyond Grainger Engineering Library How OAI Works (Technically) 6 distinct ‘verbs’ or requests OAI requests are sent via HTTP Responses are sent in valid XML Dig. Mngt. Sys. OAI H A R V E S T E R OAI Data P R O V I D E R Service Provider Data Provider HTTP Request (OAI Verb) HTTP Response (Valid XML) AGGREGATEDAGGREGATED METADATAMETADATA

April 18, 2005Basics and Beyond Grainger Engineering Library Examples of OAI Service Providers OAIster: Engineering, Computer Science, and Physics: Open Language Archives Community:

April 18, 2005Basics and Beyond Grainger Engineering Library OAI “VERBS” Identify ListMetadataFormats ListSets ListIdentifiers GetRecord ListRecords

April 18, 2005Basics and Beyond Grainger Engineering Library Challenges for the OAI Community No best practices (yet) ‘Shareablity of metadata’ Heterogeneity of items described Loss of Context / Information loss Knowledge structures differ so…. Native metadata schemas differ Controlled vocabularies differ Use and presentation of items differ

April 18, 2005Basics and Beyond Grainger Engineering Library OAI ≠ Dublin Core DC is OAI’s lowest common denominator BUT OAI supports & encourages use of other community-driven metadata schemas

April 18, 2005Basics and Beyond Grainger Engineering Library Metadata Interoperability Semantics What is the metadata format used? Mapping from one format to another Content rules How are values for the metadata elements selected and represented? Syntax How are the metadata elements encoded in machine readable form? Documentation

April 18, 2005Basics and Beyond Grainger Engineering Library

April 18, 2005Basics and Beyond Grainger Engineering Library Metadata for different communities

April 18, 2005Basics and Beyond Grainger Engineering Library Metadata for different communities

April 18, 2005Basics and Beyond Grainger Engineering Library Loss of Context: Record in OAI aggregation

April 18, 2005Basics and Beyond Grainger Engineering Library Context: Record in native database

April 18, 2005Basics and Beyond Grainger Engineering Library Loss of context / data

April 18, 2005Basics and Beyond Grainger Engineering Library Loss of context / data

April 18, 2005Basics and Beyond Grainger Engineering Library Granularity of Description: Excerpt of Metadata Record Describing “American Woven Coverlet”

April 18, 2005Basics and Beyond Grainger Engineering Library Granularity of Description: Excerpt of Metadata Record Describing "Cotton coverlet with embroidered butterfly design"

April 18, 2005Basics and Beyond Grainger Engineering Library What does this record represent? identifier: idx?view=entry;subview=detail;cc=fish3ic;entryid=X- 0802;viewid=1004_112http://images.umdl.umich.edu/cgi/i/image/image- idx?view=entry;subview=detail;cc=fish3ic;entryid=X- 0802;viewid=1004_112 publisher: UMMZ Fish Division format: jpeg type: image subject: ;1926;0812;18;Trib. to Sixteen Cr. Trib. Pine River, Manistee R.;R10W;S26; S27;JAM ;05;T21N;1926/05/18 language: UND description: Flora and Fauna of the Great Lakes Region;

April 18, 2005Basics and Beyond Grainger Engineering Library

April 18, 2005Basics and Beyond Grainger Engineering Library Data providers can: Create metadata for interoperability Reusable metadata - think beyond your local users and environment Use well structured and defined schemas; move beyond simple DC Use and identify controlled vocabularies Document, document, document

April 18, 2005Basics and Beyond Grainger Engineering Library Service Providers can… Analyze metadata and cluster and normalize some aspects Provide contextual information (such as collection descriptions) Custom interfaces and selective views for target audiences / domains

April 18, 2005Basics and Beyond Grainger Engineering Library Contact Information Sarah Shreeves Project Coordinator, IMLS Digital Collections and Content Project University of Illinois at Urbana-Champaign Presentation available: