1. OAI & NSDL Research at Grainger Briefing to UIUC Library Faculty 15 April 2003
Timothy W. Cole
William H. Mischo

2. Projects Open Archives Initiative: National Science Digital Library:
Illinois OAI Metadata Harvesting (Mellon)
IMLS Digital Collections & Content (IMLS)
Grainger OAI Resources in Science & Engineering
National Science Digital Library:
2nd Generation Math Resources (NSF / DUE)

3. OAI Protocol for Metadata Harvesting
Harvesting approach to interoperability at metadata level
Divides world into Metadata Providers & Service Providers
Builds on HTTP, XML, & Dublin Core

4. OAI is a tool All about moving metadata (not data) around
A building block, useable by many communities – supports new models of scholarly communication
Can facilitate, in some cases enable, advanced digital library services & functions
Assumes widely distributed content, but centralized indexing(!) – requires critical mass
Providers build once, share many times
Purpose of OAI is to foster interoperability

5. Harvesting vs. Federation
Competing approaches to interoperability
Federation is when services are run remotely on remote data (e.g. Broadcast Searching)
Harvesting is when data/metadata is transferred from the remote source to the destination where the services are located (e.g. Union Catalogs)
Federation requires more effort at each remote source but is easier for the central system and vice versa for harvesting
OAI focuses on harvesting

6. Reliance on HTTP, XML, DC
OAI is a REpresentational State Transfer (REST) protocol – i.e., URL-based
Z39.50, Web services, SOAP are RPC-based
OAI requests are sent via the HTTP protocol using GET or POST
OAI responses are valid XML documents
XML allows validation, increases reliability of what’s harvested (in terms of structure)
DC is OAI’s Lowest Common Denominator
Communities encouraged to use additional schemas

7. How OAI Works OAI “VERBS” Identify ListMetadataFormats ListSets
ListIdentifiers
ListRecords
GetRecord
Service Provider Metadata Provider H A R
V E S T E R
R E P O S I T O R Y
OAI
HTTP Request
OAI
(OAI Verb)
HTTP Response
(Valid XML)

8. As Compared to Z39.50 Z39.50 OAI Content (Objects) Distributed
World View
Bibliographic
Object Presentation
Data provider
Searching is
Centralized
Search done by
Service provider
Metadata searched is
Up to date
Stale
Semantic Mapping
When searching
Metadata delivery

9. Mellon-OAI Project
Create a web portal to scholarly information resources in cultural heritage harvested via OAI
Primary objectives:
Develop & make available OAI harvesting tools
Build harvesting and search services
Investigate viability and utility of searching OAI harvested resources
Explore issues of advanced search/indexing/display
Explore user needs & metadata usage patterns
Identify critical issues and best practices for using OAI with cultural heritage material

10. Mellon-OAI Achievements
Developed harvesting tools (Open Source)
Refined data provider tools (Open Source)
Investigated logistics of harvesting activities
Investigated metadata provider usage of DC, EAD
Created XSL stylesheets for metadata transformations (MARC to DC; EAD to DC)
Experimented w/ configurations to address scalability & performance issues
Usability testing with students in College of Education

11. Metadata aggregation 39 providers (OAI-compliant and surrogates)
Metadata describing resources of 580 institutions (CIMI, CDP)
1.1 million original records
2.6 million including item-level records derived from EAD finding aids

12. IMLS Digital Collections & Content
Build registry of all National Leadership Grant collections with digital content.
Assist & guide NLG projects in making item-level metadata sharable using OAI.
Build repository, search & discovery tools for integrated access to content of NLG collections
Research best practices for sharing metadata about diverse digital content & supporting interests of diverse user communities.
Collaboration between UIUC Library, GSLIS, & IMLS

13. Project Sites UIUC OAI Cultural Heritage Repository
Mellon-OAI Project Site
IMLS DCC Project Site

14. National Science Foundation NSDL Program
National Science, Technology, Engineering, Mathematics Digital Library.
Coverage: K to Grey.
National system for distributed science education; characterized by a set of exemplary resource collections and services.
Highly competitive grants: 3 years, 339 proposals, 105 funded; three main categories: collections, services and targeted research.

15. 2nd Generation Math Resources
Collaboration with UIUC Library, Wolfram Research Inc., & COE Dept of Theoretical and Applied Mechanics.
Project Objectives:
Adding interactive and graphical content to two feature-rich Wolfram sites.
Generating and extracting OAI-compliant metadata, establishing OAI Provider site, adding mathematics controlled vocabulary terms.
Developing courseware and problem libraries for TAM courses.

16. Providing Metadata to NSDL
Exposing metadata via OAI
Preferred method for bringing metadata into the NSDL repository (requires little manual intervention)
Sending metadata via ftp
Enabling metadata "scraping"
Creating and editing directly to the NSDL metadata repository
See also: NSDL Metadata Primer

17. Wolfram Functions Web Site
Source HTML Page Derived Metadata
<dc:identifier> <dc:description>
<dc:date>
<dc:rights>

18. Wolfram Functions Web Site
Source HTML Head Extracted Metadata
<dc:title>
<dc:description>
<dc:subject>
<dc:format>
<html>
<head>
<title>Square root: Primary…</title>
<meta name='Description'
content='Primary definition …' >
<meta name='Keywords'
content='Sqrt, square root, …' >
<meta http-equiv='Content-Type'
content='text/html; charset=iso-…'>
</head> …

19. Sample Metadata File for a Wolfram Functions Web Page
<oai_dc:dc … >
<dc:title>Square root: Primary definition (formula …</dc:title>
<dc:subject>Sqrt</dc:subject>
<dc:subject>square root</dc:subject> …
<dc:description>Primary definition (2 formulas)</dc:description>
<dc:description><math … </math></dc:description>
<dc:date> </dc:date>
<dc:publisher>Wolfram Research, Inc.</dc:publisher>
<dc:type>Text</dc:type>
<dc:format>text/html; charset=iso </dc:format>
<dc:identifier>
<dc:identifier>
<dc:language>en</dc:language>
<dc:rights>© 2002 Wolfram Research, Inc.</dc:rights>
</oai_dc:dc>

20. The NSDL metadata repository
Core Integration Project – Cornell, Columbia, DLESE.
The metadata repository is a resource for service providers.
It holds information about every collection and item known to the NSDL.
Services
Users
Metadata repository
From “The NSDL Metadata Strategy,” A presentation by William Y. Arms and Diane I. Hillman. Available:
Collections

21. Working Assumptions The WWW is the primary medium (for now)
Content is a mix of “born digital” and analog
There is no lack of “great piles of ‘stuff’ ”
There is a need for “piles of great ‘stuff’ ”
The “unit” of content can and will shrink
Users will increasingly be creators, and vice versa
While much of the use will be “free”, there is a need to explore multiple models of sustainability
Experimental nature of distributed digital library building - “one library, many portals”
Increasingly, analog content is produced digitally (though still distributed in analog form).
Relative mix of content will shift towards more “born digital”.
Java applets are an example of content with small granularity, but potentially very high reusability. (Granularity inversely proportional to reusability.)
Implication of decreasing granularity is that more users can become creators: reusing, repurposing content.
Also providing commentary on content will also be a contribution.
“Free” as in valued as a “public good” and hence supported publicly.

22. Related Links http://mathworld.wolfram.com/
OAI Resources in Science & Engineering