1. CS431 guest lecture Simeon Warner
The Open Archives Initiative (OAI) and the Protocol for Metadata Harvesting (OAI-PMH)
CS431 guest lecture
Simeon Warner
Work of lots of people. I have worked especially closely with Carl Lagoze, Herbert Von de Sompel and Michael Nelson.
3 March 2004

2. Origins of the OAI
“The Open Archives Initiative has been set up to create a forum to discuss and solve matters of interoperability between electronic preprint solutions, as a way to promote their global acceptance. “
(Paul Ginsparg, Rick Luce & Herbert Van de Sompel )

3. What is the OAI now? “The OAI develops and promotes interoperability
standards that aim to facilitate the efficient
dissemination of content.” (from OAI mission statement)
Technological framework around OAI-PMH protocol
Application independent
Independent of economic model for content
Also … a community and a “brand”
(and you need it for an assignment due in April)
Notes:
Quote is the first sentence of the OAI mission statement.
Should not underestimate the success of the name “Open Archives Initiative”. Contains very good buzz-word “Open”. This name is often mis-understood: “archives” means just repository in this context, it has no implication of preservation beyond the idea that items in the repository are in some sense persistent; “Open” refers to open sharing of metadata, it is not necessary that the full-content be openly accessible for an archive to participate in the OAI.

4. DEF Eprints Search Service OAIster Search Service
Where does the OAI fit?
DEF Eprints Search Service
OAIster Search Service
DSpace
OAI
EPrints
DTU
Institutional Repository
OAI is based around the notion of a collection of repositories (archives; collections) that ‘want’ to make metadata about their collections available.
OAI provides the glue so the we an aggregated virtual collection.
Services can then be built on the aggregated collection.
Scholarly Publishing and Archiving on the Web
arXiv

5. OAI and Open Access There is “A” difference
Open Archives Initiative
Open Access
The OAI is not tied to a particular political agenda - technical focus
BUT… the OAI provides functionality that is essential for many Open Access proposals

6. OAI-PMH Data Provider (Repository) Service Provider (Harvester)
PMH -> Protocol for Metadata Harvesting
Simple protocol, just 6 verbs
Designed to allow harvesting of any XML metadata (schema described)
For batch-mode not interactive use
Data Provider (Repository)
Service Provider
(Harvester)
Protocol requests (GET, POST)
XML metadata

7. ? OAI for discovery User R1 R2 R3 R4 Information islands
A number of problems with the notion of many separate repositories:
User may not know where they are or how to select appropriate ones
2) User has to deal with peculiarities on interface to each repository
3) Information may be spread over more repositories than it is practical to user directly R4
Information islands

8. Metadata harvested by service
OAI for discovery
Service layer R1 R2
Search
service
User R3
Instead we insert a “service layer”. In this example a search service harvests metadata from all repositories and presents unified “portal” to user. R4
Metadata harvested by service

9. Global network of resources exposing metadata
OAI for XYZ
Service layer R1 R2
XYZ
service
User R3
Perhaps an alerting service. Perhaps even remove the user from the picture and use the same OAI framework to exchange preservation metadata for distributed archiving of open access content. R4
Global network of resources exposing metadata

10. OAI-PMH Data Model resource item has all available metadata identifier
about this sculpture
item
Notes:
The OAI-PMH is based on the “Resource-Item-Record” model.
Resource: Details of the resource are outside the scope of the OAI-PMH, resources may be digital objects, physics objects, actors (people, institutions), collections… In this example the resource is an Inuit sculpture which lives in the Montreal museum of fine arts.
Item: This is what resides in the repository. Logically contains all available metadata about the resource. In practice the repository (say an eprint archive) may actually contain the resource (eprint) and metadata could be extracted dynamically. In this example, the item might include metadata such as the name of the sculpture, the name of the sculptor, creation date, its size, geographical origin…
Records: Particular metadata records disseminated via the OAI-PMH, may be in many overlapping and/or non-overlapping formats.
Dublin Core
metadata
MARC21
branding
records
record has identifier + metadata format + datestamp

11. OAI-PMH and HTTP
Clear separation of OAI-PMH and HTTP: OAI-PMH uses HTTP as transport
all OK at HTTP level? => 200 OK
something wrong at OAI-PMH level? => OAI-PMH error (e.g. badVerb)
HTTP codes 302 (redirect), 503 (retry-after), etc. still available to implementers, but do not represent OAI-PMH events
Not REST like

12. Normal response note no HTTP encoding of the OAI-PMH request
<?xml version="1.0" encoding="UTF-8"?>
<OAI-PMH> ….namespace info not shown here
<responseDate> T08:55:46Z</responseDate>
<request verb=“GetRecord”… …>
<GetRecord>
<record>
<header>
<identifier>oai:arXiv:cs/ </identifier>
<datestamp> </datestamp>
<setSpec>cs</setSpec>
<setSpec>math</setSpec>
</header>
<metadata>
…..
</metadata>
</record>
</GetRecord>
</OAI-PMH>
note no HTTP encoding
of the OAI-PMH request

13. Error/exception response
<?xml version="1.0" encoding="UTF-8"?>
<OAI-PMH>
<responseDate> T08:55:46Z</responseDate>
<request>
<error code=“badVerb”>ShowMe is not a valid OAI-PMH verb</error>
</OAI-PMH>
with errors, only the correct
attributes are echoed in
<request>
Same schema for all responses,
including error responses.

14. OAI-PMH verbs Function Verb listing of a single record GetRecord
listing of N records
ListRecords
OAI unique ids contained in archive
ListIdentifiers
sets defined by archive
ListSets
metadata formats supported by archive
ListMetadataFormats
description of archive
Identify
metadata
about the
repository
harvesting
verbs
most verbs take arguments: dates, sets, ids, metadata formats
and resumption token (for flow control)

15. Information about the repository, start any harvest with Identify
Identify verb
Information about the repository, start any harvest with Identify
<Identify>
<repositoryName>Library of Congress 1</repositoryName>
<baseURL>
<protocolVersion>2.0</protocolVersion>
<deletedRecord>transient</deletedRecord>
<earliestDatestamp> T00:00:00Z</earliestDatestamp>
<granularity>YYYY-MM-DDThh:mm:ssZ</granularity>
<compression>deflate</compression>

16. Identifiers
Items have identifiers (all records of same item share identifier)
Identifiers must have URI syntax (defined by RFC, a type in XML schema)
Unless you can recognize a global URI scheme, identifiers must be assumed to be local to the repository
Complete identification of a record is baseURL+identifier+metadataPrefix+datestamp
<provenance> container may be used to express harvesting/transformation history

17. Datestamps
All dates/times are UTC, encoded in ISO8601, Z notation: T20:30:00Z
Datestamps may be either fill date/time as above or date only (YYYY-MM-DD). Must be consistent over whole repository, ‘granularity’ specified in Identify response.
Earlier version of the protocol specified “local time” which caused lots of misunderstandings. Not good for global interoperability!

18. Harvesting granularity
mandatory support of YYYY-MM-DD
optional support of YYYY-MM-DDThh:mm:ssZ (must look at Identify response)
granularity of from and until agrument in ListIdentifier/ListRecords must match

19. Sets
Simple notion of grouping at the item level to support selective harvesting
Hierarchical set structure
Multiple set membership permitted
E.g: repo has sets A, A:B, A:B:C, D, D:E, D:F
If item1 is in A:B then it is in A
If item2 is in D:E then it is in D, may also be in D:F
Item3 may be in no sets at all
Don’t use sets unless you have a good reason (selective harvesting)

20. resumptionToken
Protocol supports the notion of partial responses in a very simple way: Response includes a ‘token’ at the which is used to get the next chunk.
Idempotency of resumptionToken: return same incomplete list when resumptionToken is reissued
while no changes occur in the repo: strict
while changes occur in the repo: all items with unchanged datestamp
optional attributes for the resumptionToken: expirationDate, completeListSize, cursor

21. Record headers header contains set membership of item
<identifier>oai:arXiv:cs/ </identifier>
<datestamp> </datestamp>
<setSpec>cs</setSpec>
<setSpec>math</setSpec>
</header>
<metadata>
…..
</metadata>
</record>
eliminates the need for the “double harvest” 1.x required to get all records and all set information

22. Deleted records
What happens when a record (or item) is deleted from a repository? Would be nice if harvesters could find out.
Not necessarily guaranteed in OAI that harvesters will find out. Support made optional because of problems with legacy repositories (practical constraint).
Level of support expressed in Identify (no, persistent, transient)
Status expressed in header element,
<header status=“deleted”>…</header>

23. Harvesting strategy Issue Identify request
Check all as expected (validate, version, baseURL, granularity, comporession…)
Check sets/metadata formats as necessary (ListSets, ListMetadataFormats)
Do harvest, initial complete harvest done with no from and to parameters
Subsequent incremental harvests start from datastamp that is responseDate of last response

24. Changing Scholarly Communication
Traditional journal publishing combines functions: registration, certification, awareness, archiving.
How about eprints being the starting point of a new value chain in which the raw material - the non-certified eprint - is open access?
Other functions might be fullfilled by different networked parties. This requires a communication infrastructure: OAI-PMH may be part of this.
Presentations on OAI and Scholarly Communication at