IST Tutorial OAI and OAI-PMH for Beginners An introduction to the Open Archives Initiative and the Protocol for Metadata Harvesting Uwe Müller Humboldt University Berlin, Germany Andy Powell UKOLN, University of Bath

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners Agenda Part I History and overview Part II Technical introduction Coffee/tea break Part III Implementation issues – data provider and service provider Part IV Implementation issues – XML schema and supporting multiple record formats

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners Acknowledgements Some of the slides presented here are our own! Many of them have been kindly donated by (taken from!): Herbert Van de Sompel Carl Lagoze Michael Nelson Simeon Warner (and others probably!)

IST Tutorial OAI and OAI-PMH for Beginners An introduction to the Open Archives Initiative and the Protocol for Metadata Harvesting Part I: History and overview Andy Powell UKOLN, University of Bath

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I OAI roots… the roots of OAI lie in the development of eprint archives… arXiv, CogPrints, NACA (NASA), RePEc, NDLTD, NCSTRL each offered Web interface for deposit of articles and for end-user searches difficult for end-users to work across archives without having to learn multiple different interfaces recognised need for single search interface to all archives Universal Pre-print Service (UPS)

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Searching vs. harvesting two possible approaches to building the UPS… cross-searching multiple archives based on protocol like Z39.50 harvesting metadata into one or more central services – bulk move data to the user-interface US digital library experience in this area (e.g. NCSTRL) indicated that cross-searching not preferred approach - distributed searching of N nodes viable, but only for small values of N NCSTRL: N > 100; bad

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Problems of cross-searching collection description how do you know which targets to search? query-language problem syntax varies and drifts over time between the various nodes rank-merging problem how do you meaningfully merge multiple result sets? performance tends to be limited by slowest target difficult to build browse interface

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Universal Preprint Service a cross-archive DL that that provides services on a collection of metadata harvested from multiple archives based on NCSTRL+; a modified version of Dienst demonstrated at Santa Fe NM, October 21-22, D-Lib Magazine, 6(2) 2000 (2 articles) UPS was soon renamed the Open Archives Initiative (OAI)

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I RDN experience similar experience within the UK Resource Discovery Network (RDN) cross-searching of only 5 subject gateways problems with cross-searching approach performance central browse interface looking for metadata harvesting solution

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Data and service providers UPS identified two logical groups of services… data providers handle deposit/publishing of resources in archive expose metadata about resources in archive service providers harvest metadata from data providers use it to offer single user-interface across all harvested metadata note: data provider may also be responsible for human-oriented (I.e. Web) interface to archive both functions may be offered by same service

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Human vs. machine interfaces move away from only supporting human end-user interfaces for each archive… …to supporting both human end-user interface and machine interfaces for harvesting Provider Input interface Native end-user interface Provider Input interface Native end-user interface Native harvesting interface

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Service provider harvesting Data Provider Input interface Native harvesting interface Data Provider Input interface Native end-user interface Native harvesting interface Service Provider Native end-user interface Native end-user interface optional (e.g., RePEc)

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Metadata harvesting requirements in order that harvesting approach can work need agreements about… transport protocols – HTTP vs. FTP vs. … metadata formats – DC vs. MARC vs. … quality assurance – mandatory elements, mechanisms for naming of people, subjects, etc., handling duplicated records, best-practice intellectual property and usage rights – who can do what with the records work in this area resulted in the Santa Fe Convention

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Santa Fe Convention [02/2000] goal: optimize discovery of e-prints inputs… UPS prototype RePEc/SODA data provider / service provider model Dienst protocol deliberations at Santa Fe meeting [10/1999]

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I OAI-PMH v 1.0 [01/2001] goal: optimise discovery of document-like objects inputs… Santa Fe Convention various DLF meetings on metadata harvesting deliberations at Cornell alpha-testers of OAI-PMH v 1.0 recognition of DC as best core metadata format for interoperability across multiple archives

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I OAI-PMH v 1.0 [01/2001] low-barrier interoperability specification metadata harvesting model: data provider / service provider focus on document-like objects autonomous protocol HTTP based XML responses unqualified Dublin Core experimental: months

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Whats in a name? Open Archives Initiative the protocol is openly documented, and metadata is exposed to at least some peer group (note: rights management can still apply!) archive defined as a collection of stuff -- not the archivists definition of archive. Repository used in most OAI documents. OAI is happening at break-neck speed...

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I OAI timeline before v. 2.0 October 21-22, initial UPS meeting February 15, Santa Fe Convention published in D-Lib Magazine precursor to the OAI metadata harvesting protocol June 3, workshop at ACM DL 2000 (Texas) August 25, OAI steering committee formed, DLF/CNI support September 7-8, technical meeting at Cornell University defined the core of the current OAI metadata harvesting protocol September 21, workshop at ECDL 2000 (Portugal) November 1, Alpha test group announced (~15 organizations) January 23, OAI protocol 1.0 announced, OAI Open Day in the U.S. (Washington DC) purpose: freeze protocol for months, generate critical mass February 26, OAI Open Day in Europe (Berlin) July 3, OAI protocol 1.1 announced to reflect changes in the W3Cs XML latest schema recommendation September 8, workshop at ECDL 2001 (Darmstadt)

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I OAI-PMH v.2.0 [06/2002] goal: recurrent exchange of metadata about resources between systems inputs: OAI-PMH v.1.0 feedback on OAI-implementers deliberations by OAI-tech [09/ /02] alpha test group of OAI-PMH v.2.0 [03/ /02] officially released June 14, 2002

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I OAI-PMH v.2.0 [06/2002] low-barrier interoperability specification metadata harvesting model: data provider / service provider metadata about resources autonomous protocol HTTP based XML responses unqualified Dublin Core stable

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I abouteprints document like objects resourcesmetadata OAMS unqualified Dublin Core unqualified Dublin Core transport HTTP responsesXML requests HTTP GET/POST verbs Dienst OAI-PMH natureexperimental stable model metadata harvesting metadata harvesting metadata harvesting Santa Fe convention OAI-PMH v.1.0/1.1 OAI-PMH v.2.0

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Flexible deployment simple protocol based on HTTP and XML allows for rapid deployment a number of toolkits available – see part III systems can be deployed in variety of configurations multiple service providers can harvest from multiple data providers aggregators can sit between data and service providers harvesting approach can be complemented with searching based on Z39.50 or SRW

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Multiple data and service ps Data providers Service providers Harvesting based on OAI-PMH

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Aggregators Data providers Service providers Aggregator

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Can be mixed with x-searching Data providers Service providers Harvesting based on OAI-PMH Searching based on Z39.50 or SRW

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Summary OAI-PMH – OAI Protocol for Metadata Harvesting low-cost mechanism for harvesting metadata records from one system to another from data providers to service providers development over last 2-3 years has seen move from specific (discovery of e-prints) to generic (sharing descriptions of any resources) based on HTTP and XML – Web-friendly allows client to say give me some or all of your records where some is based on date-stamps, sets, metadata formats

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Summary (2) mandates simple DC as record format but extensible to any format encoded in XML OAI-PMH is not a search protocol but use can underpin search-based services based on Z39.50 or SRW or … metadata and full-text typically made freely available – but not a requirement OAI-PMH can be used between closed groups access-control and compression mechanisms based on underlying HTTP protocol simple protocol allows easy deployment systems can be combined in variety of ways

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part I Important resources OAI Web site: OAI-PMH specification: Implementation guidelines: Discussion lists: Repository explorer: Tools:

IST Tutorial OAI and OAI-PMH for Beginners An introduction to the Open Archives Initiative and the Protocol for Metadata Harvesting Part II: Technical Introduction Uwe Müller Humboldt University Berlin, Germany

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Agenda 1.Protocol Basics 2.Protocol Details 3.Request Types 4.Examples

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II The Open Archives Initiative (OAI) Main ideas world-wide consolidation of scholarly archives free access on the archives (at least: metadata) consistent interfaces for archives and service provider low barrier protocol / effortless implementation based on existing standards (e.g. HTTP, XML, DC) Basic functioning HarvesterRepository Requests (based on HTTP) Metadata (encoded in XML) Metadata (Documents) Metadata Service ProviderData Provider Service

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II OAI: General Assumptions two groups of participants Data Providers (Open Archives, Repositories) free access of metadata not necessarily: free access to full texts / resources easy to implement, low barriers Service Providers use OAI interfaces of the Data Providers harvest and store metadata (no live requests!) may select certain subsets from Data Providers (set hierarchy, date stamp) may enrich metadata offer (value-added) service on the basis of the metadata

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II OAI-PMH: Structure Model Service Provider e-print Data Provider e-prints e-print Data Provider Images e-print Data Provider OPAC e-print Data Provider Museum e-print Data Provider Archive Requests: Identify ListMetadataformats ListSets ListIdentifiers ListRecords GetRecord Responses: General information Metadata formats Set structure Record identifier Metadata Data Provider Harvester Repository

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II OAI-PMH: Protocol Overview protocol based on HTTP request arguments as GET or POST parameters six request types e.g. verb=ListRecords&from= responses are encoded in XML syntax supports any metadata format (at least: Dublin Core) logical set hierarchy (definition: data providers) date stamps (last change of metadata set) error messages flow control

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Agenda 1.Protocol Basics 2.Protocol Details 3.Request Types 4.Examples

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Protocol Details: Definitions Harvester client application issuing OAI-PMH requests Repository network accessible server, able to process OAI-PMH requests correctly Resource object the metadata is about, nature of resources is not defined in the OAI-PMH Item component of an repository from which metadata about a resource can be disseminated has an unique identifier Record metadata in a specific metadata format Identifier unique key for an item in a repository Set optional construct for grouping items in a repository

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Protocol Details: Definitions (2) resource all available metadata about David item Dublin Core metadata MARC metadata SPECTRUM metadata records item = identifier

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Protocol Details: Records metadata of a resource in a specific format three parts 1.header (mandatory) identifier (1) datestamp (1) setSpec elements (*) status attribute for deleted item (?) 2.metadata (mandatory) XML encoded metadata with root tag, namespace repositories must support Dublin Core 3.about (optional) rights statements provenance statements

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Protocol Details: Datestamps date of last modification of a metadata set mandatory characteristic of every item two possible granularities: YYYY-MM-DD, YYYY-MM-DDThh:mm:ssZ function: information on metadata, selective harvesting ( from and until arguments) applications: incremental update mechanisms modification, creating, deletion deletion: three support levels no, persistent, transient

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Protocol Details: Metadata Schema OAI-PMH supports dissemination of multiple metadata formats from a repository properties of metadata formats id string to specify the format ( metadataPrefix ) metadata schema URL (XML schema to test validity) XML namespace URI (global identifier for metadata format) repositories must be able to disseminate unqualified Dublin Core arbitrary metadata formats can be defined and transported via the OAI-PMH returned metadata must comply with XML namespace specification

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Protocol Details: Metadata Schema (2) minimum standard: unqualified Dublin Core Dublin Core Metadata Element Set contains 15 elements elements are optional elements may be repeated The Dublin Core Metadata Element Set: TitleContributorSource CreatorDateLanguage SubjectTypeRelation DescriptionFormatCoverage PublisherIdentifierRights

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Protocol Details: Sets logical partitioning of repositories optional – archives do not have to define sets no recommendations not necessarily exhaustive not necessarily strictly hierarchical function: selective harvesting ( set parameter) applications: subject gateways, dissertation search engine, … examples (Germany, see publication types (thesis, article, …) document types (text, audio, image, …) content sets, according to DNB (medicine, biology, …)

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Protocol Details: Request Format requests must be submitted using the GET or POST methods of HTTP repositories must support both methods at least one key=value pair: verb=[RequestType] additional key=value pairs depend on request type example for GET request: verb=ListRecords&metadataPrefix=oai_dc encoding of special characters e.g. : (host port separator) becomes %3A

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Protocol Details: Response formatted as HTTP responses content type must be text/xml status codes (distinguished from OAI-PMH errors) e.g. 302 (redirect), 503 (service not available) compression: optional in OAI-PMH, only identity encoding is mandatory response format: well formed XML with markup: 1.XML declaration ( ) 2.root element named OAI-PMH with three attributes ( xmlns, xmlns:xsi, xsi:schemaLocation ) 3.three child elements 1. responseDate (UTC datetime) 2. request (request that generated this response) 3. a) error (in case of an error or exception condition) b) element with the name of the OAI-PMH request

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Protocol Details: Flow Control four of the request types return a list of entries three of them may reply large lists OAI-PMH supports partitioning decision on partitioning: repository response to a request includes incomplete list resumption token + expiration date, size of complete list, cursor (optional) new request with same request type resumption token as parameter all other parameters omitted! response includes next (maybe last) section of the list resumption token (empty if last section of list enclosed)

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Protocol Details: Flow Control (2) Example Harvester Repository Service Provider Data Provider want to have all your records archive.org/oai?verb=ListRecords& metadataPrefix=oai_dc have 267, but give you only records + resumptionToken anyID1 want more of this archive.org/oai?resumptionToken=anyID1 have 267, give you another records + resumptionToken anyID2 want more of this archive.org/oai?resumptionToken=anyID2 have 267, give you my last records + resumptionToken

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Protocol Details: Errors and Exceptions repositories must indicate OAI-PMH errors inclusion of one or more error elements defined error identifiers badArgument badResumptionToken badVerb cannotDisseminateFormat idDoesNotExist noRecordsMatch noMetaDataFormats noSetHierarchy

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Agenda 1.Protocol Basics 2.Protocol Details 3.Request Types 4.Examples

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Request Types six different request types 1.Identify 2.ListMetadataFormats 3.ListSets 4.ListIdentifiers 5.ListRecords 6.GetRecord harvester has not to use all types repository must implement all types required and optional arguments depend on request types

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Request Type: Identify function description of an archive example archive.org/oai-script?verb=Identify parameters none errors / exceptions badArgument e.g. archive.org/oai-script?verb=Identify& set=biology

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Request Type: Identify (2) ElementExample# repositoryNameMy Archive1 baseURLhttp://archive.org/oai1 protocolVersion2.01 earliestDatestamp deleteRecordsno, transient, persistent1 granularity YYYY-MM-DD, YYYY-MM-DDThh:mm:ssZ 1 compressiondeflate, compress, …* descriptionoai-identifier, eprints, friends, …* response format

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Request Type: ListMetadataFormats function retrieve available metadata formats from archive example archive.org/oai-script?verb=ListMetadataFormats& identifier=oai:HUBerlin.de: parameters identifier (optional) errors / exceptions badArgument idDoesNotExist e.g. archive.org/oai-script?verb=ListMetadataFormats& identifier=really-wrong-identifier noMetadataFormats

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Request Type: ListSets function retrieve set structure of a repository example archive.org/oai-script?verb=ListSets parameters resumptionToken (exclusive) errors / exceptions badArgument badResumptionToken e.g. archive.org/oai-script?verb=ListSets& resumptionToken=any-wrong-token noSetHierarchy

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Request Type: ListIdentifiers function abbreviated form of ListRecords, retrieving only headers example archive.org/oai-script?verb=ListIdentifiers& metadataPrefix=oai_dc&from= parameters from (optional) until (optional) metadataPrefix (required) set (optional) resumptionToken (exclusive) errors / exceptions badArgument, e.g. …&from= :45:00 badResumptionToken cannotDisseminateFormat noRecordsMatch noSetHierarchy

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Request Type: ListRecords function harvest records from a repository example archive.org/oai-script?verb=ListRecords& metadataPrefix=oai_dc&set=biology parameters from (optional) until (optional) metadataPrefix (required) set (optional) resumptionToken (exclusive) errors / exceptions badArgument badResumptionToken cannotDisseminateFormat noRecordsMatch noSetHierarchy

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Request Type: GetRecord function retrieve individual metadata record from a repository example archive.org/oai-script?verb=GetRecord& identifier=oai:HUBerlin.de: & metadataPrefix=oai_dc parameters identifier (required) metadataPrefix (required) errors / exceptions badArgument cannotDisseminateFormat idDoesNotExist

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Agenda 1.Protocol Basics 2.Protocol Details 3.Request Types 4.Examples

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Example: verb=ListIdentifiers&from= &until= & metadataPrefix=oai_dc&set=doctypes:dissertations T17:49:49+01:00 oai:HUBerlin.de: doctypes doctypes:dissertations dnb dnb:dnb33 oai:HUBerlin.de: doctypes doctypes:dissertations dnb dnb:dnb27

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Example: verb=GetRecord&identifier=oai:HUBerlin: & metadataPrefix=oai_dc <OAI-PMH xmlns=" xmlns:xsi=" xsi:schemaLocation=" T14:57:01+01:00 <request verb="GetRecord" metadataPrefix="oai_dc" identifier="oai:HUBerlin.de: "> oai:HUBerlin.de: […] <oai_dc:dc xmlns:oai_dc=" xmlns:dc=" xmlns:xsi=" xsi:schemaLocation=" Einfluß genetischer Variationen im Tumor Nekrose […] Schüttlöffel, Antje […]

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part II Technical Introduction: Questions? OAI – official site protocol specification general mailing list implementers mailing list

IST Tutorial OAI and OAI-PMH for Beginners An introduction to the Open Archives Initiative and the Protocol for Metadata Harvesting Part III: Implementation Issues Data Provider and Service Provider Uwe Müller Humboldt University Berlin, Germany

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Agenda 1.General Considerations 2.Data Provider 3.Service Provider

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III General: First Questions Data Provider Which data do I want to deliver? Which service providers do I want to provide with data? Service Provider Which Service do I want to provide? From which data providers do I get the metadata? In which way the metadata have to be processed? Data Provider & Service Provider Which aspects do we have to agree upon?

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III General: Metadata Formats / Sets required: unqualified Dublin Core special subjects / communities: other metadata specifications may be required describe resources in a specialised way definition of an XML schema (publicly available for validation) define set hierarchy sensible partitioning for selective harvesting agreement between data providers and between data and service providers

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III General: Organisational Structure aggregated data providers if harvested by a service provider, sub data providers should not be harvested by same SP (duplication...) subject gateways selective harvesting if corresponding sets have been defined and implemented

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Agenda 1.General Considerations 2.Data Provider 3.Service Provider

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Data Provider: Prerequisites metadata on resources (items) should be stored in (SQL) database possible in case of need: file system … unique identifier for each item web server, accessible via the internet e.g. apache, IIS programming interface / API e.g. Perl, PHP, Java-Servlet web server extension access to database (or filesystem) not needed: session management

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Data Provider: Prerequisites (2) archive identifier / base URL unique identifier for items metadata format (at least: unqualified Dublin Core) datestamps for metadata (created / last modified) logical set hierarchy (may have) agreement within (subject) communities flow control / implementation of resumption token (optional, larger archives should have that)

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Data Provider: Architecture OAI Data Provider OAI request (HTTP request) Web server (e.g. Apache, IIS) Programming extension (e.g. PHP, Perl, JavaServlets) DB response OAI response (XML instance) Script / Programme - parsing arguments - creating error messages - creating SQL statements -creating XML output SQL- Database SQL request

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Data Provider: General Structure Argument Parser validates OAI requests Error Generator creates XML responses with encoded error messages Database Query / Local Metadata Extraction retrieves metadata from repository according to the required metadata format XML Generator / Response Creation creates XML responses with encoded metadata information Flow Control realises incomplete list sequences for larger repositories uses resumption token as mechanism

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Data Provider: Flow Chart HTTP request verb metadata Prefix ListIdentifiers GetRecordIdentifyListMetadata-FormatsListSetsListRecordselse error: badVerb error: cannotDiss- eminateFormat else empty re sumption Token error: badResumptionToken unknown read parameters from local system valid parse the other parameters oai_dc send SQL request to database rows> 100 store parameters, store and deliver resumptionToken yes XML response error: badArgument empty verb, metadataPrefix, resump- tionToken … OAI arguments rows … size of the result list 100 … here: maximal list size for responses deliver min (rows, 100) record headers no

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Data Provider: Resumption Token should be implemented for large lists initiated by data provider store parameters ( set, from, …) and number of already delivered records properties expiration: expirationDate (optional) completeListSize (optional) already delivered records: cursor (optional) recovery from network errors (possibility to re-issue most recent resumption token) problem database changes two possible solutions duplicate data in a request table store date of first request with the other parameters use like additional until argument

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Data Provider: Resumption Token (2) Example Harvester Repository Service Provider Data Provider want to have all your records archive.org/oai?verb=ListRecords& metadataPrefix=oai_dc have 267, but give you only records + resumptionToken anyID1 want more of this archive.org/oai?resumptionToken=anyID1 have 267, give you another records + resumptionToken anyID2 want more of this archive.org/oai?resumptionToken=anyID2 have 267, give you my last records + resumptionToken

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Data Provider: Resumption Token (3) Repository Data Provider Example (2) want to have all your records archive.org/oai?verb=ListRecords& metadataPrefix=oai_dc Database have 267, but give you only records + resumptionToken anyID1 want more of this archive.org/oai?resumptionToken=anyID1 have 268, give you another records + resumptionToken anyID2 select dc-data from metadata-table records 2 insert, update, delete 3 select dc-data from metadata-table records anyID1 = { from=empty, until=empty, set=empty, mdP=oai_dc, date= T15:00:00Z, delivered=100 }

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Data Provider: Data Representation use recommended data representation dates xx-xx, 2002, language code eng, ger,... en, de, english, german multi values: use own XML element for each entity author Smith, Adam Nash, John Smith, Adam; Nash, John

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Data Provider: Compression method to reduce traffic and enhance performance optional for both sides: data and service providers handled on HTTP level harvesters may include an Accept-Encoding header in their requests –specifying preferences harvesters without Accept-Encoding header always receive uncompressed data repositories must support HTTP identity encoding repositories should specify supported encodings by including compression elements in the identify response

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Data Provider: Test and Registration create own OAI-PMH requests and send to OAI interface – check results use the Repository Explorer (VT University) provide arguments via HTML forms responses are validated browsing to other requests automatic conformance tester official registration site provide base URL extensive conformance test (incl. error conditions …) information on incorrect behaviour in case of conformance – added to the official list regular checks

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Agenda 1.General Considerations 2.Data Provider 3.Service Provider

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Service Provider: Examples Repository Explorer: search engines / subject gateways Cross Archive Searching Service: MyOAI: DINI: Physnet: internal communication ProPrint: library compounds

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Service Provider: Prerequisites internet connected server database system (relational or XML) programming environment can issue HTTP requests to web servers can issue database requests XML parser

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Service Provider: Structure (1) Archive Management selection of archives to be harvested enter entries manually or automatically add / remove archives using the official registry Request Component creates HTTP requests and sends them to OAI archives (data provider) demands metadata using the allowed verbs of the OAI-PMH possibly selective harvesting ( set parameter)

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Service Provider: Structure (2) Scheduler realises timed and regular retrieval of the associated archives simplest case: manual initiation of the jobs else: e.g. cron job … Flow Control resumption token: partitioning of the result list into incomplete sections – anew request to retrieve more results HTTP error 503 (service not available) – analysis of response to extract retry-after period

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Service Provider: Structure (3) Update Mechanism realises consolidation of metadata which have been harvested earlier (merge old and new data) easiest case: always delete all old metadata of an archive before harvesting it reasonable: incremental update ( from parameter) – insert new metadata and overwrite changed / deleted metadata (assignment using the unique identifiers) XML Parser analyses the responses received from the archives validation: using the XML schema transforms the metadata encoded in XML into the internal data structure

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Service Provider: Structure (4) Normaliser transforms data into a homogenous structure (different metadata formats) harmonises representation (e.g. date, author, language code) maps / translates different languages Database mapping the XML structure of the metadata into a relational database (multi values …) or: use an XML database

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Service Provider: Structure (5) Duplication Checker merges identical records from different data providers possibility: unique identifier for the item (e.g. URN, …) but: often not easily practicable and not risk / error free Service Module provides the actual service to the public basis: harvested and stored records of the associated archives uses only local database for requests etc.

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Service Provider: Architecture Data Provider Scheduler Flow control XML Parser Normaliser Database Service module UserHarvesterUser OAI Service Provider Dublication checker Update mechanism Administrator

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Service Provider: Resumption Token optional from the data providers point of view but: mandatory for service providers for complete lists: resume sequences of incomplete lists 1.recognise that response contains incomplete list 2.re-issue OAI request to data provider in order to get next part of the list

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Service Provider: Test and Registration harvest registered ( OAI complient!) data providers test behaviour of service provider official registration site registerasprovider.html provide institutional information web site, address,...

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part III Data & Service Provider: Questions?

IST Tutorial OAI and OAI-PMH for Beginners An introduction to the Open Archives Initiative and the Protocol for Metadata Harvesting Part IV: Implementation issues - XML schemas and support for multiple record formats Andy Powell UKOLN, University of Bath

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Agenda 1.basics 2.XML schema details 3.extending oai_dc for your application 4.using IMS metadata as new record format

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Basics OAI-PMH uses XML Schemas to define record formats you can exchange any data you like using OAI- PMH as long as you can encode it as XML and define an XML-Schema for it! OAI-PMH mandates the oai_dc XML schema OAI-PMH documentation also describes use of XML schema to exchange rfc1807: a schema for rfc1807 format metadata; marc21: a recommended schema for MARC21 metadata, provided by the Library of Congress; oai_marc: a schema for MARC format metadata

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV A closer look at oai_dc the simple DC schema used as mandatory record format in OAI-PMH defines a container schema container schema is OAI-specific container schema is hosted on the OAI Web site imports a generic DCMES schema generic DCMES schema is hosted on the DCMI Web site same model likely to be used for qualified DC schema – container schema hosted by OAI, generic schema hosted by DCMI

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV An oai_dc record… an example oai_dc record (viewed via the repository explorer) repository explorer heres the full GetRecord responseGetRecord response three important things to notice… namespace for the oia_dc format xmlns:oai_dc= namespace for DCMES elements xmlns:dc= container schema associated with the oai_dc namespace xsi:schemaLocation="

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV The XML schemas The oai_dc container schema imports DCMES schema from defines a container element called dc lists the allowed elements within the dc container (from the DCMES namespace/schema above)

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV When oai_dc isnt enough when the 15 DCMES elements are too limited – e.g. adding extra metadata elements when you need greater precision in your metadata records – e.g. adding encoding schemes to existing elements when you want to exchange other metadata formats IMS/IEEE LOM – eLearning metadata ODRL – Open Digital Rights Language

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Extending the oai_dc schema simple scenario… RDN currently uses oai_dc schema to exchange records but wants to add one additional element called accessControl note: this is not a real scenario… RDN really wants to use qualified DC records – but doing qualified DC too complicated for this tutorial! hope to write-up RDN work on exchanging qualified DC in future issue of Ariadne

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Step 1 – metadata format name the new metadata format needs a name in this case, weve chosen rdn_dc following OAIs naming of oai_dc alternative possibilities rdndc rdn etc.

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Step 2 – create namespaces two namespaces are required… namespace for the rdn_dc format namespace for the new metadata elements (properties) that we are going to use in this format note: use of Purl for the elements namespace follows DCMI usage but is not mandatory however, both these namespace URIs should be under your control to ensure uniqueness and prevent re-use in the future URIs do not need to resolve to anything

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Step 3 – local copy of DC schema make local copy of the DCMES schema in this case the copy is at this step isnt strictly necessary in fact – it is probably bad practice to do this but, currently some minor problems with the DCMI-hosted copy of the schema …working with local copy is easier

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Step 4 – schema for new terms create an XML schema for the new rdnterms in this case the schema is available at the schema defines the new element/property accessControl and adds it to the dc:any group also creates a new container type rdnterms:elementContainer note: schema URI contains a date-stamp this should make future enhancements to the schema easier to implement

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Step 5 – container schema create a container schema for the new record format in this case the schema is available at this simply imports the rdnterms schema then defines a container element called rdndc of type rdnterms:elementContainer again, the schema URI contains a date-stamp

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Step 6 – validate, validate, val… create some test records using your new schemas use the XML schema validator at

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Step 7 – ListMetadataFormats add information about the new format to your repositorys response to the ListMetadataFormats request… … rdn_dc …

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Step 8 – other verbs modify your repositorys response to the ListSets, ListIdentifiers, ListRecords and GetRecord requests accept metadataPrefix set to new format name rdn_dc return records formatted according to the new schema(s)

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Step 9 – validate again use the Repository Explorer to check that:Repository Explorer all requests work with new metadataPrefix oai_dc format still works! appropriate records are returned for each format responses validate correctly

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Summary decide on name for your new metadata format and appropriate namespaces develop XML schemas for container and new elements if appropriate create test records and validate modify your repository (source code and/or configuration files) to support the new format validate and test repository

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Other record formats can take similar approach with other metadata record formats IMS/IEEE LOM ODRL in these cases, XML schemas and namespaces have already been agreed deployment of these formats should be easier because you dont need to define your own schemas… BUT… XML schema specs continually undergoing revisions currently so sometimes hard for applications like IMS to keep up!

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners - Part IV Adding support for IMS modify ListMetadataFormats response to include extend ListSets, ListIdentifiers, ListRecords and GetRecord requests accept metadataPrefix set to ims and return records formatted appropriately … ims …

IST Tutorial OAI and OAI-PMH for Beginners An introduction to the Open Archives Initiative and the Protocol for Metadata Harvesting

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners Summary during todays tutorial we hope that you have gained an overview of the history behind the OAI- PMH and an overview of its key features been given a deeper technical insight into how the protocol works learned something about some of the main implementation issues found some useful starting points and hints that will help you as implementors

IST nd OAForum workshop - Lisbon - 5th-7th December Tutorial: OAI and OAI-PMH for Beginners Questions now… feel free to tell us what you didnt understand and ask general questions (of course!) Uwe Müller Humboldt University Berlin, Germany Andy Powell UKOLN, University of Bath