Open Archives Iniative – Protocol for Metadata Harvesting Iztok Kavkler, University of Ljubljana Some slides by Stefaan Ternier, KUL Bram Vandenputte,

Slides:



Advertisements
Similar presentations
A centre of expertise in digital information management The OAI Protocol for Metadata Harvesting Andy Powell UKOLN,
Advertisements

Gary Holton ANLC LSA Symposium: The Open Language Archives Community 4 January 2002 Creating an OLAC data provider at the Alaska Native Language Center.
Rapid Visual OAI Tool S. Kothamasa, K. Maly, M. Zubair (Old Dominion University) X. Liu (Los Alamos National Laboratory) RCDL 2003, St. Petersburg.
Y.T. a brief history of the OAI 0 Kaynak: Herbert van de Sompel.
OAI in DigiTool DigiTool Version 3.0.
Harvesting Metadata Using OAI-PMH Roy Tennant California Digital Library.
Using Multiple Metadata Formats in DSpace ARD Prasad Indian Statistical Institute Bangalore, India.
OAI-PMH Dawn Petherick, University Web Services Team Manager, Information Services, University of Birmingham MIDESS Dissemination.
Infrastructures for Using Metadata RSS and OAI-PMH CS 431 – March 14, 2005 Carl Lagoze – Cornell University.
National Science Digital Library (NSDL) Core Infrastructure Metadata Repository (“union catalog”) Naomi Dushay Cornell University.
UCLA Digital Library UC Digital Library Forum August 5, 2002 UCLA Digital Library Presenter: Curtis Fornadley Senior Programmer/Analyst.
OAI Standards for Sheet Music Meeting March 28-29, 2002 Basic OAI Principals How They Apply to Sheet Music Presenter: Curtis Fornadley, Senior Programmer/Analyst.
OAI-PMH at Yale Report on the DLF OAI Training Session November 10, 2005 Charlottesville, VA.
Basic Concepts Architecture Topology Protocols Basic Concepts Open e-Print Archive Open Archive -- generalization of e-print Data Provider and Service.
NAL-Institutional Repository: A Case Study CSIR Metadata Harvester I.R.N. Goudar Head, ICAST, NAL National Symposium on Open Access and.
A Digital Library Repository Utilizing the Open Archives Initiative Developed to meet the needs of UTK Library Special Collections.
Introduction to the OAI Metadata Harvesting Protocol Hussein Suleman, Digital Library Research Laboratory Virginia Tech.
Metadata Harvesting The Hague, 13 & 14 January 2009 Julie Verleyen Scientific Coordinator, Europeana Office EuropeanaLocal Knowledge Sharing Workshop.
Rapid Visual OAI Tool S. Kothamasa, K. Maly, M. Zubair (Old Dominion University) X. Liu (Los Alamos National Laboratory) RCDL 2003, St. Petersburg.
Metadata Harvesting Interoperable digital collections.
Lucas Mak and Dao Rong Gong Michigan State University Millennium and XML: Repurposing and Customizing Metadata May , 2009.
ALA 2002 LITA Open Source Software Open Archives Initiative Hussein Suleman AmericanSouth.org 14 June 2002.
Indo-US Workshop, June23-25, 2003 Building Digital Libraries for Communities using Kepler Framework M. Zubair Old Dominion University.
OAI-PMH The Open Archives Initiative Protocol for Metadata Harvesting Presenter: Knud Möller Friday,
1 OAI-PMH harvester for agricultural knowledge gathering (Development, testing and implementation) Francesco Castellani and Stefka Kaloyanova 4 February.
07/11/2002Thomas Baron - JACoW Workshop1 CERN Library Requirements T. Baron CERN ETT-DH-CDS.
OAI-PMH: Open Archives Initiative Protocol for Metadata Harvesting T.B. Rajashekar National Centre for Science Information (NCSI) Indian Institute of Science,
Introduction to Digital Libraries hussein suleman uct cs honours 2004.
The OAI Protocol for Metadata Harvesting Van de Sompel, Herbert Los Alamos National Laboratory – Research Library.
Metadata harvesting in regional digital libraries in PIONIER Network Cezary Mazurek, Maciej Stroiński, Marcin Werla, Jan Węglarz.
Digital Library Interoperability Architecture CS 502 – Carl Lagoze – Cornell University.
Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) Phil Barker, March © Heriot-Watt University. You may reproduce all or any part.
Open Archive Initiative – Protocol for metadata Harvesting (OAI-PMH) Surinder Kumar Technical Director NIC, New Delhi
Caltech CODA CODA: Collection of Digital Archives Caltech Scholarly Communication.
Slavic Digital Text Workshop 2006 The Open Archives Initiative Protocol for Metadata Harvesting: an Opportunity for Sharing Content in a Distributed Environment.
OAI Protocol for Metadata Harvesting hussein suleman uct cs honours 2006.
OAI Overview DLESE OAI Workshop April 29-30, 2002 John Weatherley
Bitter Harvest Metadata Harvesting Issues, Problems, and Possible Solutions Roy Tennant California Digital Library.
Search Interoperability, OAI, and Metadata Sarah Shreeves University of Illinois at Urbana-Champaign Basics and Beyond Grainger Engineering Library April.
SPASE and the VxOs Jim Thieman Todd King Aaron Roberts.
Building Interoperable Digital Libraries: A Practical Guide to creating Open Archives Hussein Suleman, Digital Library Research.
Building Interoperable and Accessible ETD Collections: A Practical Guide to Creating Open Archives Hussein Suleman, Digital.
The OAI: technical overview OAI Open Meeting – Washington DC – January 23 rd 2001 Herbert Van de Sompel & Carl Lagoze Cornell University -- Computer Science.
The Open Archives Initiative Marshall Breeding Director for Innovative Technologies and Research Vanderbilt University
Open Archives Initiative Protocol for Metadata Harvesting.
Standards OAI-Protocol Metadata: DC - Agris - MODS Marc Goovaerts Hasselt University Library ODIN-PI TRAINING OSTENDE, May 2008.
2/22/2016J Ammerman1 Open Archives Initiative What is it? What’s it good for?
NSDL & the Open Archives Initiative A Brief Introduction to OAI Timothy W. Cole Mathematics Librarian & Professor of Library Administration.
Introduction to the OAI Protocol for Metadata Harvesting Version 2.0 Hussein Suleman Virginia Tech DLRL 25 March 2002.
1 CS 430: Information Discovery Lecture 26 Architecture of Information Retrieval Systems 1.
The NSDL, OAI and Your Metadata Core Infrastructure Metadata Repository (“union catalog”) Naomi Dushay Cornell University.
Mod_oai: Metadata Harvesting for Everyone Michael L. Nelson, Herbert Van de Sompel, Xiaoming Liu, Aravind Elango
OAI and ODL Building Digital Libraries from Components Ryan Richardson Virginia Tech DLRL 18 September 2003.
OAI and ODL Building Digital Libraries from Components Hussein Suleman Virginia Tech DLRL 12 September 2002.
NDLTD Standards, Metadata and the OAI-PMH Hussein Suleman University of Cape Town October 2003.
Harvesting and Exporting Metadata 714: Metadata Margaret E.I. Kipp -
1 XML and XML in DLESE Katy Ginger November 2003.
Facet Forward: Faceted Navigation of Federated Search Results for Cultural Heritage Materials Mark E. Phillips LITA National Forum 2007 Denver, Co.
Open your Alfresco Data
Metadata Harvesting - OAI-PMH
Getting a Leg Up on OAI for the NSDL
University of Illinois at Urbana-Champaign OAI Alpha Experiences
Repository Software Marc Goovaerts, Hasselt University Library
Georges Arnaout Chaitanya Krishna
Making the most of research outputs
CS431 guest lecture Simeon Warner
OAI and Metadata Harvesting
Digitometric Services for Open Archives Environments
Open Archive Initiative
IVOA Interoperability Meeting - Boston
Presentation transcript:

Open Archives Iniative – Protocol for Metadata Harvesting Iztok Kavkler, University of Ljubljana Some slides by Stefaan Ternier, KUL Bram Vandenputte, KUL Joris Klerkx, KUL

2 What is OAI? Harvesting standard, documented at Seven service verbs – Identify – ListMetadataFormats – GetRecord – ListRecords – ListIdentifiers – ListSets Allows multiple metadata formats – DC (Dublin core) format mandatory

3 How OAI works OAI “VERBS” – Identify – ListMetadataFormats – GetRecord – ListIdentifiers – ListRecords – ListSets HARVESTERHARVESTER REPOSITORYREPOSITORY OAI Service Provider Metadata Provider HTTP Request HTTP Response (OAI Verb) (Valid XML)

4 Try it Install Apache-Tomcat or any other Java servlet container Download WAR file from Deploy WAR Demo html Or type a service verb, e.g.

5 The raw XML By default, the resulting XML has stylesheet attached for pretty rendering To remove the stylesheet comment the line OAIHandler.styleSheet=testoai/oaicat.xsl in file oaicat.properties (in WAR file or the web-app dir)

6 OAI XML example T06:48:58Z oai:oai.xyz-repository.com:exercises/ T22:38:28Z exercises <resumptionToken expirationDate=" T07:48:58Z" completeListSize="42" cursor="10">

7 OAICat - a Java implementation OAICat home at Takes care of – web service details – OAI XML specification The implementer has to provide three classes – RepositoryOAICatalog – RepositoryRecordFactory – Repository2oai_dc (lom,...) - usually more than one

8 A sample implementation (Source code and libs in Create a new web module Add servlet oaiHandler to web.xml LreOAIHandler ORG.oclc.oai.server.OAIHandler 5 LreOAIHandler /oaiHandler

9 (cont) Define properties file location properties oaicat.properties Welcome file for testing testoai/index.html

10 Sample record A record with basic fields id, url, title, descr and date SampleOAICatalog contains an array with 3 sample records

11 SampleOAICatalog.listIdentifiers Parameters – from – date to harvest from (String in iso8601 format) date or datetime - depends on granularity – to – date to harvest to – set – a set name, list only records from this set (if null, list all records) set names classify objects in natural groups every record may belong to multiple sets (or none) – metadaPrefix – list only records that support this format (sample formats: oai_dc, oai_lom,...)

12 SampleOAICatalog.listIdentifiers Must return a map with to fields – headers – a String iterator of OAI headers – identifiers – a String iterator of OAI identifiers Both created by the call (rec is a SampleRecord) String[] header = getRecordFactory().createHeader(rec); headers.add(header[0]); identifiers.add(header[1]); Create result Map listIdMap = new HashMap (); listIdMap.put("headers", headers.iterator()); listIdMap.put("identifiers", identifiers.iterator()); return listIdMap ;

13 getRecordFactory().createHeader(rec) Creates header by calling the methods in SampleRecordFactory String getOAIIdentifier(Object rec) – return full oai identifier “oai:oay.rep.com:id001” String getDatestamp(Object rec) – returns date in iso8601 format Iterator getSetSpecs (Object rec) ArrayList list = new ArrayList (); list.add(...); return list.iterator(); Iterator getAbouts (Object rec) String fromOAIIdentifier(String id) – helper method – convert id to a local id

14 SampleOAICatalog.listSets takes no parameters, returns the list of all sets in this repository – each ListIdentifiers or ListRecords query may contain a set name, limiting the results to just one set

15 SampleOAICatalog.getSchemaLocations like GetRecord, but returns the Vector of all metadata schema locations the record supports – to obtain them, just call getRecordFactory().getSchemaLocations(rec);

16 SampleOAICatalog.getRecord String getRecord(String id, String metadataPrefix) – find record and convert it to xml string ( element) – id is in global format – to get local value call getRecordFactory().fromOAIIdentifier(id) – throw IdDoesNotExistException if record not found – to generate XML use constructRecord constructRecord(rec, metadataPrefix)

17 SampleOAICatalog.listRecords just like ListIdentifiers, only generates a list of XML elements return a map with one element Map listRecMap = new HashMap (); listRecMap.put(“records", records.iterator()); return listRecMap;

18 Crosswalks Conversions of native record type to XML like Sample2oai_lom or Sample2oai_dc Only two methods per implementation – boolean isAvailableFor(Object rec) – String createMetadata(Object rec) SampleRecord record = (SampleRecord) rec; return LOMFormat.writeStringWithSchema(record.toLOM()); throw CannotDisseminateFormatException if the metadata not available in this format

19 SampleRecord.toLOM uses LOM-j lib to quickly hack together LOM – automatic serialization/deserialization of LOM and DC XML formats Example lom.newGeneral().newIdentifier(0).newCatalog().setString("lre"); lom.newGeneral().newIdentifier(0).newEntry().setString("sample:" + id); lom.newTechnical().newLocation(-1).setString(url); lom.newGeneral().newTitle().newString(0).newLanguage().setValue("en"); lom.newGeneral().newTitle().newString(0).setString(title);

20 Resumption A repository usually has fixed limit on the numer of records to return in one call – if there are more available, it returns a resumption token, allowing to receive next packet – Implemented by functions listIdentifiers(String resumptionToken), listRecords(String resumptionToken) – see XYZOAICatalog for details

21 References SIO/Trubar OAI url