OCLC Online Computer Library Center Interoperability Standards & Searching Multiple Repositories Ralph LeVan/OCLC Ray Denenberg/Library of Congress

The Problem How do I provide a common interface for my users? How do I combine results from multiple sources?

How do I provide a common interface for my users? How do I convert my queries into the Content Provider’s (CP’s) queries? How do I ask for 10 records? How do I ask for more records? How do I interpret their response?

How do I convert my queries into the CP’s queries? My user said “author=twain and title=huck finn” Google expects: +twain +”huck finn” Z39.50: twain/1=1003;4=2 “huck finn”/1=4;4=1 and Lucene: creator:twain and titlePhrase:”huck finn”

How do I ask for 10 records? Amazon won’t let you RedLightGreen: MAXRECORDS=n British Library: records=n

How do I ask for more records? Amazon: page=n RedLightGreen: STARTINDEX=n British Library: start=n

How do I interpret their response? How many records did I retrieve? Did something go wrong? How do I convert the CP’s records into something my users will recognize?

How many records did I retrieve? Amazon: Books (334) RedLightGreen: Viewing: 1-10 of 239 results British Library 190

Did Something Go Wrong? RedLightGreen: We didn't find any matches for dog and. British Library: Nothing found due to an error Too many hits. Refine your request.

How do I convert the records? Amazon: Thud! (Discworld, Book 32) by Terry Pratchett ( Hardcover - Sep 13, 2005) Books: See all 334 items Buy new : $24.95 $15.72 Used & new from $3.76 Usually ships in 24 hours Excerpt from page 2 : "... Terry Pratchett "Most of the news is... " See more references to pratchett in this book. Surprise me! See a random page in this book.

Converting Records Cont. RedLightGreen: Hogfather, by Terry Pratchett 3 editions published between 1996 and 1998 in English. Primary Subject: Discworld Imaginary Place - Fiction 2.

Converting Records Cont. British Library: Thud! / Terry Pratchett. doc- set&doc_number= &l_base=BLL01& from=A9OpenSearch Pratchett, Terry. ; London : Doubleday, ISBN (hbk.) : £ (Added : )

How do I combine results from multiple sources? Things you might want the server to do for you: –Common Record Format –Common Sort Order –Common Rank Order

Functional Matrix Request Record Starting Point Request Number of Records Request Record Schema Defined Query Grammar Specify Sort Order Specify Ranking Order Diagnostic Messages XML Response Record Count In Response Records In Known Schema

The Old Solutions Screen Scraping Private API’s Z39.50

Screen Scraping A query has to be generated and embedded in a CP specific URL Code has to be written to examine the HTML returned by a CP Prone to breakage –Web sites change formatting frequently Every site is unique –Separate code to be maintained for every site

Private API’s Often only a slight improvement over screen scraping Provides documentation on how to construct the URL Might provide documentation on how to construct the query Might guarantee a stable response format Still requires unique code for each site

Z39.50 Guarantees a standard request and response But… –Not HTTP or HTML Binary encoding over raw TCP/IP –Complicated 11 services 7 extended services –Easy to be compliant and not interoperable –Unfriendly The response to a protocol error was to drop the connection

Why Use A Standard API? Defined requests and responses Reusable code across sites Open Source code

The New Solutions OpenSearch 1.1 MXG –Levels 0-2 SRU

OpenSearch 1.1 From Wikipedia –OpenSearch is a collection of technologies that allow publishing of search results in a format suitable for syndication. It is a way for search engines to publish their search results in a standard and accessible format

OpenSearch 1.1 (cont.) Defines a Description Record with information about the CP –ShortName and LongName –Description –Tags –URL template Example:

OpenSearch 1.1 (cont.) URL Template –Server Indicates how to specify OpenSearch request parameters –Parameters not specified in the template are unavailable –The only mandatory parameter is {searchTerms}

OpenSearch 1.1 (cont.) Request Parameters –{searchTerms} –{count} –{startIndex} –{startPage} –{language} –{outputEncoding} –{inputEncoding}

OpenSearch 1.1 (cont.) Uses RSS 2.0 with a few extra elements for the response –RSS define title, description and link elements –OpenSearch adds the totalResults, startIndex, itemsPerPage, link and Query elements bin/OSxml1.cgi/?q=levan&format=rss

Functional Matrix OS 1.1 Request Record Starting Point● Request Number of Records ○ Request Record Schema Defined Query Grammar Specify Sort Order Specify Ranking Order Diagnostic Messages XML Response ○ Record Count In Response ○ Records In Known Schema ○ Key: ●==Full Support ○==Limited Support

Cool Feature The RSS mechanism in OpenSearch provides the ability to have persistent and periodic queries!

NISO MetaSearch XML Gateway MXG MXG has been designed to provide a low implementation barrier to content providers that want to make their databases available to metasearch engines. Interoperability across content providers was explicitly not a goal of MXG

MXG Levels of Support Level 0: Requests are simple URL’s using any query grammar and responses are XML records Level 1: Adds a description record for the database Level 2: Support a limited subset of a standard query grammar: CQL

MXG Request Version (mandatory) Query (mandatory) StartRecord MaximumRecords s?version=1.1&query="levan"&startRec ord=1&maximumRecords=10

MXG Response <searchRetrieveResponse xmlns=" … "stuff"

MXG Response Records info:srw/schema/1/dc-v1.1 xml … 1

MXG Response recordData rrl1234 Dog and Cat

MXG Error Messages info:srw/diagnostic/1/51 66ntqk list.html

Functional Matrix MXG Level 0 Request Record Starting Point● Request Number of Records● Request Record Schema ○ Defined Query Grammar Specify Sort Order Specify Ranking Order Diagnostic Messages● XML Response● Record Count In Response● Records In Known Schema● Key: ●==Full Support ○==Limited Support

MXG Level 1 Add a description record for the database

Functional Matrix MXG Level 1 Request Record Starting Point● Request Number of Records● Request Record Schema● Defined Query Grammar Specify Sort Order Specify Ranking Order Diagnostic Messages● XML Response● Record Count In Response● Records In Known Schema● Key: ●==Full Support ○==Limited Support

MXG Level 2 Support a limited subset of a standard query grammar: CQL Supports indexes and Booleans ery=dc.author=levan&maximumRecords=1

Functional Matrix MXG Level 2 Request Record Starting Point● Request Number of Records● Request Record Schema● Defined Query Grammar ○ Specify Sort Order Specify Ranking Order Diagnostic Messages● XML Response● Record Count In Response● Records In Known Schema● Key: ●==Full Support ○==Limited Support

SRU MXG Level 2 Plus: –Full Query Grammar (CQL) –Full Sort Specification

CQL: Common Query Language Loosely based on CCL Search Boolean & Proximity Operators Index Sets & Indexes String Indexes vs. Keyword Indexes Truncation Characters ‘*’, ‘#’ & ‘?’ Relations: ‘=‘, all, any, exact, within Example: dc.title=“harry potter” or bib1.isbn= x

Sort sortKeys parameter with the following comma separated values specified: –Xpath (path to the element to be sorted on) –Schema (that the xpath comes from) –Ascending (value is 1==true or 0==false, default==true) –CaseSensitive (value is 1==true or 0==false, default==false) –missingValue (values are omit, abort, highValue or lowValue, default==highValue) e.g. &sortKeys=title,onix,0

Functional Matrix SRU Request Record Starting Point● Request Number of Records● Request Record Schema● Defined Query Grammar● Specify Sort Order● Specify Ranking Order ○ Diagnostic Messages● XML Response● Record Count In Response● Records In Known Schema● Key: ●==Full Support ○==Limited Support

Cool Feature Combining SRU response data and echoed data with javascript and stylesheets allows for thin, browser based, clients s?version=1.1&query="levan"&startRec ord=1&maximumRecords=10

Functional Matrix OS 1.1 MXG L0 MXG L1 MXG L2 SRU Request Record Starting Point●●●●● Request Number of Records ○ ●●●● Request Record Schema ○ ●●● Defined Query Grammar ○ ● Specify Sort Order● Specify Ranking Order ○ Diagnostic Messages●●●● XML Response ○ ●●●● Record Count In Response ○ ●●●● Records In Known Schema ○ ●●●● Key: ●==Full Support ○==Limited Support