Breaking Out of the Box: Creating Customized Metasearch Services Using an XML API Roy Tennant, California Digital Library

A Cautionary Tale Wherein a NitWit finds the courage to face his Disastrous and Idiotic Mistakes while his respected colleagues Justifiably Point and Laugh

What is Breaking Out of the Box? Using an XML-based Application Program Interface (API) to an application instead of the native interface Requires building an interface layer to: –Accept user input, package it up, and send it to the API –Receive the XML response from the API and process it, perhaps spawning additional requests to the API –Perform (optionally) other tasks not performed by the application; e.g., querying another application and merging the response into the user interface Or: You do the work, the villain does not

What is Breaking Out of the Box? Using an XML-based Application Program Interface (API) to an application instead of the native interface Requires building an interface layer to: –Accept user input, package it up, and send it to the API –Receive the XML response from the API and process it, perhaps spawning additional requests to the API –Perform (optionally) other tasks not performed by the application; e.g., querying another application and merging the response into the user interface Or: You do the work, the vendor does not

Why Break Out of the Box? Much greater interface flexibility Interface customizations remain despite system upgrades Increased ability to integrate with other systems Ability to add new functions and services not supported by the vendor: –Spell checking –Recommendation services Upon which it finally occurs to the NitWit that Arrogance has its Price

A Visit to the Scene of the Crime: Part the 1 st Started shopping for metasearch software in late 2002 Purchased MetaLib from ExLibris in Spring 2004 Trained in Fall 2004, then began studying how to make it easy for campuses to customize the interface Interlude: A Discovery is Made

Wherein the NitWit discovers the full magnitude of his Error To customize one screen, change all of the following file fragments: quick-1, quick-2-end, quick-3-body, quick-details, quick-1- body, quick-2-end.orig, quick-3-head, quick-details-info, quick- 1-head, quick-2-head, quick-3-js, quick-full, quick-1-tail, quick- 2-tail, quick-3-no-results, quick-help, quick-1-toolbar, quick-2- toolbar, quick-3-tail, quick-help-main, quick-1-toolbar-pd, quick-2-wait, quick-3-toolbar, quick-js, quick-2, quick-3, quick- 8-body.spa, quick-main-help Then, watch other screens break later in the flow Rinse and Repeat

Why this was truly Tragic Our vision is of many search portals: –Tailored to specific audiences (earth science faculty and grad students) and/or purposes (a few good things) –Branded locally (we serve ten campuses) Integrating many types of content or services not typical of library metasearch applications The native interface presented difficult, systemic barriers to customization The native interface was rife with browser-specific Javascript and substandard HTML

Undergrad portal with suggestions of subject databases

A Visit to the Scene of the Crime : Part the 2 nd After struggling with interface changes, decided to go with API access in late 2004 Interlude: the NitWit discovers there is no API worthy of the name

In which the NitWit overcomes despair by Blind, Ill-Considered action The MetaLib API was primitive, poorly documented, buggy, and lacked most of the functionality of the native interface We marshalled the community: –Established an X-Server listserv –Gathered input on upgrades, wrote document summarizing our priorities Worked with ExLibris to improve the API Began integrating MetaLib with the Common Framework

The Attentive Audience Asks: The What?

Common Framework: Principles Clear separation of applications from underlying services Consistent exposure of services thru SOAP and Java client API Separation of services from data storage and other resources Easy integration of 3rd-party solutions Platform independence

The CDL Common Framework Programmed in Java Based on Web Services (e.g., SOAP/REST interfaces) Services often also have Java and/or Web-based client applications Uses METS & PREMIS (for metadata) and ARKs (for persistent links), among other standards or draft standards (e.g., OAIS reference model) Integrates the Extensible Text Framework (XTF) XML searching and publishing platform — see xtf.sourceforge.net

Common Framework Capabilities Presently: –Ingesting, managing, and preserving digital content –Serving XML to the Web –Searching XML metadata and content In Development: –Web crawling –OAI harvesting (based on the UofM harvester) –Metasearching (using the MetaLib X-Server)

A Visit to the Scene of the Crime : Part the 3 rd Installed first major upgrade to the API in August 2005 Still coding (no Alpha yet!) Next major upgrade to the API expected in late Spring 2005 Wherein the NitWit discovers that implementation schedules are to be broken … once … twice … thrice Interlude: An attempt by the NitWit to justify his Dubious Decisions and StaggeringMistakes

A panoply of Lame Excuses Most metasearch software is not designed for multiple, tailored deployments Product APIs are often incomplete or nonexistent We would rather delay deployment than give up on our model of tailored portals It wasn’t my fault Meanwhile, the abject failure of this NitWit is made all the more clear by the work of David Walker at CSU San Marcos, who Rocks!

The Agony is Nearly Over We conclude our Unfortunate Tale with advice for those who may be unable to avoid such Tragedy as befell this most Deserving Wretch

The Sad Morals of Our Tale Proprietary solutions sucketh; proprietary solutions without an API sucketh rocks “Excellent” is the sworn enemy of “easy” Neither early adopter nor latecomer be (Pain is best experienced by others) The first task of a Project Manager is to select upon whom to pin the Blame