Open Search David Wolber

Overview Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Webtop Metasearch Clients

Contributors Michael Kepe Igor Ranitovic Iman Sadreddin Senior Team ’03 Ken Chong Rudd Stevens Colin Bean Tim Chan Julian Chan Pooja Garg

Information Source Explosion Google, Amazon APIs Internet Archive Technorati– The World Live Web Technorati Domain Specific: – ACM Digital Library for CS – Lexis-Nexis for law – MLA for literature

End-User Created Digital Libraries Personal Web (shared Google desktop) Personal Web Neighborhood Topic-Specific Personal Crawlers Ordinary people creating search engines as easily as web pages 2 nd Degree 1 st Degree Nth Degree Personal Web

Subsets of the Web

Motivation for Small, Independent Subsets of the Web Avoid information being channeled through a single portal: Googleopoly Google does no evil, but… – Censorship in China – Creeping level of commercialization – Unregulated manipulation of secret ranking algorithms (see PageKing case) Other media is lost, this is the last frontier

Little support for using multiple search engines

Overview Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Webtop Metasearch Clients

Metasearch Help users discover and use digital libraries Send queries to multiple, selected search engines filter, process, and unify results A9.com – Amazon’s metasearch A9.com

Web Services Basis server html serversoftware xml server html Web Page Model Web Service Model

How does metasearch evolve? New Digital library

How does metasearch evolve? New Digital library Metasearch clients discover it

How does metasearch evolve? New Digital library Metasearch clients discover it Metasearch Programmers write adaptor/scraper

How does metasearch evolve? New Digital library Metasearch clients discover Metasearch Programmers write adaptor/scraper User can access within metasearch SLOWLY…

Overview Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Webtop Metasearch Clients

Goal: Automate the Process Metasearch engines should provide users with up-to- date lists of existing digital libraries Digital libraries should be able to register and be made immediately available to all Metasearch clients. Metasearch and Library development is independent.

What is Necessary? Standard Search API – So Metasearch clients can use polymorphism to access sources. for each source s in sourceList { searchEngine.endPointUrl = s.endPointUrl; resultList += searchEngine.keywordSearch(keywords) } Search API Registry – Metasearch clients can get dynamic list

Web Service Standards WSDL – Web Service Description Language SOAP – Simple Object Access Protocol UDDI – Universal Description, Discovery, and Integration

Standards on top of Web Services WSDL, SOAP, UDDI basis for standards in many domains. – e.g., MS initiated for securities information providers Businesses agree on a standard, then client applications can use polymorphism and new businesses can register services. In this case, we want cross-domain standard.

Open Search Architecture Open Search Protocol (OSP) – Cross-Domain: Search-related services – Not just keyword search, but citations, authorOf, etc. Open Search Registry – Based on UDDI – Can add customization, e.g., parsing to find out which search operations are implemented. – Web and web service access

Open Search Architecture OSP metasearch clients source list Register service OSP-Conforming Libraries OS Registry

User Can Choose Sources

Open Search Protocol Keyword search Citations (inward links, outward links) AuthorOf and other associative operations… Metadata object results based on Dublin Core Restriction object for “advanced search” stuff

Publishing a Library Access OSP WSDL Specification from webtop.cs.usfca.edu Generate code in language of choice Implement the search operations for the digital library Deploy the service Register with Open Search registry

Deploying an Open Search Lib. programmer 1. OS wsdl wsdl2java 2.wsdl 3. skeleton code Open Search information Registry Library server 4. deployed service 5. registration info

Wrapping a Library Custom search API, e.g., Google API Open Search Wrapper Metasearch Client 1. OSP Query4. OSP Result 2. Custom query 3. Custom Result Located on 3 rd party server

Wrappers Developed at USF Google Amazon (sort of) Internet Archive Technorati Feedster

Overview Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Webtop Metasearch Clients

PublishMe Like Google Desktop, but shared. Periodically updates inverse index and linkbase on PC Deploys Web Service on User’s PC Auto-Registers with Open Search Registry

Metasearch with P2P Knowledge Sharing WEBTOP

Integrating Global and Personal Libraries

Motivation for Sharing Personal Webs People create knowledge everyday when they bookmark, annotate, link, organize, and synthesize. Communication is a separate step which often doesn’t happen

Experts Collaborative Work Motivation for Sharing Personal Webs

Computers are designed using our brains for a model Knowledge creation and dissemination separate Explicit effort required to communicate Just as we model our word processors on paper.

Additions to OSP for P2P GetFile OnLine(ip) – Handles user starting up – Dynamic IPs OffLine

But What About PRIVACY? The Big Question: How much of the information hidden within your personal web is hidden due to privacy concerns?

I Want you to be a Search Engine!

Overview Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Metasearch Clients

Goal: Implement Vannevar Bush’s Association Trails View a document/thing in context History of an idea

Thinkmap-like Interface

Association Types Outward links Inward links Similar-Content links People Links – author, people referenced in paper Domain-Specific links – law citations – movie-actor Associations specified by Annotators

Webtop Tree View webtop.cs.usfca.edu

Expanding a Tree Bird’s Eye View Local/Web files integrated Follow different Associative Trails Ins of Outs of Ins, etc. Siblings Weird though, as ins and outs both expand right

Webtop Side Panel View

Project Status Too many bugs, Dad

Future Work Open Search Protocol – In-depth study of existing search APIs – Provide Rest alternative to SOAP Metasearch development – Complete and refine existing clients – Dream up new ones Thinkmap Graph Automated Source Selection and Reputation System Page Ranking Initiate grass-roots involvement

Future Work: Documents and Things resource associations annotations document htmlword pdf person filmbook creative work

Stop talking about Webtop daddy! webtop.cs.usfca.edu