1. Open Search David Wolber

2. Overview Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Webtop Metasearch Clients

3. Contributors Michael Kepe Igor Ranitovic Iman Sadreddin Senior Team ’03 Ken Chong Rudd Stevens Colin Bean Tim Chan Julian Chan Pooja Garg

4. Information Source Explosion Google, Amazon APIs Internet Archive Technorati– The World Live Web Technorati Domain Specific: – ACM Digital Library for CS – Lexis-Nexis for law – MLA for literature

5. End-User Created Digital Libraries Personal Web (shared Google desktop) Personal Web Neighborhood Topic-Specific Personal Crawlers Ordinary people creating search engines as easily as web pages 2 nd Degree 1 st Degree Nth Degree Personal Web

6. Subsets of the Web

7. Motivation for Small, Independent Subsets of the Web Avoid information being channeled through a single portal: Googleopoly Google does no evil, but… – Censorship in China – Creeping level of commercialization – Unregulated manipulation of secret ranking algorithms (see PageKing case) Other media is lost, this is the last frontier

8. Little support for using multiple search engines

9. Overview Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Webtop Metasearch Clients

10. Metasearch Help users discover and use digital libraries Send queries to multiple, selected search engines filter, process, and unify results A9.com – Amazon’s metasearch A9.com

11. Web Services Basis server html serversoftware xml server html Web Page Model Web Service Model

12. How does metasearch evolve? New Digital library

13. How does metasearch evolve? New Digital library Metasearch clients discover it

14. How does metasearch evolve? New Digital library Metasearch clients discover it Metasearch Programmers write adaptor/scraper

15. How does metasearch evolve? New Digital library Metasearch clients discover Metasearch Programmers write adaptor/scraper User can access within metasearch SLOWLY…

16. Overview Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Webtop Metasearch Clients

17. Goal: Automate the Process Metasearch engines should provide users with up-to- date lists of existing digital libraries Digital libraries should be able to register and be made immediately available to all Metasearch clients. Metasearch and Library development is independent.

18. What is Necessary? Standard Search API – So Metasearch clients can use polymorphism to access sources. for each source s in sourceList { searchEngine.endPointUrl = s.endPointUrl; resultList += searchEngine.keywordSearch(keywords) } Search API Registry – Metasearch clients can get dynamic list

19. Web Service Standards WSDL – Web Service Description Language SOAP – Simple Object Access Protocol UDDI – Universal Description, Discovery, and Integration

20. Standards on top of Web Services WSDL, SOAP, UDDI basis for standards in many domains. – e.g., MS initiated for securities information providers Businesses agree on a standard, then client applications can use polymorphism and new businesses can register services. In this case, we want cross-domain standard.

21. Open Search Architecture Open Search Protocol (OSP) – Cross-Domain: Search-related services – Not just keyword search, but citations, authorOf, etc. Open Search Registry – Based on UDDI – Can add customization, e.g., parsing to find out which search operations are implemented. – Web and web service access

22. Open Search Architecture OSP metasearch clients source list Register service OSP-Conforming Libraries OS Registry

23. User Can Choose Sources

24. Open Search Protocol Keyword search Citations (inward links, outward links) AuthorOf and other associative operations… Metadata object results based on Dublin Core Restriction object for “advanced search” stuff

25. Publishing a Library Access OSP WSDL Specification from webtop.cs.usfca.edu Generate code in language of choice Implement the search operations for the digital library Deploy the service Register with Open Search registry

26. Deploying an Open Search Lib. programmer 1. OS wsdl wsdl2java 2.wsdl 3. skeleton code Open Search information Registry Library server 4. deployed service 5. registration info

27. Wrapping a Library Custom search API, e.g., Google API Open Search Wrapper Metasearch Client 1. OSP Query4. OSP Result 2. Custom query 3. Custom Result Located on 3 rd party server

28. Wrappers Developed at USF Google Amazon (sort of) Internet Archive Technorati Feedster

29. Overview Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Webtop Metasearch Clients

30. PublishMe Like Google Desktop, but shared. Periodically updates inverse index and linkbase on PC Deploys Web Service on User’s PC Auto-Registers with Open Search Registry

31. Metasearch with P2P Knowledge Sharing WEBTOP

32. Integrating Global and Personal Libraries

33. Motivation for Sharing Personal Webs People create knowledge everyday when they bookmark, annotate, link, organize, and synthesize. Communication is a separate step which often doesn’t happen

34. Experts Collaborative Work Motivation for Sharing Personal Webs

35. Computers are designed using our brains for a model Knowledge creation and dissemination separate Explicit effort required to communicate Just as we model our word processors on paper.

36. Additions to OSP for P2P GetFile OnLine(ip) – Handles user starting up – Dynamic IPs OffLine

37. But What About PRIVACY? The Big Question: How much of the information hidden within your personal web is hidden due to privacy concerns?

38. I Want you to be a Search Engine!

39. Overview Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Metasearch Clients

40. Goal: Implement Vannevar Bush’s Association Trails View a document/thing in context History of an idea

41. Thinkmap-like Interface

42. Association Types Outward links Inward links Similar-Content links People Links – author, people referenced in paper Domain-Specific links – law citations – movie-actor Associations specified by Annotators

43. Webtop Tree View webtop.cs.usfca.edu

44. Expanding a Tree Bird’s Eye View Local/Web files integrated Follow different Associative Trails Ins of Outs of Ins, etc. Siblings Weird though, as ins and outs both expand right

45. Webtop Side Panel View

46. Project Status Too many bugs, Dad

47. Future Work Open Search Protocol – In-depth study of existing search APIs – Provide Rest alternative to SOAP Metasearch development – Complete and refine existing clients – Dream up new ones Thinkmap Graph Automated Source Selection and Reputation System Page Ranking Initiate grass-roots involvement

48. Future Work: Documents and Things resource associations annotations document htmlword pdf person filmbook creative work

49. Stop talking about Webtop daddy! webtop.cs.usfca.edu