1. www.sharon-it.com1 Choosing a Search Engine Taly Sharon Thanks to Ariel Frank, Bar-Ilan University taly@sharon-it.com sharont@alum.mit.edu

2. www.sharon-it.com2 82% Loyal to SE iProspect

3. www.sharon-it.com3 Search Engines Diverging Looking at the organic or natural listings for more than 485,000 first page search results, the study found that: Dogpile

4. www.sharon-it.com4 Experienced Searchers use More Search Engines HarvestDigital

5. www.sharon-it.com5 General rules for choosing SEs Use "major" SEs that are both well-known and well-used (and that hopefully won’t be downgraded or disappear soon ). Prefer SEs that employ both a huge index and a comprehensive directory (gives better results; can also switch between). Stick to SEs of established companies that treat search as their main business/expertise.

6. www.sharon-it.com6 Google Trends http://www.google.com/trends?q=google%2C+yahoo%2C+live%2C+ ask&ctab=0&geo=all&date=all&sort=0

7. www.sharon-it.com7 Criteria for Choosing SEs 1.Database (different) 2.Ranking algorithm 3.Query options (site, intitle, inrurl…) 4.Added values/features (clustering, define, NLP, …) 5.User Interface (UI)

8. www.sharon-it.com8 Who Powers Whom? Major distinct databases: –Google –Yahoo –MSN –Ask –Wisenut, Exalead, etc. The rest of the search engines use the same databases as the above search engines – different retrieval algorithms, see: http://www.ihelpyou.com/search-engine-chart.html http://www.ihelpyou.com/search-engine-chart.html

9. www.sharon-it.com9 SE Database Facts Summary Google is feeding from DMOZ Google is feeding Excite, Hotbot, iwon, Netscape and Aol search Yahoo! Is fed from Inktomi and feeding excite Ask is fed from Google and dmoz Directories –Yahoo! Is not fed from dmoz –But almost everyone else is!

10. www.sharon-it.com10 Google

11. www.sharon-it.com11 Yahoo!

12. www.sharon-it.com12 Ask

13. www.sharon-it.com13 Directories

14. www.sharon-it.com14 Why use Google? (1) Biggest, most comprehensive coverage:  ~8 billion Web pages (but ~1 billion of it isn’t full-text searchable!)  ~11 billion documents, if you count images and newsgroup postings. Fastest around. Most relevant results (voted 3 times most outstanding SE by Search Engine Watch readers). Provides good directory results (PageRanks results of DMOZ Open Directory).

15. www.sharon-it.com15 Why use Google? (2) Has thinnest/cleanest interface around. But provides rich set of advanced search features/tools(/hacks). Finds similar/related pages. Supports Web pages translation. Cached (HTML) copy of pages (great for quick view of DOCs/PDFs and for 404s ). Google alert – use of push technology.

16. www.sharon-it.com16 Share of Searches Share Of Searches: July 2006

17. www.sharon-it.com17 Why use Yahoo! search? (1) Has brand new Yahoo! search – gives highly relevant Web results (at Google level ). Still supports an expert’s humanly-compiled directory (dir.yahoo.com).dir.yahoo.com Has (also) a thin interface ( search.yahoo.com ) while providing a rich set of advanced search features/shortcuts. search.yahoo.com

18. www.sharon-it.com18 Why use Yahoo! search? (2) For legacy reasons (oldest of all directories). Puts particular emphasis on personalization and customization ( my.yahoo.com ). my.yahoo.com Had enough of Googlism ( www.googlism.com ). www.googlism.com It devoured/uses (know-how from) Overture (Inktomi, AltaVista and AllTheWeb, etc…) Has many specialty SEs – better than Google.

19. www.sharon-it.com19 Hidden Gem Yahoo! Search Subscriptions

20. www.sharon-it.com20 Google in 1998 – looking up at Yahoo!? Source: Internet archive’s Wayback machine www.archive.orgwww.archive.org

21. www.sharon-it.com21 Search Relevancy http://www.rustybrick.com/rustysearch-results.php

22. www.sharon-it.com22 6 Reasons to use Yahoo! 1.Long queries (>32 terms, >256 chars) –Especially useful when using OR 2.Search for XML/RSS 3.Better link: search More extensive results More options (linkdomain:, linksite:) 4.Mix syntax Link:http://mit.edu site:gov 5.Google is the most exposed to Spams. 6.Some special services.

23. www.sharon-it.com23 Why use MSN? Relatively new -- re-written in 2004. One of the 3 Major DBs. Direct answers -- from Microsoft Encarta®, encyclopedia. Direct actions -- to MSN channels. 1.When you need more results 2.When you need some unique query options: –prefer: –ip: –contains: (music contains:wma) –Feed:, hasfeed: 3.When you need UI options (especially sorting): Date Popularity Exact/approximate match

24. www.sharon-it.com24 Why use Ask? Small Index but interesting results Provides ExpertRank -“subject specific” ranking of pages. Provides a Natural Language interface (uses NLP). Refine: Suggests related searches.  Comments: Name AskJeeves changed to Ask Teoma gone with the Resources (results, refine, resources)

25. www.sharon-it.com25 Why Use Ask? Query suggestions/fill Q&A engine Smart Answers Query refinements Different results

26. www.sharon-it.com26 Ask

27. www.sharon-it.com27 Why Use Exalead New Search engine Another stand-alone database Advanced search features: –Words starting with –Words at proximity –Search method: exact search/automatic word stemming/phonetic search/approximate spelling –Document sorting: relevance/oldest/newest –Modification date: simply write date!!!!

28. www.sharon-it.com28 Why use Exalead Preferences – instant page translation Filters/refinements: –Related terms –Related categories (DMOZ) –Web site location –Document type (PDF/TXT/DOC/PPT) –Result presentation (documents/ documents+thumbnails/thumbnails) –Preview

29. www.sharon-it.com29 A9 Great UI Searches also books Visual Yellow pages and street photos Leader of innovative services –Search history –split view Good for obscure topics (because it searches books)

30. www.sharon-it.com30 Some Notes A9 – customize, special features AOL – good for beginners Looksmart – Findarticle Lycos – what people are talking about (people,forums) MSN – generally less results but growing Yahoo – MM, local/people searches and more Gigablast – site: search (but small index) up to 500 sites!

31. www.sharon-it.com31 GigaBlast

32. www.sharon-it.com32 Practical recommendations Two major SEs (usually use both): 1.Google (GG) 2.Yahoo! search (YH) or MSN One Meta-SE (as a backup): 3.Dogpile or Clusty Don’t forget the invisible web! Note: Choices are not Hebrew oriented.

33. www.sharon-it.com33 Hebrew Search Engines? MSN, Google, Yahoo Clusty (MSE) Netex (directory), Walla, Nana Morfix Start, a Many more (try Heb query of your favorite SE)

34. www.sharon-it.com34 Bibliography/Credits  http://reviews.cnet.com/4520-10572_7-6219242-1.html?tag=back http://reviews.cnet.com/4520-10572_7-6219242-1.html?tag=back  searchenginewatch.com searchenginewatch.com  searchengineshowdown.com/ searchengineshowdown.com/  www.noodletools.com/debbie/literacies/information/5locate/advicee ngine.html www.noodletools.com/debbie/literacies/information/5locate/advicee ngine.html  infopeople.org/search infopeople.org/search  www.lib.berkeley.edu/TeachingLib/Guides/Internet/ www.lib.berkeley.edu/TeachingLib/Guides/Internet/  www.monash.com/spidap.html www.monash.com/spidap.html  www.searchlore.org www.searchlore.org  http://www.linet-pro.net/nodeweb.asp?t=24462 (Hebrew) http://www.linet-pro.net/nodeweb.asp?t=24462  http://www.isedb.com/news/article/1094 http://www.isedb.com/news/article/1094  http://www.researchbuzz.com/FourThingsFinal.pdf http://www.researchbuzz.com/FourThingsFinal.pdf  http://www.researchbuzz.com/archives/001944.shtml http://www.researchbuzz.com/archives/001944.shtml  http://library.albany.edu/internet/choose.html http://library.albany.edu/internet/choose.html

35. www.sharon-it.com35 Exercises 1.Find the page this was quoted from: "EcoOcean cooperates with the Heschel Center in educating" 2.Find a page that has a Flash communication demo. 3.Who provides search feed to Netscape? 4.Search for pages, books, and pictures about the invisible web. 5.Find a picture of ABC Pizza House in Cambridge MA. 6.Find information about “Meryl Stripp”. You are not sure of the correct spelling (try with the given spelling). Which Search engine is useful here? 7.You get a list of 10 websites you want to run a query on. Which Search engine can run them together? –Example: "taly sharon" (site:acm.org OR site:dblp.com OR site:googleguide.co.il OR site:googleguide.com OR site:sharon-it.com OR site:ifla.org OR site:media.mit.edu OR technion.ac.il OR site:netanya.ac.il OR site:biu.ac.il) 8.What if you had 400 websites? 9.What is the west wing? Suggest options to narrow this search.