2. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Lecture 5: Search Engines

3. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Outline Search engines: key tools for ecommerce –Buyers and sellers must find each other How do they work? How much do they index? Are they reliable? How are hits ordered? Can the order be changed?

4. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Search Engines Tools for finding information on the Web –Problem: “hidden” databases, e.g. New York TimesNew York Times Directory –A hand-constructed hierarchy of topics (e.g. Yahoo)Yahoo Search engine –A machine-constructed index (usually by keyword) So many search engines, we now need search engines to find them. Searchenginecollosus.comSearchenginecollosus.com

5. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Indexing Arrangement of data (data structure) to permit fast searching Which list is easier to search? sow fox pig eel yak hen ant cat dog hog ant cat dog eel fox hen hog pig sow yak Sorting helps. Why? –Permits binary search. About log 2 n probes into list log 2 (1 billion) ~ 30 –Permits interpolation search. About log 2 (log 2 n) probes log 2 log 2 (1 billion) ~ 5

6. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Inverted Files A file is a list of words by position –First entry is the word in position 1 (first word) –Entry 4562 is the word in position 4562 (4562 nd word) –Last entry is the last word An inverted file is a list of positions by word! POS 1 10 20 30 36 FILE a (1, 4, 40) entry (11, 20, 31) file (2, 38) list (5, 41) position (9, 16, 26) positions (44) word (14, 19, 24, 29, 35, 45) words (7) 4562 (21, 27) INVERTED FILE

7. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Inverted Files for Multiple Documents DOCID OCCUR POS 1 POS 2...... “jezebel” occurs 6 times in document 34, 3 times in document 44, 4 times in document 56... LEXICON WORD INDEX

8. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Search Engine Architecture Spider –Crawls the web to find pages. Follows hyperlinks. Never stops Indexer –Produces data structures for fast searching of all words in the pages Retriever –Query interface –Database lookup to find hits 2 billion documents 4 TB RAM, many terabytes of disk –Ranking

9. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Crawlers (Spiders, Bots) Retrieve web pages for indexing by search engines Start with an initial page P 0. Find URLs on P 0 and add them to a queue When done with P 0, pass it to an indexing program, get a page P 1 from the queue and repeat Can be specialized (e.g. only look for email addresses) Issues –Which page to look at next? (Special subjects, recency) –Avoid overloading a site –How deep within a site to go (drill-down)? –How frequently to visit pages?

10. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Query Specification Boolean –AND, OR, NOT, PHRASE “ ”, NEAR ~ –But keyword query is artificial Question-answering (simulated) –“Who offers a master’s degree in ecommerce? Date range Relevance specification –In Altavista, can specify terms by importance (separate from query specification) Content –multimedia, MP3,.PPT files Stemming

11. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS “Advanced” Query Specification Multimedia, e.g. GoogleGoogle Date range Relevance specification –In Altavista, can specify terms by importance (separate from query specification) Content –multimedia, MP3,.PPT files Stemming Language Search depth (from site’s front page)

12. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Ranking (Scoring) Hits Hits must be presented in some order What order? –Relevance, recency, popularity, reliability? Some ranking methods –Presence of keywords in title of document –Closeness of keywords to start of document –Frequency of keyword in document –Link popularity (how many pages point to this one) Can the user control? Can the page owner control? Can you find out what order is used? Spamdexing: influencing retrieval ranking by altering a web page. (Puts “spam” in the index)

13. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Link Popularity How many pages link to this page? –on the whole Web –in our database? www.linkpopularity.com Link popularity is used for ranking –Many measures –Number of links in –Weighted number of links in (by weight of referring page)

14. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Mercator Spider Structure 1. Remove URL from queue 2. Simulate network protocols & REP 3. Read in RewindInputStream (RIS) mode 4. Has this doc been seen before? (Uses checksums and fingerprints) 5. Extract links 6. Should we download new URL? 7. Has new URL been seen before? 8. Add URL to frontier Document fingerprints

15. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Mercator Statistics HISTOGRAM OF DOCUMENT SIZES PAGE TYPE PERCENT text/html 69.2% image/gif 17.9% image/jpeg 8.1% text/plain 1.5 pdf 0.9% audio 0.4% zip 0.4% postscript 0.3% other 1.4% Exponentially increasing size

16. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Search Engine Sizes (Dec. 11, 2001) SOURCE: SEARCHENGINEWATCH.COMSEARCHENGINEWATCH.COM AVAltavista EXExciteFAST GGGoogle GoGo (Infoseek) INKInktomi NLNorthern Light WTWebTop.com SHADED DATA FOR GG AND INKTOMI INCLUDES PAGES INDEXED BUT NOT VISITED SEARCHES/DAY (MILLIONS) 150 12 50 80 1700 per second!

17. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Search Engines Disjointness SOURCE: SEARCHENGINESHOWDOWN

18. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Search Engine EKG SOURCE: SEARCHENGINEWATCH.COM Shows activity of the Lycos crawler at one sample site, calafia.com, by number of pages visited during each crawl

19. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Search Engine EKG Comparison SOURCE: SEARCHENGINEWATCH.COM

20. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Search Engine Differences Coverage (number of documents) Spidering algorithms (visit SpiderCatcher)SpiderCatcher –Frequency, depth of visits Inexing policies Search interfaces Ranking One solution: use a metasearcher (search agent)

21. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Metasearchers All the engines operate differently. Different –sizes –query languages –crawling algorithms –storage policies (stop words, punctuation, fonts) –freshness –ranking Submit the same query to many engines and collect the results Metacrawler

22. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Search Spying Peeking at queries as they are being submitted Metaspy. Spies on MetacrawlerMetaspy AskJeeves TheSearchSite Epicurious (recipes)Epicurious StockCharts.com Yahoo buzz indexYahoo Kanoodle

23. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Time Spent Per Visitor (minutes) by Search Engine, Feb. 2002 SOURCE: JUPITERJUPITER AJAsk JeevesAsk Jeeves AOLAmerica Online AVAltavistaAltavista EXExcite GGGoogle ISPInfoSpaceInfoSpace LSLookSmartLookSmart LYLycosMSN NSNetscape OVROvertureOverture YHYahooYahoo

24. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Audience Reach by Search Engine, Mar. 2002 SOURCE: JUPITERJUPITER AJAsk Jeeves AOLAmerica Online AVAltavista GGGoogle ISPInfoSPace LSLookSmart LYLycos NSNetscape OVROVERTURE YHYahoo Audience Reach = % of active surfers visiting during month. Totals exceed 100% because of overlap

25. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Key Takeaways Engines are a critical Web resource Very sophisticated, high technology But they’re secret They don’t cover the Web comletely Spamdexing is a problem New paradigms needed as Web grows –Keywords are not enough. Try hyperbolic treeshyperbolic trees What about images, music, video? –www.corbis.comwww.corbis.com

26. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Q A &

27. Robot Exclusion You may not want certain pages indexed but still viewable by browsers. Can’t protect directory. Some crawlers conform to the Robot Exclusion Protocol. Compliance is voluntary. One way to enforce: firewall They look for file robots.txt at highest directory level in domain. If domain is www.ecom.cmu.edu, robots.txt goes in www.ecom.cmu.edu/robots.txt A specific document can be shielded from a crawler by adding the line:

28. 20-751 ECOMMERCE TECHNOLOGY SUMMER 2002 COPYRIGHT © 2002 MICHAEL I. SHAMOS Robots Exclusion Protocol Format of robots.txt –Two fields. User-agent to specify a robot –Disallow to tell the agent what to ignore To exclude all robots from a server: User-agent: * Disallow: / To exclude one robot from two directories: User-agent: WebCrawler Disallow: /news/ Disallow: /tmp/ View the robots.txt specification.specification