Search Engine 101 Qu, Miao Nov. 2003

Agenda Definition and Types Architecture Robot Overview How Search Engine Works? Problems of Current Search Engines An example: Google The Future of Search Engine Search Engine vs. Directory Reference

What Is Search Engine? Search engines are tools that use computer programs called Spiders and Robots to gather information automatically. They can create specific databases according to the query of the user. Source: “Authority Guide to Evaluating Information on the Internet”, Alison Cooke, 1999

The Types of Search Engines Individual Search Engines compile their own searchable databases on the web. Google. Meta Search Engines do not compile databases. Instead, they search the databases of multiple sets of individual engines simultaneously Metacrawler, vivisimo Source: http://www.sc.edu/beaufort/library/lesson1.html

Web Search Engine Layers From description of the FAST search engine, by Knut Risvik http://www.infonortics.com/searchengines/sh00/risvik_files/frame.htm

Standard Web Search Engine Architecture Check for duplicates, store the documents DocIds crawl the web user query create an inverted index Inverted index Search engine servers Show results To user Source:http://www.sims.berkeley.edu/academics/courses/is202/f02/Lectures/Lecture22_2002_11_14_tbd.ppt

Anatomy of Search Engine Working Procedures Crawling the web (Robot) Establish the database (Robot) Query (searcher) Search the Database (Search Engine Software) Ranking (Search Engine Software) Interface with client Components Robot Index catalog, database of what the spider finds Search Engine Software program that sifts through the millions of pages recorded in the index to find matches to a search and rank them in order of what it believes is most relevant. Source:http://searchenginewatch.com/webmasters/article.php/2168031

Robot Overview It is essential ingredient of all current web search tools. A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced.It could be written in Perl, Java, C, C++ or others (e.g. Tcl/Tk) Also be known as: spiders, wanderers, worms, crawlers, gatherers, intelligent agents Could have other functions: Measuring the size and scope of the web; Maintaining a database of web page by checking old links for updates and relocation; Mirroring sites; Email address harvesting; Etc. (Source: Susan Maze, David Moxley and Donna J. Smith, “Authoritative Guide to Web Search Engines”, 1997, P13. http://www.robotstxt.org/wc/faq.html#what)

Robot Overview (cont.) Interested in getting source code? http://webharvest.sourceforge.net/ng/ (harvest, perl) http://www.lub.lu.se/combine/ (combine, perl) http://www.acme.com/java/software/Acme.Spider.html (Acme. spider, Java)

Establishing Database “It is important to remember that when you are using a search engine, you are NOT searching the entire web as it exists at this moment. You are actually searching a portion of the web, captured in a fixed index created at an earlier date.” Source: http://www.sc.edu/beaufort/library/lesson1.html

How Robot Searches Web Pages Robot does not wander in the web itself. It use HTTP to require documents from server. “In general they start from a historical list of URLs, especially of documents with many links elsewhere, such as server lists, "What's New" pages, and the most popular sites on the Web.” Strategy: Depth-first: create a relatively comprehensive database on a few subjects; Breadth-first: create databases touching more lightly on a wider variety of documents. (Source: Susan Maze, David Moxley and Donna J. Smith, “Authoritative Guide to Web Search Engines”, 1997, P13. http://www.robotstxt.org/wc/faq.html#what )

If I Don’t Want to Be Indexed by Robot? Robot.tex A plaint text document which would be checked by robots; An example; Robots.txt implements the Robots Exclusion Protocol, which allows the web site administrator to define what parts of the site are off-limits to specific robot user agent names. Robots META tag Sample entry: <META name="ROBOTS" content="NOINDEX"> <META name="ROBOTS" ontent="NOFOLLOW"> Many, but not all, search engine robots will recognize this tag and follow the rules for each page http://www.searchtools.com/robots/

Ranking I Once a search engine has used your search terms to gather "hits" from its database, it lists or "ranks" the resulting sites in order of its own estimation of their relevance. In most cases, the rule for ranking is the Relevance Prediction. Currently, search engines predict relevance based on two sets of factors: those based on a site's content ; those external to the site: http://www.searchengines.com/searchBasics1.html

Ranking II Factors based on a web site's content Word frequency (How often search terms occur in a page in relationship to other text) Location of search terms in the document (Are they in the title? Are they near the top of the page?) Relational clustering (How many pages in the site contain the search terms?) The site's design (Does it use frames? How fast does it load?) http://www.searchengines.com/searchBasics1.html

Ranking III Factors external to the site Link popularity -- Sites with more links pointing to them are prioritized Click popularity -- Sites visited more often are prioritized "Sector" popularity -- Sites visited by certain demographic or social groups are prioritized (Note: This system requires user-provided information) Business alliances among services -- Results from a partner search service are ranked higher Pay-for-placement rankings -- Site owners pay for high rankings http://www.searchengines.com/searchBasics1.html

How Search Engine Different from One Another? The Robot The database How is the database cleaned up and filtered? The frequency with which sites are spidered affects the database's freshness. The formula (Algorithms) Different search engines employ different search retrieval formulas, or algorithms, to provide relevant content in response to a user's query. Features and functionality The various search engines have different bells and whistles to appeal to searchers' different experience levels or individual tastes. The look Engines' graphical user interfaces vary, as do the format in which they present their results. 11/18/2018 http://www.searchengines.com/searchDiffer1.html

What Are the Problems of Current Search Engines? The biggest problem facing users of web search engines today is the quality of the results they get back. While the results are often amusing and expand users’ horizons, they are often frustrating and consume precious time. The Anatomy of a Large-Scale Hypertextual Web Search Engine Sergey Brin and Lawrence Page

Challenges in Search Engines Search Engine Spam; Quality of Content; Quality Evaluation; Web Conventions; Avoid Duplicate Search (Host); Vaguely Structured Data Source: Challenges in Search Engines, ACM SIGIR Forum, Volume 36 ,  Issue 2   Fall 2002 , Monika R. Henzinger , Rajeev Motwani, Craig Silverstein

An Example: Google http://www.bu.edu/mfeldman/Google/

Some Features: What's not indexed Content and location HTML Title Registration pages, text in graphics and multimedia files (use Alt tags), XML, Java applets, comment tags, Acrobat files, spammers. Content and location Keywords should be close to each other; Content should include keywords in text or links. HTML Title Seems to be a fact. Meta tags No. Link popularity Very important, especially from relevant page.

Architecture Source: “ The Anatomy of a Large-Scale Hypertextual Web Search Engine”, Sergey Brin and Lawrence Page

Main Technology Applied in Google: PageRankTM: A system for ranking web pages Anchor Text Source: “ The Anatomy of a Large-Scale Hypertextual Web Search Engine”, Sergey Brin and Lawrence Page

The Future of Search Engine Theme Search Engine best match between the page content and the evaluation of its page. Get more straightforward answers. More customized

Directories vs. Search Engines Hand-selected sites Search over the contents of the descriptions of the pages Organized in advance into categories Search Engines All pages in all sites Search over the contents of the pages themselves Organized after the query by relevance rankings or other scores Source:http://www.sims.berkeley.edu/academics/courses/is202/f02/Lectures/Lecture22_2002_11_14_tbd.ppt

Reference Alison Cooke, “Authority Guide to Evaluating Information on the Internet”, 1999, http://www.sc.edu/beaufort/library/lesson1.html Danny Sullivan, “How Search Engines Work”, 2002, http://searchenginewatch.com/webmasters/article.php/2168031. Knut Risvik, “From description of the FAST search engine”, ? http://www.infonortics.com/searchengines/sh00/risvik_files/frame.htm Berkeley University, 2002, Source:http://www.sims.berkeley.edu/academics/courses/is202/f02/Lectures/Lecture22_2002_11_14_tbd.ppt http://www.robotstxt.org/wc/faq.html#what) Susan Maze, David Moxley and Donna J. Smith, “Authoritative Guide to Web Search Engines”, 1997, P13. http://www.searchtools.com/robots/ http://www.searchengines.com/searchBasics1.html http://www.searchengines.com/textkeywords.htm

Reference http://www.searchengines.com/urlkeywords.html http://www.searchengines.com/ranking_factors.html http://www.searchengines.com/searchDiffer1.html www.google.com Danny Sullivan, “Major Search Engines and Directories”, 2003, http://searchenginewatch.com/links/article.php/2156221 www.alltheweb.com http://www.searchengines.com/partnerships.html The Anatomy of a Large-Scale Hypertextual Web Search Engine, Sergey Brin and Lawrence Page, http://google.stanford.edu/~backrub/google.html. Monika R. Henzinger, Rajeev Motwani, Craig Silverstein, 2002, “Challenges in Web Search Engines”, SIGIR FORUM, Fall 2002, Vol.36, No.02. Robin Nobles, 2003, “The Future Of Search Engine Optimizing”, http://www.searchengineworkshops.com/articles/se-optimization-future.html Gary H. Anthes, 2002, “The Future of the Search Engine”, http://www.computerworld.com/databasetopics/data/story/0,10801,70037,00.html