Dr. Frank McCown Comp 250 – Web Development Harding University If a Web Page Can't be Googled, Does it Really Exist? What Web Developers Need to Know About Web Search Engines Dr. Frank McCown Comp 250 – Web Development Harding University This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License

Web Developers & Search Engines Search engines are the primary way users find information on the Web If a web page is not indexed by a search engine, it will not be seen (by many) Web developers need to know… How search engines work How to make pages discoverable How to make pages rank highly (SEO)

How do you locate information on the Web? When seeking information online, one must choose the best way to fulfill one’s information need Most popular: Web directories Search engines – primary focus of this lecture Social media

Web Directories Pages ordered in a hierarchy Usually powered by humans Yahoo started as a web directory in 1994 and still maintains one: http://dir.yahoo.com/ Open Directory Project (ODP) is largest and is maintained by volunteers http://www.dmoz.org/

Search Engines Most often used to fill an information need Users enter search terms into text box get back a SERP (search engine result page) Queries are generally modified and resubmitted to the SE if the desired results are not found on the first few pages of results Types of search engines: Intranet search engines (Nutch, Solr) Web search engines (Google, Bing, Baidu) Metasearch engines – includes Deep Web (Dogpile, WebCrawler) Vertical (or focused) search engines (Google Scholar, Google Shopping)

Components of a Search Engine Crawler: Downloads web pages and looks for links to new pages in the pages it downloads Indexer: Indexes the content in the pages downloaded by the crawler Search: User’s query is matched to data in the indexes and to the ad indexes and search results shown to the user Figure from Introduction to Information Retrieval by Manning et al., Ch 19.

Search query SERP Paid results Page title Organic results

Web Crawling Web crawlers or robots fetch a page, place all the page’s links in a queue, fetch the next link from the queue, and repeat Large search engines use thousands (millions?) of continually running web crawlers to (re-)discover web content Web crawlers are usually polite Identify themselves through the http User-Agent request header (e.g., googlebot) Throttle requests to a web server, crawl at off-peak times Honor robots exclusion protocol (robots.txt). Example: User-agent: * Disallow: /private

Halloween Easter Egg http://www.mattcutts.com/blog/google-protects-itself-from-zombies/

Indexing After crawling, web pages are indexed by the Indexer Inverted index is built containing the words and the documents that contain the words Example: cat > 2, 5  dog > 1, 5, 6  fish > 1, 2  bird > 4  Query for dog results in pages 1, 5, and 6 Query for dog and cat results in page 5

Google data center in Iowa https://www.google.com/about/datacenters/gallery/#/tech/1

Ranking Web Pages When a user enters a query, the search engine produces a SERP with the most “relevant” results How is relevancy determined? Broad range of techniques ranging from textual analysis to web graph analysis According to Google: “Relevancy is determined by over 200 factors…”

Textual Analysis Term frequency: How often does the page use the search term? Can easily be defeated by including more usage of the term, so usually there is a limit Location and emphasis: Where and how does the search term appear? Title, URL In bold, large font, headings Anchor text: Is the term used by others to create links to the web page? Google-bombing

Web Graph Analysis PageRank: Some pages are more “important” if they garner more links from other “important” web pages

Search Engine Optimization (SEO) Economic incentive to rank highly Cottage industry called SEO Process of getting traffic from search results White hat Create content that is meaningful and uses search terms in ideal locations Get others to create meaningful links, esp from social media Black hat Create web pages designed only for search engine consumption which trick it into thinking the page is about certain topics Create link farms to increase PageRank of certain pages

More Reading Introduction to Web Search Engines by McCown Google’s Webmaster Guidelines What is SEO?