1. Autumn 20111 Web Information retrieval (Web IR) Handout #0: Introduction Ali Mohammad Zareh Bidoki ECE Department, Yazd University alizareh@yaduni.ac.ir

2. Autumn 20112 Outline Web challenges Search engines Web crawling Web ranking –Ranking algorithms –Ranking challenges

3. Autumn 20113 Web Challenges Huge size of information –11.5 billions pages (2005) –64 billions pages (05 June, 2008) Proliferation and dynamic nature –New pages are created at the rate of 8% per week –Only 20% of the current pages will be accessible after one year –New links are created at rate 25% per week Heterogeneous contents –HTML/Text/Audio/…

4. Autumn 20114 Web Structure Web graph has Bow-tie shape It has scale-free topology –Many features of graph follow a power-law distribution –The core has small-world property the shortest directed path from any page in the core to any other page in the core involves 16–20 links on average

5. Autumn 20115 Web Retrieval User Space Information Space Matching Retrieval Browsing Index terms Full text Full text + Structure (e.g. hypertext) Search Engine

6. Autumn 20116 Search Engines Trends 625 million search queries are received by major search engines each day 80% of web surfers discover the new sites that they visit through search engines Web search currently generates more than 85% of the traffic to most web sites

7. Autumn 20117 Components of Search Engines Crawling Indexing Ranking

8. Autumn 20118 Architecture of Search Engines Crawler(s) Page Repository Indexer Module Collection Analysis Module Query Engine Ranking Client Indexes : Text Structure Utility Queries Results Web

9. Autumn 20119 Web Crawling Issues Coverage –Google, the biggest search engine, covers only 70% of web content –We must focus on high quality pages Freshness –Keep the copy in synchronize with the source pages Politeness –Do it without disrupting the web and obeying the webmasters constrains

10. Autumn 201110 Web Crawling Issues

11. Autumn 201111 Web crawling Crawler

12. Autumn 201112 Crawling Scheduling Breadth-First Back-link count PageRank,…

13. Autumn 201113 Crawling scheduling Downloader Web Web Repository Ranking Algorithm URLs and Links

14. Autumn 201114 Indexing Text Operations forms index words (tokens). –Stopword removal –Stemming Indexing constructs an inverted index of word to document pointers.

15. Autumn 201115 Comparing IR to databases ( vs data retrieval ) DatabasesIR Data StructuredUnstructured Fields Clear semantics (SSN, age) No fields (other than text) Queries Defined (relational algebra, SQL) Free text (“natural language”), Boolean Query specification CompleteIncomplete Matching Exact (results are always “correct”) Imprecise (need to measure effectiveness) Error response SensitiveInsensitive

16. Autumn 201116 Indexing Systems Google file system MG4J (Managing Gigabytes for Java) Lucene (Java-GPL) Swish-e (C++-Linux)

17. Autumn 201117 Ranking : Definition Ranking is the process which estimates the quality of a set of results retrieved by a search engine Ranking is the most important part of a search engine

18. Autumn 201118 Ranking Types Content-based –Classical IR Connectivity based (web) –Query independent –Query dependent User-behavior based

19. Autumn 201119 Ranking is a function of query term frequency within the document (tf) and across all documents (idf) –Vector space –Probabilistic Classical Information Retrieval WordsDocs 1 2 w 1 2 n Query

20. Autumn 201120 Classical Information Retrieval This works because of the following assumptions in classical IR: –Queries are long and well specified –Documents (e.g., newspaper articles) are coherent, well authored, and are usually about one topic –The vocabulary is small and relatively well understood

21. Autumn 201121 Web information retrieval Queries are short: 2.35 terms in avg. Huge variety in documents: language, quality, duplication Huge vocabulary: 100s millions terms Deliberate misinformation Spamming! –Its rank is completely under the control of Web page’s author

22. Autumn 201122 Ranking in Web IR Ranking is a function of the query terms and of the hyperlink structure –Using content of other pages to rank current pages It is out of the control of the page’s author –Spamming is hard WordsDocs 1 2 w 1 2 n 1 2 n Web graph Query

23. Autumn 201123 Books –Introduction to Information Retrieval, by Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schuetze –Modern Information Retrieval, by Ricardo Baeza-Yates & Berthier Ribeiro-Neto, Addison-Wisley, 1999.

24. Autumn 201124 Grading Exam: 50% Project & Homework: 30% Paper Review:10% A paper presentation 10%

25. Web Site http://ce.yazduni.ac.ir/zareh/courses/webi r/ Autumn 201125

26. Next paper for Review Impact of Search Engines on Page Popularity by Cho Autumn 201126

27. Autumn 201127 Course Outline Web Structure Crawling/Ranking/Indexing in Web search engines Retrieval in Persian documents –Query Processing –Indexing solutions Cross-language Information Retrieval Semantic web

28. Next Paper for Review Impact of Search Engines on Page Popularity, by cho Autumn 201128