Presentation is loading. Please wait.

Presentation is loading. Please wait.

Principles of IR Hacettepe University Department of Information Management DOK 324: Principles of IR.

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "Principles of IR Hacettepe University Department of Information Management DOK 324: Principles of IR."— Presentation transcript:

1 Principles of IR Hacettepe University Department of Information Management DOK 324: Principles of IR

2 Search engines Ray Larson Some Slides taken from: Ray Larson

3 “DOK 324 PIR” Y.T. The beginnings - Yahoo  Yet Another Hierarchical Officious Oracle  David Filo and Jerry Yang, Stanford University, spring 1994 –keep track of their personal interests on the Internet –converted later on onto a accessible database  fall 1994 - 1 million hits, 100,000 unique visitors  March 1995 - moved into business  Today…’also a search engine’ –But focused on offering other services –The search technology is actually licensed from Google

4 “DOK 324 PIR” Y.T. The current favourite - Google  Indexes –3,5 billion web pages (1.6 billion) –35 million non-HTML files (22 million) –700 million newsgroup messages (650 million) –250 million images  Serves 200 million queries / day (150 million) Note: the figures from last year are in brackets

5 “DOK 324 PIR” Y.T. Google’s ‘life of a query’ 3tiersystem Front-endFront-end DatabaseDatabase ProcessingProcessing

6 “DOK 324 PIR” Y.T. Why is it good? - technical reasons!  Powerful – cluster of 10,000 Linux servers  PageRank technology –A link from Page A to Page B is a "vote" by Page A for Page B.  The more links refer to page B, the higher page B will score –The score of page A will be used when voting for page B  The more important page A is, the higher page B will score  Hypertext-Matching Analysis: analyse page content in terms of headings, fonts, position, neighbours –Differentiate between ‘title’ text and ‘small-print’ text

7 “DOK 324 PIR” Y.T. What can go wrong?  Victim of its own success –Google becomes the web directory – information that cannot be found in it may be regarded as inexistent –Sued for: rank errors, addresses dropped from database  The attraction of money –‘bid-for-placing’ web searches – rank websites based on how much they have paid –Google is, after all, a business company

8 “DOK 324 PIR” Y.T. Search engines  Web Crawling  Web Search Engines and Algorithms

9 “DOK 324 PIR” Y.T. Standard Web Search Engine Architecture crawl the web create an inverted index Check for duplicates, store the documents Inverted index Search engine servers user query Show results To user DocIds

10 “DOK 324 PIR” Y.T. Web Crawling  How do the web search engines get all of the items they index?  Main idea: –Start with known sites –Record information for these sites –Follow the links from each site –Record information found at new sites –Repeat

11 “DOK 324 PIR” Y.T. Web Crawlers  How do the web search engines get all of the items they index?  More precisely: –Put a set of known sites on a queue –Repeat the following until the queue is empty:  Take the first page off of the queue  If this page has not yet been processed: –Record the information found on this page Positions of words, links going out, etc –Add each link on the current page to the queue –Record that this page has been processed  In what order should the links be followed?

12 “DOK 324 PIR” Y.T. Page Visit Order  Animated examples of breadth-first vs depth-first search on trees: –http://www.rci.rutgers.edu/~cfs/472_html/AI_SEARCH/ExhaustiveSearch.htmlhttp://www.rci.rutgers.edu/~cfs/472_html/AI_SEARCH/ExhaustiveSearch.html Structure to be traversed

13 “DOK 324 PIR” Y.T. Page Visit Order  Animated examples of breadth-first vs depth-first search on trees: –http://www.rci.rutgers.edu/~cfs/472_html/AI_SEARCH/ExhaustiveSearch.htmlhttp://www.rci.rutgers.edu/~cfs/472_html/AI_SEARCH/ExhaustiveSearch.html Breadth-first search (must be in presentation mode to see this animation)

14 “DOK 324 PIR” Y.T. Page Visit Order  Animated examples of breadth-first vs depth-first search on trees: –http://www.rci.rutgers.edu/~cfs/472_html/AI_SEARCH/ExhaustiveSearch.htmlhttp://www.rci.rutgers.edu/~cfs/472_html/AI_SEARCH/ExhaustiveSearch.html Depth-first search (must be in presentation mode to see this animation)

15 “DOK 324 PIR” Y.T. Page Visit Order  Animated examples of breadth-first vs depth-first search on trees: http://www.rci.rutgers.edu/~cfs/472_html/AI_SEARCH/ExhaustiveSearch.html

16 “DOK 324 PIR” Y.T. Sites Are Complex Graphs, Not Just Trees Page 1 Page 3 Page 2 Page 1 Page 2 Page 1 Page 5 Page 6 Page 4 Page 1 Page 2 Page 1 Page 3 Site 6 Site 5 Site 3 Site 1 Site 2

17 “DOK 324 PIR” Y.T. Web Crawling Issues  Keep out signs –A file called robots.txt tells the crawler which directories are off limits  Freshness –Figure out which pages change often –Recrawl these often  Duplicates, virtual hosts, etc –Convert page contents with a hash function –Compare new pages to the hash table  Lots of problems –Server unavailable –Incorrect html –Missing links –Infinite loops  Web crawling is difficult to do robustly!

18 “DOK 324 PIR” Y.T. Searching the Web  Web Directories versus Search Engines  Some statistics about Web searching  Challenges for Web Searching  Search Engines –Crawling –Indexing –Querying

19 “DOK 324 PIR” Y.T. Directories vs. Search Engines  Directories –Hand-selected sites –Search over the contents of the descriptions of the pages –Organized in advance into categories  Search Engines –All pages in all sites –Search over the contents of the pages themselves –Organized after the query by relevance rankings or other scores

20 “DOK 324 PIR” Y.T. Search Engines vs. Internal Engines  Not long ago HotBot, GoTo, Yahoo and Microsoft were all powered by Inktomi  Today Google is the search engine behind many other search services (such as Yahoo up until very recently and AOL’s search service)

21 “DOK 324 PIR” Y.T. Statistics from Inktomi  Statistics from Inktomi, August 2000, for one client, one week –Total # queries: 1315040 –Number of repeated queries: 771085 –Number of queries with repeated words: 12301 –Average words/ query: 2.39 –Query type: All words: 0.3036; Any words: 0.6886; Some words:0.0078 –Boolean: 0.0015 (0.9777 AND / 0.0252 OR / 0.0054 NOT) –Phrase searches: 0.198 –URL searches: 0.066 –URL searches w/http: 0.000 –email searches: 0.001 –Wildcards: 0.0011 (0.7042 '?'s )  frac '?' at end of query: 0.6753  interrogatives when '?' at end: 0.8456  composed of: –who: 0.0783 what: 0.2835 when: 0.0139 why: 0.0052 how: 0.2174 where 0.1826 where-MIS 0.0000 can,etc.: 0.0139 do(es)/did: 0.0

22 “DOK 324 PIR” Y.T. What Do People Search for on the Web?  Topics –Genealogy/Public Figure:12% –Computer related:12% –Business:12% –Entertainment: 8% –Medical: 8% –Politics & Government 7% –News 7% –Hobbies 6% –General info/surfing 6% –Science 6% –Travel 5% –Arts/education/shopping/images 14% (from Spink et al. 98 study)

23 “DOK 324 PIR” Y.T. Challenges for Web Searching: Data  Distributed data  Volatile data/”Freshness”: 40% of the web changes every month  Exponential growth  Unstructured and redundant data: 30% of web pages are near duplicates  Unedited data  Multiple formats  Commercial biases  Hidden data

24 “DOK 324 PIR” Y.T. Challenges for Web Searching: Users  Users unfamiliar with search engine interfaces (e.g., Does the query “apples oranges” mean the same thing on all of the search engines?)  Users unfamiliar with the logical view of the data (e.g., Is a search for “Oranges” the same things as a search for “oranges”?)  Many different kinds of users

25 “DOK 324 PIR” Y.T. Web Search Queries  Web search queries are SHORT –~2.4 words on average  User Expectations –Many say “the first item shown should be what I want to see”! –This works if the user has the most popular/common notion in mind

Download ppt "Principles of IR Hacettepe University Department of Information Management DOK 324: Principles of IR."

Similar presentations

Crawling, Ranking and Indexing. Organizing the Web The Web is big. Really big. –Over 3 billion pages, just in the indexable Web The Web is dynamic Problems:

Crawling, Ranking and Indexing. Organizing the Web The Web is big. Really big. –Over 3 billion pages, just in the indexable Web The Web is dynamic Problems:
2007.04.19 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.

SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.
“ The Anatomy of a Large-Scale Hypertextual Web Search Engine ” Presented by Ahmed Khaled Al-Shantout ICS 542 - 072.

“ The Anatomy of a Large-Scale Hypertextual Web Search Engine ” Presented by Ahmed Khaled Al-Shantout ICS
Information Retrieval in Practice

Information Retrieval in Practice
CpSc 881: Information Retrieval. 2 How hard can crawling be?  Web search engines must crawl their documents.  Getting the content of the documents is.

CpSc 881: Information Retrieval. 2 How hard can crawling be?  Web search engines must crawl their documents.  Getting the content of the documents is.
Best Web Directories and Search Engines Order Out of Chaos on the World Wide Web.

Best Web Directories and Search Engines Order Out of Chaos on the World Wide Web.
2011.04.18- SLIDE 1IS 240 – Spring 2011 Prof. Ray Larson University of California, Berkeley School of Information Principles of Information Retrieval Lecture.

SLIDE 1IS 240 – Spring 2011 Prof. Ray Larson University of California, Berkeley School of Information Principles of Information Retrieval Lecture.
2004.09.14 SLIDE 1IS 202 – FALL 2004 Lecture 05: Web Search Issues and Algorithms Prof. Ray Larson & Prof. Marc Davis UC Berkeley SIMS Tuesday and Thursday.

SLIDE 1IS 202 – FALL 2004 Lecture 05: Web Search Issues and Algorithms Prof. Ray Larson & Prof. Marc Davis UC Berkeley SIMS Tuesday and Thursday.
2009.04.20 - SLIDE 1IS 240 – Spring 2009 Prof. Ray Larson University of California, Berkeley School of Information Principles of Information Retrieval.

SLIDE 1IS 240 – Spring 2009 Prof. Ray Larson University of California, Berkeley School of Information Principles of Information Retrieval.
SIMS 202 Information Organization and Retrieval Prof. Marti Hearst and Prof. Ray Larson UC Berkeley SIMS Tues/Thurs 9:30-11:00am Fall 2000.

SIMS 202 Information Organization and Retrieval Prof. Marti Hearst and Prof. Ray Larson UC Berkeley SIMS Tues/Thurs 9:30-11:00am Fall 2000.
2010.04.12 - SLIDE 1IS 240 – Spring 2010 Prof. Ray Larson University of California, Berkeley School of Information Principles of Information Retrieval.

SLIDE 1IS 240 – Spring 2010 Prof. Ray Larson University of California, Berkeley School of Information Principles of Information Retrieval.
Anatomy of a Large-Scale Hypertextual Web Search Engine (e.g. Google)

Anatomy of a Large-Scale Hypertextual Web Search Engine (e.g. Google)
2003.11.20 SLIDE 1IS 202 – FALL 2003 Lecture 21: Web Search Issues and Algorithms Prof. Ray Larson & Prof. Marc Davis UC Berkeley SIMS Tuesday and Thursday.

SLIDE 1IS 202 – FALL 2003 Lecture 21: Web Search Issues and Algorithms Prof. Ray Larson & Prof. Marc Davis UC Berkeley SIMS Tuesday and Thursday.
1 ETT 429 Spring 2007 Microsoft Publisher II. 2 World Wide Web Terminology Internet Web pages Browsers Search Engines.

1 ETT 429 Spring 2007 Microsoft Publisher II. 2 World Wide Web Terminology Internet Web pages Browsers Search Engines.
Searching the Web. The Web Why is it important: –“Free” ubiquitous information resource –Broad coverage of topics and perspectives –Becoming dominant.

Searching the Web. The Web Why is it important: –“Free” ubiquitous information resource –Broad coverage of topics and perspectives –Becoming dominant.
ISP 433/633 Week 7 Web IR. Web is a unique collection Largest repository of data Unedited Can be anything –Information type –Sources Changing –Growing.

ISP 433/633 Week 7 Web IR. Web is a unique collection Largest repository of data Unedited Can be anything –Information type –Sources Changing –Growing.
2007.04.24 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.

SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.
CS 345 Data Mining Lecture 1 Introduction to Web Mining.

CS 345 Data Mining Lecture 1 Introduction to Web Mining.
2010.04.14 - SLIDE 1IS 240 – Spring 2010 Prof. Ray Larson University of California, Berkeley School of Information Principles of Information Retrieval.

SLIDE 1IS 240 – Spring 2010 Prof. Ray Larson University of California, Berkeley School of Information Principles of Information Retrieval.
Searching and Researching the World Wide: Emphasis on Christian Websites Developed from the book: Searching and Researching on the Internet and World Wide.

Searching and Researching the World Wide: Emphasis on Christian Websites Developed from the book: Searching and Researching on the Internet and World Wide.

Similar presentations

Ads by Google