Algoritmi per IR Ranking. The big fight: find the best ranking...

Slides:



Advertisements
Similar presentations
Introduction to Information Retrieval Introduction to Information Retrieval Lecture 7: Scoring and results assembly.
Advertisements

Faster TF-IDF David Kauchak cs160 Fall 2009 adapted from:
Information Retrieval Lecture 6bis. Recap: tf-idf weighting The tf-idf weight of a term is the product of its tf weight and its idf weight. Best known.
| 1 › Gertjan van Noord2014 Zoekmachines Lecture 4.
Introduction to Information Retrieval (Manning, Raghavan, Schutze) Chapter 6 Scoring term weighting and the vector space model.
TF-IDF David Kauchak cs160 Fall 2009 adapted from:
Information Retrieval IR 7. Recap of the last lecture Vector space scoring Efficiency considerations Nearest neighbors and approximations.
CS276 Information Retrieval and Web Mining
Hinrich Schütze and Christina Lioma
Introduction to Information Retrieval Introduction to Information Retrieval Hinrich Schütze and Christina Lioma Lecture 7: Scores in a Complete Search.
Dictionary search Making one-side errors Paper on Bloom Filter.
Information Retrieval IR 6. Recap of the last lecture Parametric and field searches Zones in documents Scoring documents: zone weighting Index support.
Vector Space Model : TF - IDF
CES 514 Data Mining March 11, 2010 Lecture 5: scoring, term weighting, vector space model (Ch 6)
Chapter 5: Information Retrieval and Web Search
CSCI 5417 Information Retrieval Systems Jim Martin Lecture 6 9/8/2011.
CS276A Text Information Retrieval, Mining, and Exploitation Lecture 4 15 Oct 2002.
Information Retrieval Basic Document Scoring. Similarity between binary vectors Document is binary vector X,Y in {0,1} v Score: overlap measure What’s.
Documents as vectors Each doc j can be viewed as a vector of tf.idf values, one component for each term So we have a vector space terms are axes docs live.
Web search basics (Recap) The Web Web crawler Indexer Search User Indexes Query Engine 1 Ad indexes.
Boolean and Vector Space Models
Document ranking Paolo Ferragina Dipartimento di Informatica Università di Pisa.
Scoring, Term Weighting, and Vector Space Model Lecture 7: Scoring, Term Weighting and the Vector Space Model Web Search and Mining 1.
Web search basics (Recap) The Web Web crawler Indexer Search User Indexes Query Engine 1.
Faster TF-IDF David Kauchak cs458 Fall 2012 adapted from:
PrasadL09VSM-Ranking1 Vector Space Model : Ranking Revisited Adapted from Lectures by Prabhakar Raghavan (Google) and Christopher Manning (Stanford)
Computing Scores in a Complete Search System Lecture 8: Scoring and results assembly Web Search and Mining 1.
Information Retrieval and Web Search Lecture 7 1.
Term Weighting and Ranking Models Debapriyo Majumdar Information Retrieval – Spring 2015 Indian Statistical Institute Kolkata.
Quality of a search engine Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 8.
Information Retrieval Lecture 7. Recap of the last lecture Vector space scoring Efficiency considerations Nearest neighbors and approximations.
Basic ranking Models Boolean and Vector Space Models.
Xiaoying Gao Computer Science Victoria University of Wellington Intelligent Agents COMP 423.
Advanced topics in Computer Science Jiaheng Lu Department of Computer Science Renmin University of China
Information Retrieval and Data Mining (AT71. 07) Comp. Sc. and Inf
Chapter 6: Information Retrieval and Web Search
1 ITCS 6265 Information Retrieval and Web Mining Lecture 7.
Evaluation of (Search) Results How do we know if our results are any good? Evaluating a search engine  Benchmarks  Precision and recall Results summaries:
Introduction to Information Retrieval Introduction to Information Retrieval COMP4210: Information Retrieval and Search Engines Lecture 5: Scoring, Term.
Introduction to Information Retrieval (Manning, Raghavan, Schutze) Chapter 7 Computing scores in a complete search system.
Introduction to Information Retrieval CSE 538 MRS BOOK – CHAPTER VI SCORING, TERM WEIGHTING AND THE VECTOR SPACE MODEL 1.
Introduction to Information Retrieval Introduction to Information Retrieval Modified from Stanford CS276 slides Chap. 7: Scoring and results assembly.
Lecture 6: Scoring, Term Weighting and the Vector Space Model
Information Retrieval Techniques MS(CS) Lecture 7 AIR UNIVERSITY MULTAN CAMPUS Most of the slides adapted from IIR book.
Information Retrieval and Web Search IR models: Vector Space Model Instructor: Rada Mihalcea [Note: Some slides in this set were adapted from an IR course.
Introduction to Information Retrieval Introduction to Information Retrieval CS276 Information Retrieval and Web Search Chris Manning and Pandu Nayak Efficient.
Information Retrieval Quality of a Search Engine.
PrasadL09VSM-Ranking1 Vector Space Model : Ranking Revisited Adapted from Lectures by Prabhakar Raghavan (Yahoo and Stanford) and Christopher Manning (Stanford)
Xiaoying Gao Computer Science Victoria University of Wellington COMP307 NLP 4 Information Retrieval.
Information Retrieval Lecture 3 Introduction to Information Retrieval (Manning et al. 2007) Chapter 8 For the MSc Computer Science Programme Dell Zhang.
Introduction to Information Retrieval Introduction to Information Retrieval CS276 Information Retrieval and Web Search Christopher Manning and Prabhakar.
Introduction to Information Retrieval Introduction to Information Retrieval Lecture 9: Scoring, Term Weighting and the Vector Space Model.
Introduction to Information Retrieval Introduction to Information Retrieval CS276: Information Retrieval and Web Search Pandu Nayak and Prabhakar Raghavan.
Information Retrieval and Data Mining (AT71. 07) Comp. Sc. and Inf
The Vector Space Models (VSM)
Information Retrieval and Web Search
Top-K documents Exact retrieval
Ch 6 Term Weighting and Vector Space Model
Query processing: phrase queries and positional indexes
Prof. Paolo Ferragina, Algoritmi per "Information Retrieval"
Prof. Paolo Ferragina, Algoritmi per "Information Retrieval"
Prof. Paolo Ferragina, Algoritmi per "Information Retrieval"
Prof. Paolo Ferragina, Algoritmi per "Information Retrieval"
CS276 Lecture 7: Scoring and results assembly
CS276 Information Retrieval and Web Search
Term Frequency–Inverse Document Frequency
Presentation transcript:

Algoritmi per IR Ranking

The big fight: find the best ranking...

Ranking: Google vs Google.cn

Algoritmi per IR Text-based Ranking (1° generation)

Similarity between binary vectors Document is binary vector X,Y in {0,1} D Score: overlap measure What’s wrong ?

Normalization Dice coefficient (wrt avg #terms) : Jaccard coefficient (wrt possible terms) : OK, triangular NO, triangular

What’s wrong in doc-similarity ? Overlap matching doesn’t consider: Term frequency in a document Talks more of t ? Then t should be weighted more. Term scarcity in collection of commoner than baby bed Length of documents score should be normalized

A famous “weight”: tf-idf Frequency of term t in doc d = #occ t / |d| tf t,d where n t = #docs containing term t n = #docs in the indexed collection log n n idf t t         Vector Space model

Why distance is a bad idea Sec. 6.3

A graphical example Postulate: Documents that are “close together” in the vector space talk about the same things. Euclidean distance sensible to vector length !! t1t1 d2d2 d1d1 d3d3 d4d4 d5d5 t3t3 t2t2  cos(  ) = v  w / ||v|| * ||w|| The user query is a very short doc Easy to Spam Sophisticated algos to find top-k docs for a query Q

cosine(query,document) Dot product q i is the tf-idf weight of term i in the query d i is the tf-idf weight of term i in the document cos(q,d) is the cosine similarity of q and d … or, equivalently, the cosine of the angle between q and d. Sec. 6.3

Cos for length-normalized vectors For length-normalized vectors, cosine similarity is simply the dot product (or scalar product): for q, d length-normalized. 12

Cosine similarity amongst 3 documents termSaSPaPWH affection jealous10711 gossip206 wuthering0038 How similar are the novels SaS: Sense and Sensibility PaP: Pride and Prejudice, and WH: Wuthering Heights? Term frequencies (counts) Sec. 6.3 Note: To simplify this example, we don’t do idf weighting.

3 documents example contd. Log frequency weighting termSaSPaPWH affection jealous gossip wuthering After length normalization termSaSPaPWH affection jealous gossip wuthering cos(SaS,PaP) ≈ × × × × 0.0 ≈ 0.94 cos(SaS,WH) ≈ 0.79 cos(PaP,WH) ≈ 0.69 Why do we have cos(SaS,PaP) > cos(SAS,WH)? Sec. 6.3

Vector spaces and other operators Vector space OK for bag-of-words queries Clean metaphor for similar-document queries Not a good combination with operators: Boolean, wild-card, positional, proximity First generation of search engines Invented before “spamming” web search

Algoritmi per IR Top-k retrieval

Speed-up top-k retrieval Costly is the computation of the cos Find a set A of contenders, with K < |A| << N A does not necessarily contain the top K, but has many docs from among the top K Return the top K docs in A, according to the score The same approach is also used for other (non- cosine) scoring functions Will look at several schemes following this approach Sec

Index elimination Consider docs containing at least one query term. Hence this means… Take this further: 1. Only consider high-idf query terms 2. Only consider docs containing many query terms Sec

High-idf query terms only For a query such as catcher in the rye Only accumulate scores from catcher and rye Intuition: in and the contribute little to the scores and so don’t alter rank-ordering much Benefit: Postings of low-idf terms have many docs  these (many) docs get eliminated from set A of contenders Sec

Docs containing many query terms For multi-term queries, compute scores for docs containing several of the query terms Say, at least 3 out of 4 Imposes a “soft conjunction” on queries seen on web search engines (early Google) Easy to implement in postings traversal Sec

3 of 4 query terms Brutus Caesar Calpurnia Antony Scores only computed for docs 8, 16 and 32. Sec

Champion Lists Preprocess: Assign to each term, its m best documents Search: If |Q| = q terms, merge their preferred lists (  mq answers). Compute COS between Q and these docs, and choose the top k. Need to pick m>k to work well empirically. Now SE use tf-idf PLUS PageRank (PLUS other weights)

Complex scores Consider a simple total score combining cosine relevance and authority net-score(q,d) = PR(d) + cosine(q,d) Can use some other linear combination than an equal weighting Now we seek the top K docs by net score Sec

Advanced: Fancy-hits heuristic Preprocess: Assign docID by decreasing PR weight Define FH(t) = m docs for t with highest tf-idf weight Define IL(t) = the rest (i.e. incr docID = decr PR weight) Idea: a document that scores high should be in FH or in the front of IL Search for a t-term query: First FH: Take the common docs of their FH compute the score of these docs and keep the top-k docs. Then IL: scan ILs and check the common docs Compute the score and possibly insert them into the top-k. Stop when M docs have been checked or the PR score becomes smaller than some threshold.

Algoritmi per IR Speed-up querying by clustering

Visualization Query LeaderFollower Sec

Cluster pruning: preprocessing Pick  N docs at random: call these leaders For every other doc, pre-compute nearest leader Docs attached to a leader: its followers; Likely: each leader has ~  N followers. Sec

Cluster pruning: query processing Process a query as follows: Given query Q, find its nearest leader L. Seek K nearest docs from among L’s followers. Sec

Why use random sampling Fast Leaders reflect data distribution Sec

General variants Have each follower attached to b 1 =3 (say) nearest leaders. From query, find b 2 =4 (say) nearest leaders and their followers. Can recur on leader/follower construction. Sec

Algoritmi per IR Relevance feedback

Relevance Feedback Relevance feedback: user feedback on relevance of docs in initial set of results User issues a (short, simple) query The user marks some results as relevant or non-relevant. The system computes a better representation of the information need based on feedback. Relevance feedback can go through one or more iterations. Sec. 9.1

Rocchio’s Algorithm The Rocchio algorithm uses the vector space model to pick a relevance feed-back query Rocchio seeks the query q opt that maximizes Sec

Rocchio (SMART) Used in practice: D r = set of known relevant doc vectors D nr = set of known irrelevant doc vectors q m = modified query vector; q 0 = original query vector; α, β, γ : weights (hand-chosen or set empirically) New query moves toward relevant documents and away from irrelevant documents Sec

Relevance Feedback: Problems Users are often reluctant to provide explicit feedback It’s often harder to understand why a particular document was retrieved after applying relevance feedback There is no clear evidence that relevance feedback is the “best use” of the user’s time.

Relevance Feedback on the Web Some search engines offer a similar/related pages feature (this is a trivial form of relevance feedback) Google (link-based) Altavista Stanford WebBase Some don’t because it’s hard to explain to users: Alltheweb bing Yahoo Excite initially had true relevance feedback, but abandoned it due to lack of use. α / β / γ ?? Sec

Pseudo relevance feedback Pseudo-relevance feedback automates the “manual” part of true relevance feedback. Retrieve a list of hits for the user’s query Assume that the top k are relevant. Do relevance feedback (e.g., Rocchio) Works very well on average But can go horribly wrong for some queries. Several iterations can cause query drift. Sec

Query Expansion In relevance feedback, users give additional input (relevant/non-relevant) on documents, which is used to reweight terms in the documents In query expansion, users give additional input (good/bad search term) on words or phrases Sec

How augment the user query? Manual thesaurus (costly to generate) E.g. MedLine: physician, syn: doc, doctor, MD Global Analysis (static; all docs in collection) Automatically derived thesaurus (co-occurrence statistics) Refinements based on query-log mining Common on the web Local Analysis (dynamic) Analysis of documents in result set Sec

Query assist Would you expect such a feature to increase the query volume at a search engine?

Algoritmi per IR Zone indexes

Parametric and zone indexes Thus far, a doc has been a term sequence But documents have multiple parts: Author Title Date of publication Language Format etc. These are the metadata about a document Sec. 6.1

Zone A zone is a region of the doc that can contain an arbitrary amount of text e.g., Title Abstract References … Build inverted indexes on fields AND zones to permit querying E.g., “find docs with merchant in the title zone and matching the query gentle rain” Sec. 6.1

Example zone indexes Encode zones in dictionary vs. postings. Sec. 6.1

Tiered indexes Break postings up into a hierarchy of lists Most important … Least important Inverted index thus broken up into tiers of decreasing importance At query time use top tier unless it fails to yield K docs If so drop to lower tiers Sec

Example tiered index Sec

Query term proximity Free text queries: just a set of terms typed into the query box – common on the web Users prefer docs in which query terms occur within close proximity of each other Would like scoring function to take this into account – how? Sec

Query parsers E.g. query rising interest rates Run the query as a phrase query If <K docs contain the phrase rising interest rates, run the two phrase queries rising interest and interest rates If we still have <K docs, run the vector space query rising interest rates Rank matching docs by vector space scoring Sec

Algoritmi per IR Quality of a Search Engine

Is it good ? How fast does it index Number of documents/hour (Average document size) How fast does it search Latency as a function of index size Expressiveness of the query language

Measures for a search engine All of the preceding criteria are measurable The key measure: user happiness …useless answers won’t make a user happy

Happiness: elusive to measure Commonest approach is given by the relevance of search results How do we measure it ? Requires 3 elements: 1.A benchmark document collection 2.A benchmark suite of queries 3.A binary assessment of either Relevant or Irrelevant for each query-doc pair

Evaluating an IR system Standard benchmarks TREC: National Institute of Standards and Testing (NIST) has run large IR testbed for many years Other doc collections: marked by human experts, for each query and for each doc, Relevant or Irrelevant  On the Web everything is more complicated since we cannot mark the entire corpus !!

General scenario Relevant Retrieved collection

Precision: % docs retrieved that are relevant [issue “junk” found] Precision vs. Recall Relevant Retrieved collection Recall: % docs relevant that are retrieved [issue “info” found]

How to compute them Precision: fraction of retrieved docs that are relevant Recall: fraction of relevant docs that are retrieved Precision P = tp/(tp + fp) Recall R = tp/(tp + fn) RelevantNot Relevant Retrievedtp (true positive) fp (false positive) Not Retrievedfn (false negative) tn (true negative)

Some considerations Can get high recall (but low precision) by retrieving all docs for all queries! Recall is a non-decreasing function of the number of docs retrieved Precision usually decreases

Precision-Recall curve We measures Precision at various levels of Recall Note: it is an AVERAGE over many queries precision recall x x x x

A common picture precision recall x x x x

F measure Combined measure (weighted harmonic mean) : People usually use balanced F 1 measure i.e., with  = ½ thus 1/F = ½ (1/P + 1/R) Use this if you need to optimize a single measure that balances precision and recall.

Information Retrieval Recommendation Systems

Recommendations We have a list of restaurants with  and  ratings for some Which restaurant(s) should I recommend to Dave?

Basic Algorithm Recommend the most popular restaurants say # positive votes minus # negative votes What if Dave does not like Spaghetti?

Smart Algorithm Basic idea: find the person “most similar” to Dave according to cosine-similarity (i.e. Estie), and then recommend something this person likes. Perhaps recommend Straits Cafe to Dave  Do you want to rely on one person’s opinions?