Lecture 2: Retrieval Models Maya Ramanath. QQ1 Vector space model: 0 for non-presence of a term, 1 for presence: Query: q1 AND q2 AND q3 Compare the set.

Slides:

Advertisements

Similar presentations

Relevance Feedback User tells system whether returned/disseminated documents are relevant to query/information need or not Feedback: usually positive sometimes.

Advertisements

Information Retrieval and Organisation Chapter 12 Language Models for Information Retrieval Dell Zhang Birkbeck, University of London.

Information Retrieval and Organisation Chapter 11 Probabilistic Information Retrieval Dell Zhang Birkbeck, University of London.

Improvements and extras Paul Thomas CSIRO. Overview of the lectures 1.Introduction to information retrieval (IR) 2.Ranked retrieval 3.Probabilistic retrieval.

Language Models Naama Kraus (Modified by Amit Gross) Slides are based on Introduction to Information Retrieval Book by Manning, Raghavan and Schütze.

Basic IR: Modeling Basic IR Task: Slightly more complex:

1 Language Models for TR (Lecture for CS410-CXZ Text Info Systems) Feb. 25, 2011 ChengXiang Zhai Department of Computer Science University of Illinois,

INSTRUCTOR: DR.NICK EVANGELOPOULOS PRESENTED BY: QIUXIA WU CHAPTER 2 Information retrieval DSCI 5240.

Probabilistic Information Retrieval Chris Manning, Pandu Nayak and

Introduction to Information Retrieval Information Retrieval and Data Mining (AT71.07) Comp. Sc. and Inf. Mgmt. Asian Institute of Technology Instructor:

CpSc 881: Information Retrieval

Probabilistic Ranking Principle

Bag-of-Words Methods for Text Mining CSCI-GA.2590 – Lecture 2A

Ranking models in IR Key idea: We wish to return in order the documents most likely to be useful to the searcher To do this, we want to know which documents.

Information Retrieval Models: Probabilistic Models

IR Models: Overview, Boolean, and Vector

Hinrich Schütze and Christina Lioma

Database Management Systems, R. Ramakrishnan1 Computing Relevance, Similarity: The Vector Space Model Chapter 27, Part B Based on Larson and Hearst’s slides.

SLIDE 1IS 240 – Spring 2010 Logistic Regression The logistic function: The logistic function is useful because it can take as an input any.

Introduction to Information Retrieval Introduction to Information Retrieval Hinrich Schütze and Christina Lioma Lecture 12: Language Models for IR.

Introduction to Information Retrieval Introduction to Information Retrieval Hinrich Schütze and Christina Lioma Lecture 11: Probabilistic Information Retrieval.

1 CS 430 / INFO 430 Information Retrieval Lecture 12 Probabilistic Information Retrieval.

Probabilistic IR Models Based on probability theory Basic idea : Given a document d and a query q, Estimate the likelihood of d being relevant for the.

Modern Information Retrieval Chapter 2 Modeling. Can keywords be used to represent a document or a query? keywords as query and matching as query processing.

Modeling Modern Information Retrieval

Vector Space Model CS 652 Information Extraction and Integration.

The Vector Space Model …and applications in Information Retrieval.

1 CS 430 / INFO 430 Information Retrieval Lecture 10 Probabilistic Information Retrieval.

Retrieval Models II Vector Space, Probabilistic.  Allan, Ballesteros, Croft, and/or Turtle Properties of Inner Product The inner product is unbounded.

Introduction to Language Models Evaluation in information retrieval Lecture 4.

Modern Information Retrieval Chapter 2 Modeling. Can keywords be used to represent a document or a query? keywords as query and matching as query processing.

IR Models: Review Vector Model and Probabilistic.

CS246 Basic Information Retrieval. Today’s Topic  Basic Information Retrieval (IR)  Bag of words assumption  Boolean Model  Inverted index  Vector-space.

Chapter 5: Information Retrieval and Web Search

INFORMATION RETRIEVAL VECTOR SPACE MODEL IN-DEPTH PART 3 Thomas Tiahrt, MA, PhD CSC492 – Advanced Text Analytics.

INFORMATION RETRIEVAL VECTOR SPACE MODEL IN-DEPTH PART 2 Thomas Tiahrt, MA, PhD CSC492 – Advanced Text Analytics.

CSCI 5417 Information Retrieval Systems Jim Martin Lecture 6 9/8/2011.

Probabilistic Models in IR Debapriyo Majumdar Information Retrieval – Spring 2015 Indian Statistical Institute Kolkata Using majority of the slides from.

Modeling (Chap. 2) Modern Information Retrieval Spring 2000.

1 Vector Space Model Rong Jin. 2 Basic Issues in A Retrieval Model How to represent text objects What similarity function should be used? How to refine.

Modern Information Retrieval: A Brief Overview By Amit Singhal Ranjan Dash.

Xiaoying Gao Computer Science Victoria University of Wellington Intelligent Agents COMP 423.

Weighting and Matching against Indices. Zipf’s Law In any corpus, such as the AIT, we can count how often each word occurs in the corpus as a whole =

Term Frequency. Term frequency Two factors: – A term that appears just once in a document is probably not as significant as a term that appears a number.

Context-Sensitive Information Retrieval Using Implicit Feedback Xuehua Shen : department of Computer Science University of Illinois at Urbana-Champaign.

Chapter 6: Information Retrieval and Web Search

1 Computing Relevance, Similarity: The Vector Space Model.

Introduction to Digital Libraries hussein suleman uct cs honours 2003.

CPSC 404 Laks V.S. Lakshmanan1 Computing Relevance, Similarity: The Vector Space Model Chapter 27, Part B Based on Larson and Hearst’s slides at UC-Berkeley.

Comparing and Ranking Documents Once our search engine has retrieved a set of documents, we may want to Rank them by relevance –Which are the best fit.

Lecture 1: Overview of IR Maya Ramanath. Who hasn’t used Google? Why did Google return these results first ? Can we improve on it? Is this a good result.

LANGUAGE MODELS FOR RELEVANCE FEEDBACK Lee Won Hee.

Chapter 23: Probabilistic Language Models April 13, 2004.

Bag-of-Words Methods for Text Mining CSCI-GA.2590 – Lecture 2A Ralph Grishman NYU.

Lecture 3: Retrieval Evaluation Maya Ramanath. Benchmarking IR Systems Result Quality Data Collection – Ex: Archives of the NYTimes Query set – Provided.

User-Friendly Systems Instead of User-Friendly Front-Ends Present user interfaces are not accepted because the underlying systems are too difficult to.

Xiaoying Gao Computer Science Victoria University of Wellington COMP307 NLP 4 Information Retrieval.

Introduction to Information Retrieval Introduction to Information Retrieval Lecture Probabilistic Information Retrieval.

Introduction to Information Retrieval Probabilistic Information Retrieval Chapter 11 1.

3: Search & retrieval: Structures. The dog stopped attacking the cat, that lived in U.S.A. collection corpus database web d1…..d n docs processed term-doc.

Probabilistic Retrieval LBSC 708A/CMSC 838L Session 4, October 2, 2001 Philip Resnik.

Information Retrieval Models: Probabilistic Models

موضوع پروژه : بازیابی اطلاعات Information Retrieval

John Lafferty, Chengxiang Zhai School of Computer Science

Language Model Approach to IR

CS590I: Information Retrieval

Retrieval Utilities Relevance feedback Clustering

Information Retrieval and Web Design

Probabilistic Information Retrieval

VECTOR SPACE MODEL Its Applications and implementations

Presentation transcript:

Lecture 2: Retrieval Models Maya Ramanath

QQ1 Vector space model: 0 for non-presence of a term, 1 for presence: Query: q1 AND q2 AND q3 Compare the set of results returned by the vector space model and boolean model.

Term weighting (1/3): tf The lion (Panthera leo) is one of four big cats … Highly distinctive, the male lion is easily recognised by its mane… The lion (Panthera leo) is one of four big cats … Highly distinctive, the male lion is easily recognised by its mane… The cat (Felis catus), also known as the domestic cat … Cats are similar in anatomy to the other felids… The cat (Felis catus), also known as the domestic cat … Cats are similar in anatomy to the other felids… The New World monkeys are classified within the parvorder Platyrrhini, whereas the Old World monkeys form part of the parvorder Catarrhini, which also includes the hominoids… DOCCOUNT LionCatThe D1212 D2032 D3005 D1 D2 D3 Query: Cat Query: Lion Query: The Lion D2 D1 D3 D1 D2 D3 D1 D2 D3

Term weighting (2/3): tf.idf The lion (Panthera leo) is one of four big cats … Highly distinctive, the male lion is easily recognised by its mane… The lion (Panthera leo) is one of four big cats … Highly distinctive, the male lion is easily recognised by its mane… The cat (Felis catus), also known as the domestic cat … Cats are similar in anatomy to the other felids… The cat (Felis catus), also known as the domestic cat … Cats are similar in anatomy to the other felids… The New World monkeys are classified within the parvorder Platyrrhini, whereas the Old World monkeys form part of the parvorder Catarrhini, which also includes the hominoids… DOCCOUNT LionCatThe D1212 D2032 D3005 D1 D2 D3 Query: Cat Query: Lion Query: The Lion D2D2 D2D2 D1D1 D1D1 D3D3 D3D3 D1D1 D1D1 D2D2 D2D2 D3D3 D3D3 D1D1 D1D1 D2D2 D2D2 D3D3 D3D3 DOCWEIGHT LionCatThe D12/11/22/3 D203/22/3 D3005/3

Term weighting (3/3): doc length Shorter the text, more important the match Longer the text, more likely you “accidentally” fine a match “Let me tell you about the cat, a domestic animal” “Let me tell you about all the animals in the whole world (including the cat) !”

PROBABILISTIC RANKING

Binary Independence Model Rank documents in decreasing probability of relevance Derivation is long, but not difficult!

Let if

We still need relevant/irrelevant documents Sample of corpus, exhaustively judged Relevance feedback Pseudo-relevance feedback 2-poisson model …

LANGUAGE MODELS

Intuition (1/2) Document D Document Q This is the observation Can we figure out the source?

Intuition (2/2) PDPD PDPD PQPQ PQPQ Document DDocument Q These are the observations Can we estimate P D and P Q ?

Query as a sample Estimated using MLE

References For term weighting and the long derivation – Introduction to Information Retrieval: Raghavan, Manning and Shuetze, Cambridge University Press, Also available from: book/html/htmledition/irbook.htmlhttp://nlp.stanford.edu/IR- book/html/htmledition/irbook.html Language Models – Statistical Language Models for Information Retrieval: A Critical Review. ChengXiang Zhai, Foundations and Trends in IR 2(3), 2008 – Also available in the IR book above

QUESTIONS ?