Language Models LBSC 796/CMSC 828o Session 4, February 16, 2004 Douglas W. Oard

Questions The meaning of “maybe” Probabilistic retrieval Comparison with vector space model Agenda

Muddiest Points Why distinguish utility and relevance? How coordination measure is ranked Boolean Why use term weights? The meaning of DF The problem with log(1)=0 How the vectors are built How to do cosine normalization (5) Why to do cosine normalization (2) Okapi graphs

Supporting the Search Process Source Selection Search Query Selection Ranked List Examination Document Delivery Document Query Formulation IR System Indexing Index Acquisition Collection     

Looking Ahead We ask “is this document relevant?” –Vector space: we answer “somewhat” –Probabilistic: we answer “probably” The key is to know what “probably” means –First, we’ll formalize that notion –Then we’ll apply it to retrieval

Probability What is probability? –Statistical: relative frequency as n   –Subjective: degree of belief Thinking statistically –Imagine a finite amount of “stuff” –Associate the number 1 with the total amount –Distribute that “mass” over the possible events

Statistical Independence A and B are independent if and only if: P(A and B) = P(A)  P(B) Independence formalizes “unrelated” –P(“being brown eyed”) = 85/100 –P(“being a doctor”) = 1/1000 –P(“being a brown eyed doctor”) = 85/100,000

Dependent Events Suppose” –P(“having a B.S. degree”) = 2/10 –P(“being a doctor”) = 1/1000 Would you expect –P(“having a B.S. degree and being a doctor”) = 2/10,000 ??? Extreme example: –P(“being a doctor”) = 1/1000 –P(“having studied anatomy”) = 12/1000

Conditional Probability P(A | B)  P(A and B) / P(B) A B A and B P(A) = prob of A relative to the whole space P(A|B) = prob of A considering only the cases where B is known to be true

More on Conditional Probability Suppose –P(“having studied anatomy”) = 12/1000 –P(“being a doctor and having studied anatomy”) = 1/1000 Consider –P(“being a doctor” | “having studied anatomy”) = 1/12 But if you assume all doctors have studied anatomy –P(“having studied anatomy” | “being a doctor”) = 1 Useful restatement of definition: P(A and B) = P(A|B) x P(B)

Some Notation Consider –A set of hypotheses: H1, H2, H3 –Some observable evidence O P(O|H1) = probability of O being observed if we knew H1 were true P(O|H2) = probability of O being observed if we knew H2 were true P(O|H3) = probability of O being observed if we knew H3 were true

An Example Let –O = “Joe earns more than $80,000/year” –H1 = “Joe is a doctor” –H2 = “Joe is a college professor” –H3 = “Joe works in food services” Suppose we do a survey and we find out –P(O|H1) = 0.6 –P(O|H2) = 0.07 –P(O|H3) = What should be our guess about Joe’s profession?

Bayes’ Rule What’s P(H1|O)? P(H2|O)? P(H3|O)? Theorem: P(H | O) = P(O | H) x P(H) P(O) Posterior probability Prior probability Notice that the prior is very important!

Back to the Example Suppose we also have good data about priors: –P(O|H1) = 0.6P(H1) = doctor –P(O|H2) = 0.07P(H2) = prof –P(O|H3) = 0.001P(H3) = 0.2 food We can calculate –P(H1|O) = / P(“earning > $70K/year”) –P(H2|O) = / P(“earning > $70K/year”) –P(H3|O) = / P(“earning > $70K/year”)

Key Ideas Defining probability using frequency Statistical independence Conditional probability Bayes’ rule

Questions Defining probability Using probability for retrieval –Language modeling –Inference networks Comparison with vector space model Agenda

Probability Ranking Principle Assume binary relevance/document independence –Each document is either relevant or it is not –Relevance of one doc reveals nothing about another Assume the searcher works down a ranked list –Seeking some number of relevant documents Theorem (provable from assumptions): –Documents should be ranked in order of decreasing probability of relevance to the query, P(d relevant-to q)

Probabilistic Retrieval Strategy Estimate how terms contribute to relevance –How do TF, DF, and length influence your judgments about document relevance? (e.g., Okapi) Combine to find document relevance probability Order documents by decreasing probability

Where do the probabilities fit? Comparison Function Representation Function Query Formulation Human Judgment Representation Function Retrieval Status Value Utility Query Information NeedDocument Query RepresentationDocument Representation Query Processing Document Processing P(d is Rel | q)

Binary Independence Model Binary refers again to binary relevance Assume “term independence” –Presence of one term tells nothing about another Assume “uniform priors” –P(d) is the same for all d

“Okapi” Term Weights TF componentIDF component

Stochastic Language Models Models probability of generating any string 0.2the 0.1a 0.01man 0.01woman 0.03said 0.02likes … themanlikesthewoman multiply Model M P(s | M)

Language Models, cont’d Models probability of generating any string 0.2the 0.1a 0.01man 0.01woman 0.03said 0.02likes … Model M1 0.2the 0.1yon 0.001class 0.01maiden 0.03sayst 0.02pleaseth … Model M2 maidenclasspleasethyonthe P(s|M2) > P(s|M1)

Retrieval with Language Models Treat each document as the basis for a model Rank document d based on P(d | q) –P(d | q) = P(q | d) x P(d) / P(q) P(q) is same for all documents, can’t change ranks P(d) [the prior] is often treated as the same for all d –But we could use criteria like authority, length, genre P(q | d) is the probability of q given d’s model –Same as ranking by P(q | d)

Computing P(q | d) Build a smoothed language model for d –Count the frequency of each term in d –Count the frequency of each term in the collection –Combine the two in some way –Redistribute probabilities to unobserved events Example: add 1 to every count Combine the probability for the full query –Summing over the terms in q is a soft “OR”

Key Ideas Probabilistic methods formalize assumptions –Binary relevance –Document independence –Term independence –Uniform priors –Top-down scan Natural framework for combining evidence –e.g., non-uniform priors

Inference Networks A flexible way of combining term weights –Boolean model –Binary independence model –Probabilistic models with weaker assumptions Key concept: rank based on P(d | q) –P(d | q) = P(q | d) x P(d) / P(q) Efficient large-scale implementation –InQuery text retrieval system from U Mass

A Boolean Inference Net bat d1d2d3d4 catfathatmatpatratvat ANDOR sat AND I Information need

A Binary Independence Network bat d1d2d3d4 catfathatmatpatratvatsat query

Probability Computation Turn on exactly one document at a time –Boolean: Every connected term turns on –Binary Ind: Connected terms gain their weight Compute the query value –Boolean: AND and OR nodes use truth tables –Binary Ind: Fraction of the possible weight

A Critique Most of the assumptions are not satisfied! –Searchers want utility, not relevance –Relevance is not binary –Terms are clearly not independent –Documents are often not independent The best known term weights are quite ad hoc –Unless some relevant documents are known

But It Works! Ranked retrieval paradigm is powerful –Well suited to human search strategies Probability theory has explanatory power –At least we know where the weak spots are –Probabilities are good for combining evidence Good implementations exist (InQuery, Lemur) –Effective, efficient, and large-scale

Comparison With Vector Space Similar in some ways –Term weights can be based on frequency –Terms often used as if they were independent Different in others –Based on probability rather than similarity –Intuitions are probabilistic rather than geometric

One Minute Paper Which assumption underlying the probabilistic retrieval model causes you the most concern, and why?