1. Information Retrieval CSE 8337 Spring 2005 Modeling Material for these slides obtained from: Modern Information Retrieval by Ricardo Baeza-Yates and Berthier Ribeiro-Neto http://www.sims.berkeley.edu/~hearst/irbook/ http://www.sims.berkeley.edu/~hearst/irbook/ Introduction to Modern Information Retrieval by Gerard Salton and Michael J. McGill, McGraw-Hill, 1983.

2. CSE 8337 Spring 2005 2 Modeling TOC Introduction Classic IR Models Boolean Model Vector Model Probabilistic Model Set Theoretic Models Fuzzy Set Model Extended Boolean Model Generalized Vector Model Latent Semantic Indexing Neural Network Model Alternative Probabilistic Models Inference Network Belief Network

3. CSE 8337 Spring 2005 3 Introduction IR systems usually adopt index terms to process queries Index term: a keyword or group of selected words any word (more general) Stemming might be used: connect: connecting, connection, connections An inverted file is built for the chosen index terms

4. CSE 8337 Spring 2005 4 Introduction Docs Information Need Index Terms doc query Ranking match

5. CSE 8337 Spring 2005 5 Introduction Matching at index term level is quite imprecise No surprise that users get frequently unsatisfied Since most users have no training in query formation, problem is even worst Frequent dissatisfaction of Web users Issue of deciding relevance is critical for IR systems: ranking

6. CSE 8337 Spring 2005 6 Introduction A ranking is an ordering of the documents retrieved that (hopefully) reflects the relevance of the documents to the query A ranking is based on fundamental premisses regarding the notion of relevance, such as: common sets of index terms sharing of weighted terms likelihood of relevance Each set of premisses leads to a distinct IR model

7. CSE 8337 Spring 2005 7 IR Models Non-Overlapping Lists Proximal Nodes Structured Models Retrieval: Adhoc Filtering Browsing U s e r T a s k Classic Models boolean vector probabilistic Set Theoretic Fuzzy Extended Boolean Probabilistic Inference Network Belief Network Algebraic Generalized Vector Lat. Semantic Index Neural Networks Browsing Flat Structure Guided Hypertext

8. CSE 8337 Spring 2005 8 IR Models

9. CSE 8337 Spring 2005 9 Classic IR Models - Basic Concepts Each document represented by a set of representative keywords or index terms An index term is a document word useful for remembering the document main themes Usually, index terms are nouns because nouns have meaning by themselves However, search engines assume that all words are index terms (full text representation)

10. CSE 8337 Spring 2005 10 Classic IR Models - Basic Concepts The importance of the index terms is represented by weights associated to them k i - an index term d j - a document w ij - a weight associated with (k i,d j ) The weight w ij quantifies the importance of the index term for describing the document contents

11. CSE 8337 Spring 2005 11 Classic IR Models - Basic Concepts t is the total number of index terms K = {k 1, k 2, …, k t } is the set of all index terms w ij >= 0 is a weight associated with (k i,d j ) w ij = 0 indicates that term does not belong to doc d j = (w 1j, w 2j, …, w tj ) is a weighted vector associated with the document d j g i (d j ) = w ij is a function which returns the weight associated with pair (k i,d j )

12. CSE 8337 Spring 2005 12 The Boolean Model Simple model based on set theory Queries specified as boolean expressions precise semantics and neat formalism Terms are either present or absent. Thus, w ij  {0,1} Consider q = k a  (k b   k c ) q dnf = (1,1,1)  (1,1,0)  (1,0,0) q cc = (1,1,0) is a conjunctive component

13. CSE 8337 Spring 2005 13 The Boolean Model q = k a  (k b   k c ) sim(q,d j ) = 1 if  q cc | (q cc  q dnf )  (  k i, g i (d j )= g i (q cc )) 0 otherwise (1,1,1) (1,0,0) (1,1,0) KaKa KbKb KcKc

14. CSE 8337 Spring 2005 14 Drawbacks of the Boolean Model Retrieval based on binary decision criteria with no notion of partial matching No ranking of the documents is provided Information need has to be translated into a Boolean expression The Boolean queries formulated by the users are most often too simplistic As a consequence, the Boolean model frequently returns either too few or too many documents in response to a user query

15. CSE 8337 Spring 2005 15 The Vector Model Use of binary weights is too limiting Non-binary weights provide consideration for partial matches These term weights are used to compute a degree of similarity between a query and each document Ranked set of documents provides for better matching

16. CSE 8337 Spring 2005 16 The Vector Model w ij > 0 whenever k i appears in d j w iq >= 0 associated with the pair (k i,q) d j = (w 1j, w 2j,..., w tj ) q = (w 1q, w 2q,..., w tq ) To each term k i is associated a unitary vector i The unitary vectors i and j are assumed to be orthonormal (i.e., index terms are assumed to occur independently within the documents) The t unitary vectors i form an orthonormal basis for a t-dimensional space where queries and documents are represented as weighted vectors

17. CSE 8337 Spring 2005 17 The Vector Model Sim(q,d j ) = cos(  ) = [d j  q] / |d j | * |q| = [  w ij * w iq ] / |d j | * |q| Since w ij > 0 and w iq > 0, 0 <= sim(q,d j ) <=1 A document is retrieved even if it matches the query terms only partially i j dj q 

18. CSE 8337 Spring 2005 18 Weights w ij and w iq ? One approach is to examine the frequency of the occurence of a word in a document: Absolute frequency: tf factor, the term frequency within a document freq i,j - raw frequency of k i within d j Both high-frequency and low-frequency terms may not actually be significant Relative frequency: tf divided by number of words in document Normalized frequency: f i,j = (freq i,j )/(max l freq l,j )

19. CSE 8337 Spring 2005 19 Inverse Document Frequency Importance of term may depend more on how it can distinguish between documents. Quantification of inter-documents separation Dissimilarity not similarity idf factor, the inverse document frequency

20. CSE 8337 Spring 2005 20 IDF N be the total number of docs in the collection n i be the number of docs which contain k i The idf factor is computed as idf i = log (N/n i ) the log is used to make the values of tf and idf comparable. It can also be interpreted as the amount of information associated with the term k i. IDF Ex: N=1000, n 1 =100, n 2 =500, n 3 =800 idf 1 = 3 - 2 = 1 idf 2 = 3 – 2.7 = 0.3 idf 3 = 3 – 2.9 = 0.1

21. CSE 8337 Spring 2005 21 The Vector Model The best term-weighting schemes take both into account. w ij = f i,j * log(N/n i ) This strategy is called a tf-idf weighting scheme

22. CSE 8337 Spring 2005 22 The Vector Model For the query term weights, a suggestion is w iq = (0.5 + [0.5 * freq i,q / max(freq l,q ]) * log(N/n i ) The vector model with tf-idf weights is a good ranking strategy with general collections The vector model is usually as good as any known ranking alternatives. It is also simple and fast to compute.

23. CSE 8337 Spring 2005 23 The Vector Model Advantages: term-weighting improves quality of the answer set partial matching allows retrieval of docs that approximate the query conditions cosine ranking formula sorts documents according to degree of similarity to the query Disadvantages: Assumes independence of index terms (??); not clear that this is bad though

24. CSE 8337 Spring 2005 24 The Vector Model: Example I d1 d2 d3 d4d5 d6 d7 k1 k2 k3

25. CSE 8337 Spring 2005 25 The Vector Model: Example II d1 d2 d3 d4d5 d6 d7 k1 k2 k3

26. CSE 8337 Spring 2005 26 The Vector Model: Example III d1 d2 d3 d4d5 d6 d7 k1 k2 k3

27. CSE 8337 Spring 2005 27 Probabilistic Model Objective: to capture the IR problem using a probabilistic framework Given a user query, there is an ideal answer set Querying as specification of the properties of this ideal answer set (clustering) But, what are these properties? Guess at the beginning what they could be (i.e., guess initial description of ideal answer set) Improve by iteration

28. CSE 8337 Spring 2005 28 Probabilistic Model An initial set of documents is retrieved somehow User inspects these docs looking for the relevant ones (in truth, only top 10-20 need to be inspected) IR system uses this information to refine description of ideal answer set By repeting this process, it is expected that the description of the ideal answer set will improve Have always in mind the need to guess at the very beginning the description of the ideal answer set Description of ideal answer set is modeled in probabilistic terms

29. CSE 8337 Spring 2005 29 Probabilistic Ranking Principle Given a user query q and a document d j, the probabilistic model tries to estimate the probability that the user will find the document d j interesting (i.e., relevant). Ideal answer set is referred to as R and should maximize the probability of relevance. Documents in the set R are predicted to be relevant. But, how to compute probabilities? what is the sample space?

30. CSE 8337 Spring 2005 30 The Ranking Probabilistic ranking computed as: sim(q,d j ) = P(d j relevant-to q) / P(d j non-relevant- to q) This is the odds of the document d j being relevant Taking the odds minimize the probability of an erroneous judgement Definition: w ij  {0,1} P(R | d j ) : probability that given doc is relevant P(  R | d j ) : probability doc is not relevant

31. CSE 8337 Spring 2005 31 The Ranking sim(d j,q) = P(R | d j ) / P(  R | d j ) = [P(d j | R) * P(R)] [P(d j |  R) * P(  R)] ~ P(d j | R) P(d j |  R) P(d j | R) : probability of randomly selecting the document dj from the set R of relevant documents

32. CSE 8337 Spring 2005 32 The Ranking sim(d j,q)~ P(d j | R) P(d j |  R) ~ [  P(k i | R)] * [  P(  k i | R)] [  P(k i |  R)] * [  P(  k i |  R)] P(k i | R) : probability that the index term k i is present in a document randomly selected from the set R of relevant documents

33. CSE 8337 Spring 2005 33 The Ranking sim(d j,q) ~ log [  P(k i | R)] * [  P(  k j | R)] [  P(k i |  R)] * [  P(  k i |  R)] ~ K * [ log  P(k i | R) + log  P(k i |  R) ] P(  k i | R) P(  k i |  R) where P(  k i | R) = 1 - P(k i | R) P(  k i |  R) = 1 - P(k i |  R)

34. CSE 8337 Spring 2005 34 The Initial Ranking sim(d j,q) ~  w iq * w ij * (log P(k i | R) + log P(k i |  R) ) P(  k i | R) P(  k i |  R) Probabilities P(k i | R) and P(k i |  R) ? Estimates based on assumptions: P(k i | R) = 0.5 P(k i |  R) = n i N Use this initial guess to retrieve an initial ranking Improve upon this initial ranking

35. CSE 8337 Spring 2005 35 Improving the Initial Ranking Let V : set of docs initially retrieved V i : subset of docs retrieved that contain k i Reevaluate estimates: P(k i | R) = V i V P(k i |  R) = n i - V i N - V Repeat recursively

36. CSE 8337 Spring 2005 36 Improving the Initial Ranking To avoid problems with V=1 and Vi=0: P(k i | R) = V i + 0.5 V + 1 P(k i |  R) = n i - V i + 0.5 N - V + 1 Also, P(k i | R) = V i + n i /N V + 1 P(k i |  R) = n i - V i + n i /N N - V + 1

37. CSE 8337 Spring 2005 37 Pluses and Minuses Advantages: Docs ranked in decreasing order of probability of relevance Disadvantages: need to guess initial estimates for P(k i | R) method does not take into account tf and idf factors

38. CSE 8337 Spring 2005 38 Brief Comparison of Classic Models Boolean model does not provide for partial matches and is considered to be the weakest classic model Salton and Buckley did a series of experiments that indicate that, in general, the vector model outperforms the probabilistic model with general collections This seems also to be the view of the research community

39. CSE 8337 Spring 2005 39 Set Theoretic Models The Boolean model imposes a binary criterion for deciding relevance The question of how to extend the Boolean model to accomodate partial matching and a ranking has attracted considerable attention in the past We discuss now two set theoretic models for this: Fuzzy Set Model Extended Boolean Model

40. CSE 8337 Spring 2005 40 Fuzzy Set Model This vagueness of document/query matching can be modeled using a fuzzy framework, as follows: with each term is associated a fuzzy set each doc has a degree of membership in this fuzzy set Here, we discuss the model proposed by Ogawa, Morita, and Kobayashi (1991)

41. CSE 8337 Spring 2005 41 Fuzzy Set Theory A fuzzy subset A of U is characterized by a membership function  (A,u) : U  [0,1] which associates with each element u of U a number  (u) in the interval [0,1] Definition Let A and B be two fuzzy subsets of U. Also, let ¬A be the complement of A. Then,  (¬A,u) = 1 -  (A,u)  (A  B,u) = max(  (A,u),  (B,u))  (A  B,u) = min(  (A,u),  (B,u))

42. CSE 8337 Spring 2005 42 Fuzzy Information Retrieval Fuzzy sets are modeled based on a thesaurus This thesaurus is built as follows: Let c be a term-term correlation matrix Let c i,l be a normalized correlation factor for (k i,k l ): c i,l = n i,l n i + n l - n i,l n i : number of docs which contain k i n l : number of docs which contain k l n i,l : number of docs which contain both k i and k l We now have the notion of proximity among index terms.

43. CSE 8337 Spring 2005 43 Fuzzy Information Retrieval The correlation factor c i,l can be used to define fuzzy set membership for a document dj as follows:  i,j = 1 -  (1 - c i,l ) k i  d j  i,j : membership of doc d j in fuzzy subset associated with k i The above expression computes an algebraic sum over all terms in the doc d j

44. CSE 8337 Spring 2005 44 Fuzzy Information Retrieval A doc d j belongs to the fuzzy set for k i, if its own terms are associated with k i If doc d j contains a term k l which is closely related to k i, we have c i,l ~ 1  i,j ~ 1 index k i is a good fuzzy index for doc

45. CSE 8337 Spring 2005 45 Fuzzy IR: An Example q = k a  (k b   k c ) q dnf = (1,1,1) + (1,1,0) + (1,0,0) = cc 1 + cc 2 + cc 3  q,dj =  cc1+cc2+cc3,j = 1 - (1 -  a,j  b,j)  c,j ) * (1 -  a,j  b,j (1-  c,j )) * (1 -  a,j (1-  b,j ) (1-  c,j )) cc1 cc3 cc2 KaKa KbKb KcKc

46. CSE 8337 Spring 2005 46 Fuzzy Information Retrieval Fuzzy IR models have been discussed mainly in the literature associated with fuzzy theory Experiments with standard test collections are not available Difficult to compare at this time

47. CSE 8337 Spring 2005 47 Extended Boolean Model Boolean model is simple and elegant. But, no provision for a ranking As with the fuzzy model, a ranking can be obtained by relaxing the condition on set membership Extend the Boolean model with the notions of partial matching and term weighting Combine characteristics of the Vector model with properties of Boolean algebra

48. CSE 8337 Spring 2005 48 The Idea The Extended Boolean Model (introduced by Salton, Fox, and Wu, 1983) is based on a critique of a basic assumption in Boolean algebra Let, q = k x  k y w xj = f xj * idf x associated with [k x,d j ] max(idf i ) Further, w xj = x and w yj = y

49. CSE 8337 Spring 2005 49 The Idea: q and = k x  k y ; w xj = x and w yj = y djdj d j +1 y = w yj x = w xj (0,0) (1,1) kxkx kyky sim(q and,dj) = 1 - sqrt( (1-x) + (1-y) ) 2 22 AND

50. CSE 8337 Spring 2005 50 The Idea: q or = k x  k y ; w xj = x and w yj = y (1,1) sim(q or,dj) = sqrt( x + y ) 2 22 djdj d j +1 y = w yj x = w xj (0,0)kxkx kyky OR

51. CSE 8337 Spring 2005 51 Generalizing the Idea We can extend the previous model to consider Euclidean distances in a t- dimensional space This can be done using p-norms which extend the notion of distance to include p-distances, where 1  p   is a new parameter

52. CSE 8337 Spring 2005 52 Generalizing the Idea A generalized disjunctive query is given by q or = k 1 k 2... k t A generalized conjunctive query is given by q and = k 1 k 2... k t p  p  p   p  p  p sim(q or,d j ) = (x 1 + x 2 +... + x m ) m p pp p 1 sim(q and,d j )=1 - ((1-x 1 ) + (1-x 2 ) +... + (1-x m ) ) m p 1 p p p

53. CSE 8337 Spring 2005 53 Properties If p = 1 then (Vector like) sim(q or,d j ) = sim(q and,d j ) = x 1 +... + x m m If p =  then (Fuzzy like) sim(q or,d j ) = max (w xj ) sim(q and,d j ) = min (w xj ) By varying p, we can make the model behave as a vector, as a fuzzy, or as an intermediary model

54. CSE 8337 Spring 2005 54 Properties This is quite powerful and is a good argument in favor of the extended Boolean model q = (k 1 k 2 ) k 3 k 1 and k 2 are to be used as in a vector retrieval while the presence of k 3 is required. sim(q,d j ) = ( (1 - ( (1-x 1 ) + (1-x 2 ) ) ) + x 3 ) 2 ______ 2    2

55. CSE 8337 Spring 2005 55 Conclusions Model is quite powerful Properties are interesting and might be useful Computation is somewhat complex However, distributivity operation does not hold for ranking computation: q1 = (k1  k2)  k3 q2 = (k1  k3)  (k2  k3) sim(q1,dj)  sim(q2,dj)