1. Recuperação de Informação B Modern Information Retrieval Cap. 2: Modeling Section 2.8 : Alternative Probabilistic Models September 20, 1999

2. Alternative Probabilistic Models n Probability Theory u Semantically clear u Computationally clumsy n Why Bayesian Networks? u Clear formalism to combine evidences u Modularize the world (dependencies) u Bayesian Network Models for IR F Inference Network (Turtle & Croft, 1991) F Belief Network (Ribeiro-Neto & Muntz, 1996)

3. Bayesian Inference Schools of thought in probability u freqüentist u epistemological

4. Bayesian Inference Basic Axioms: F 0 < P(A) < 1 ; F P(sure)=1; F P(A V B)=P(A)+P(B) if A and B are mutually exclusive

5. Bayesian Inference Other formulations F P(A)=P(A  B)+P(A  ¬B) F P(A)=   i P(A  B i ), where B i,  i is a set of exhaustive and mutually exclusive events F P(A) + P(¬A) = 1 F P(A|K) belief in A given the knowledge K F if P(A|B)=P(A), we say:A and B are independent F if P(A|B  C)= P(A|C), we say: A and B are conditionally independent, given C F P(A  B)=P(A|B)P(B) F P(A)=   i P(A | B i )P(B i )

6. Bayesian Inference Bayes’ Rule : the heart of Bayesian techniques P(H|e) = P(e|H)P(H) / P(e) Where, H : a hypothesis and e is an evidence P(H) : prior probability P(H|e) : posterior probability P(e|H) : probability of e if H is true P(e) : a normalizing constant, then we write: P(H|e) ~ P(e|H)P(H)

7. Bayesian Networks Definition: Bayesian networks are directed acyclic graphs (DAGS) in which the nodes represent random variables, the arcs portray causal relationships between these variables, and the strengths of these causal influences are expressed by conditional probabilities.

8. Bayesian Networks y i : parent nodes (in this case, root nodes) x : child node y i cause x Y the set of parents of x The influence of Y on x can be quantified by any function F(x,Y) such that   x F(x,Y) = 1 0 < F(x,Y) < 1 For example, F(x,Y)=P(x|Y)

9. Bayesian Networks Given the dependencies declared in a Bayesian Network, the expression for the joint probability can be computed as a product of local conditional probabilities, for example, P(x 1, x 2, x 3, x 4, x 5 )= P(x 1 ) P(x 2 | x 1 ) P(x 3 | x 1 ) P(x 4 | x 2, x 3 ) P(x 5 | x 3 ). P(x 1 ) : prior probability of the root node

10. Bayesian Networks In a Bayesian network each variable x is conditionally independent of all its non-descendants, given its parents. For example: P(x 4, x 5 | x 2, x 3 )= P(x 4 | x 2, x 3 ) P( x 5 | x 3 )

11. Inference Network Model n Epistemological view of the IR problem n Random variables associated with documents, index terms and queries n A random variable associated with a document d j represents the event of observing that document

12. Inference Network Model Nodes documents (d j ) index terms (k i ) queries (q, q 1, and q 2 ) user information need (I) Edges from d j to its index term nodes k i indicate that the observation of d j increase the belief in the variables k i.

13. Inference Network Model d j has index terms k 2, k i, and k t q has index terms k 1, k 2, and k i q 1 and q 2 model boolean formulation q 1 =((k 1  k 2 ) v k i ); I = (q v q 1 )

14. Inference Network Model Definitions: k 1, d j,, and q random variables. k=(k 1, k 2,...,k t ) a t-dimensional vector k i,  i  {0, 1}, then k has 2 t possible states d j,  j  {0, 1};  q  {0, 1} The rank of a document d j is computed as P(q  d j ) q and d j, are short representations for q=1 and d j =1 (d j stands for a state where d j = 1 and  l  j  d l =0, because we observe one document at a time)

15. Inference Network Model P(q  d j )=   k P(q  d j | k) P(k) =   k P(q  d j  k) =   k P(q | d j  k) P(d j  k) =   k P(q | k) P(k | d j ) P( d j ) P(¬(q  d j )) = 1 - P(q  d j )

16. Inference Network Model As the instantiation of d j makes all index term nodes mutually independent P(k | d j ) can be a product,then P(q  d j )=   k [ P(q | k) (   i|g i (k)=1 P(k i | d j ) ) (   i|g i (k)=0 P(¬k i | d j ) ) P( d j ) ] remember that: g i (k)= 1 if k i =1 in the vector k 0 otherwise

17. Inference Network Model The prior probability P(d j ) reflects the probability associated to the event of observing a given document d j F Uniformly for N documents P(d j ) = 1/N P(¬d j ) = 1 - 1/N F Based on norm of the vector d j P(d j )= 1/|d j | P(¬d j ) = 1 - 1/|d j |

18. Inference Network Model For the Boolean Model P(d j ) = 1/N 1 if g i (d j )=1 P(k i | d j ) = 0 otherwise P(¬k i | d j ) = 1 - P(k i | d j )  only nodes associated with the index terms of the document d j are activated

19. Inference Network Model For the Boolean Model 1 if  q cc | (q cc  q dnf )  (  k i, g i (k)= g i (q cc ) P(q | k) = 0 otherwise P(¬q | k) = 1 - P(q | k)  one of the conjunctive components of the query must be matched by the active index terms in k

20. Inference Network Model For a tf-idf ranking strategy P(d j )= 1 / |d j | P(¬d j ) = 1 - 1 / |d j |  prior probability reflects the importance of document normalization

21. Inference Network Model For a tf-idf ranking strategy P(k i | d j ) = f i,j P(¬k i | d j )= 1- f i,j  the relevance of the a index term k i is determined by its normalized term-frequency factor f i,j = freq i,j / max freq l,j

22. Inference Network Model For a tf-idf ranking strategy Define a vector k i given by k i = k | ((g i (k)=1)  (  j  i g j (k)=0))  in the state k i only the node k i is active and all the others are inactive

23. Inference Network Model For a tf-idf ranking strategy idf i if k = k i  g i (q)=1 P(q | k) = 0 if k  k i v g i (q)=0 P(¬q | k) = 1 - P(q | k)  we can sum up the individual contributions of each index term by its normalized idf

24. Inference Network Model For a tf-idf ranking strategy As P(q|k)=0  k  k i, we can rewrite P(q  d j ) as P(q  d j ) =   ki [ P(q | k i ) P(k i | d j ) (   l|l  i P(¬k l | d j ) ) P( d j ) ] = (   i P(¬k l | d j ) ) P( d j )   ki [ P(k i | d j ) P(q | k i ) / P(¬k i | d j ) ]

25. Inference Network Model For a tf-idf ranking strategy Applying the previous probabilities we have P(q  d j ) = C j (1/|d j |)   i [ f i,j idf i (1/(1- f i,j )) ]  C j vary from document to document  the ranking is distinct of the one provided by the vector model

26. Inference Network Model Combining evidential source Let I = q v q 1 P(I  d j )=   k P(I | k) P(k | d j ) P( d j ) =   k [1 - P(¬q|k)P(¬q 1 | k)] P(k| d j ) P( d j )  it might yield a retrieval performance which surpasses the retrieval performance of the query nodes in isolation (Turtle & Croft)

27. Belief Network Model n As the Inference Network Model u Epistemological view of the IR problem u Random variables associated with documents, index terms and queries n Contrary to the Inference Network Model u Clearly defined sample space u Set-theoretic view u Different network topology

28. Belief Network Model The Probability Space Define: K={k 1, k 2,...,k t } the sample space (a concept space) u  K a subset of K (a concept) k i an index term (an elementary concept) k=(k 1, k 2,...,k t ) a vector associated to each u such that g i (k)=1  k i  u k i a binary random variable associated with the index term k i, (k i = 1  g i (k)=1  k i  u)

29. Belief Network Model A Set-Theoretic View Define: a document d j and query q as concepts in K a generic concept c in K a probability distribution P over K, as P(c)=   u P(c|u) P(u) P(u)=(1/2) t P(c) is the degree of coverage of the space K by c

30. Belief Network Model Network topology query side document side

31. Belief Network Model Assumption P(d j |q) is adopted as the rank of the document d j with respect to the query q. It reflects the degree of coverage provided to the concept d j by the concept q.

32. Belief Network Model The rank of d j P(d j |q) = P(d j  q) / P(q) ~ P(d j  q) ~   u P(d j  q | u) P(u) ~   u P(d j | u) P(q | u) P(u) ~   k P(d j | k) P(q | k) P(k)

33. Belief Network Model For the vector model Define Define a vector k i given by k i = k | ((g i (k)=1)  (  j  i g j (k)=0))  in the state k i only the node k i is active and all the others are inactive

34. Belief Network Model For the vector model Define (w i,q / |q|) if k = k i  g i (q)=1 P(q | k) = 0 if k  k i v g i (q)=0 P(¬q | k) = 1 - P(q | k)  (w i,q / |q|) is a normalized version of weight of the index term k i in the query q

35. Belief Network Model For the vector model Define (w i,j / |d j |) if k = k i  g i (d j )=1 P(d j | k) = 0 if k  k i v g i (d j )=0 P(¬ d j | k) = 1 - P(d j | k)  (w i,j / |d j |) is a normalized version of the weight of the index term k i in the document d,j

36. Bayesian Network Models Comparison F Inference Network Model is the first and well known F Belief Network adopts a set-theoretic view F Belief Network adopts a clearly define sample space F Belief Network provides a separation between query and document portions F Belief Network is able to reproduce any ranking produced by the Inference Network while the converse is not true (for example: the ranking of the standard vector model)

37. Bayesian Network Models Computational costs F Inference Network Model one document node at a time then is linear on number of documents F Belief Network only the states that activate each query term are considered F The networks do not impose additional costs because the networks do not include cycles.

38. Bayesian Network Models Impact The major strength is net combination of distinct evidential sources to support the rank of a given document.