1. UIUC CS 497: Section EA Lecture #7 Reasoning in Artificial Intelligence Professor: Eyal Amir Spring Semester 2004 (Based on slides by Gal Elidan (Hebrew U))

2. Last Time Probabilistic graphical models Exact reasoning –Variable elimination –Junction tree algorithm Applications (of Bayes Networks): –Sensor networks, medical diagnosis, computer diagnosis (MS Windows), classification

3. Approximate Inference Large treewidth –Large, highly connected graphical models –Treewidth may be large (>40) in sparse networks In many applications, approximation are sufficient –Example: P(X = x|e) = 0.3183098861838 –Maybe P(X = x|e)  0.3 is a good enough approximation –e.g., we take action only if P(X = x|e) > 0.5

4. Today: Approximate reasoning via sampling 1.Monte Carlo techniques 1.Rejection sampling 2.Likelihood weighting 3.Importance sampling 2.Markov Chain Monte Carlo (MCMC) 1.Gibbs sampling 2.Metropolis-Hastings 3. Applications du jour: ?

5. Types of Approximations Absolute error An estimate q of P(X = x | e) has absolute error , if P(X = x|e) -   q  P(X = x|e) +  equivalently q -   P(X = x|e)  q +  Not always what we want: error 0.001 –Unacceptable if P(X = x | e) = 0.0001, –Overly precise if P(X = x | e) = 0.3 0 1 q 22

6. Types of Approximations Relative error An estimate q of P(X = x | e) has relative error , if P(X = x|e)(1-  )  q  P(X = x|e)(1+  ) equivalently q/(1+  )  P(X = x|e)  q/(1-  ) Sensitivity of approximation depends on actual value of desired result 0 1 q q/(1+  ) q/(1-  )

7. Complexity Recall, exact inference is NP-hard Is approximate inference any easier? Construction for exact inference: –Input: a 3-SAT problem  –Output: a BN such that P(X=t) > 0 iff  is satisfiable

8. Complexity: Relative Error Suppose that q is a relative error estimate of P(X = t), If  is not satisfiable, then P(X = t)(1 -  )  q  P(X = t)(1 +  )0 = P(X = t)(1 -  )  q  P(X = t)(1 +  ) = 0 Thus, if q > 0, then  is satisfiable An immediate consequence: Thm: Given , finding an  -relative error approximation is NP- hard

9. Complexity: Absolute error Thm: If  < 0.5, then finding an estimate of P(X=x|e) with  absulote error approximation is NP-Hard

10. Search Algorithms Idea: search for high probability instances Suppose x[1], …, x[N] are instances with high mass We can approximate: If x[i] is a complete instantiation, then P(e|x[i]) is 0 or 1

11. Search Algorithms (cont) Instances that do not satisfy e, do not play a role in approximation We need to focus the search to find instances that do satisfy e Clearly, in some cases this is hard (e.g., the construction from our NP-hardness result

12. Stochastic Simulation Suppose we can sample instances according to P (X 1,…,X n ) What is the probability that a random sample satisfies e? –This is exactly P(e) We can view each sample as tossing a biased coin with probability P(e) of “Heads”

13. Stochastic Sampling Intuition: given a sufficient number of samples x[1],…,x[N], we can estimate Law of large number implies that as N grows, our estimate will converge to p whp How many samples do we need to get a reliable estimation? Use Chernof’s bound for binomial distributions

14. Sampling a Bayesian Network If P (X 1,…,X n ) is represented by a Bayesian network, can we efficiently sample from it? Idea: sample according to structure of the network –Write distribution using the chain rule, and then sample each variable given its parents

15. Samples: B E A C R Logic sampling P(b) 0.03 P(e) 0.001 P(a) b e 0.98 0.4 0.7 0.01 P(c) a a 0.8 0.05 P(r) e e 0.30.001 b Earthquake Radio Burglary Alarm Call 0.03

16. Samples: B E A C R Logic sampling P(b) 0.03 P(e) 0.001 P(a) b e 0.98 0.4 0.7 0.01 P(c) a a 0.8 0.05 P(r) e e 0.30.001 eb Earthquake Radio Burglary Alarm Call 0.001

17. Samples: B E A C R Logic sampling P(b) 0.03 P(e) 0.001 P(a) b e 0.98 0.4 0.7 0.01 P(c) a a 0.8 0.05 P(r) e e 0.30.001 eab 0.4 Earthquake Radio Burglary Alarm Call

18. Samples: B E A C R Logic sampling P(b) 0.03 P(e) 0.001 P(a) b e 0.98 0.4 0.7 0.01 P(c) a a 0.8 0.05 P(r) e e 0.30.001 eacb Earthquake Radio Burglary Alarm Call 0.8

19. Samples: B E A C R Logic sampling P(b) 0.03 P(e) 0.001 P(a) b e 0.98 0.4 0.7 0.01 P(c) a a 0.8 0.05 P(r) e e 0.30.001 eacb r 0.3 Earthquake Radio Burglary Alarm Call

20. Logic Sampling Let X 1, …, X n be order of variables consistent with arc direction for i = 1, …, n do –sample x i from P(X i | pa i ) –(Note: since Pa i  {X 1,…,X i-1 }, we already assigned values to them) return x 1, …,x n

21. Logic Sampling Sampling a complete instance is linear in number of variables –Regardless of structure of the network However, if P(e) is small, we need many samples to get a decent estimate

22. Can sample from P( X 1,…,X n |e)? If evidence is in roots of network, easily If evidence is in leaves of network, we have a problem –Our sampling method proceeds according to order of nodes in graph Note, we can use arc-reversal to make evidence nodes root. –In some networks, however, this will create exponentially large tables...

23. Likelihood Weighting Can we ensure that all of our sample satisfy e? One simple solution: –When we need to sample a variable that is assigned value by e, use the specified value For example: we know Y = 1 –Sample X from P(X) –Then take Y = 1 Is this a sample from P( X,Y |Y = 1) ? X Y

24. Likelihood Weighting Problem: these samples of X from P(X) Solution: –Penalize samples in which P(Y=1|X) is small We now sample as follows: –Let x[i] be a sample from P(X) –Let w[i] be P(Y = 1|X = x [i]) X Y

25. Likelihood Weighting Why does this make sense? When N is large, we expect to sample NP(X = x) samples with x[i] = x Thus, When we normalize, we get approximation of the conditional probability

26. Samples: B E A C R Likelihood Weighting P(b) 0.03 P(e) 0.001 P(a) b e 0.98 0.4 0.7 0.01 P(c) a 0.8 0.05 P(r) e e 0.30.001 b Earthquake Radio Burglary Alarm Call 0.03 Weight = r a = a

27. Samples: B E A C R Likelihood Weighting P(b) 0.03 P(e) 0.001 P(a) b e 0.98 0.4 0.7 0.01 P(c) a a 0.8 0.05 P(r) e e 0.30.001 eb Earthquake Radio Burglary Alarm Call 0.001 Weight = r = a

28. Samples: B E A C R Likelihood Weighting P(b) 0.03 P(e) 0.001 P(a) b e 0.98 0.4 0.7 0.01 P(c) a a 0.8 0.05 P(r) e e 0.30.001 eb 0.4 Earthquake Radio Burglary Alarm Call Weight = r = a 0.6 a

29. Samples: B E A C R Likelihood Weighting P(b) 0.03 P(e) 0.001 P(a) b e 0.98 0.4 0.7 0.01 P(c) a a 0.8 0.05 P(r) e e 0.30.001 ecb Earthquake Radio Burglary Alarm Call 0.05 Weight = r = a a 0.6

30. Samples: B E A C R Likelihood Weighting P(b) 0.03 P(e) 0.001 P(a) b e 0.98 0.4 0.7 0.01 P(c) a a 0.8 0.05 P(r) e e 0.30.001 ecb r 0.3 Earthquake Radio Burglary Alarm Call Weight = r = a a 0.6 *0.3

31. Likelihood Weighting Let X 1, …, X n be order of variables consistent with arc direction w = 1 for i = 1, …, n do –if X i = x i has been observed w  w* P(X i = x i | pa i ) –else sample x i from P(X i | pa i ) return x 1, …,x n, and w

32. Importance Sampling A method for evaluating expectation of f under P(x), P(X) Discrete: Continuous: If we could sample from P

33. Importance Sampling A general method for evaluating P(X) when we cannot sample from P(X). Idea: Choose an approximating distribution Q(X) and sample from it Using this we can now sample from Q and then W(X) If we could generate samples from P(X) Now that we generate the samples from Q(X)

34. (Unnormalized) Importance Sampling 1. For m=1:M Sample X[m] from Q(X) Calculate W(m) = P(X)/Q(X) 2. Estimate the expectation of f(X) using Requirements: P(X)>0  Q(X)>0 (don’t ignore possible scenarios) Possible to calculate P(X),Q(X) for a specific X=x It is possible to sample from Q(X)

35. Normalized Importance Sampling Assume that we cannot evalute P(X=x) but can evaluate P’(X=x) =  P(X=x) (for example we can evaluate P(X) but not P(X|e) in a Bayesian network) We define w’(X) = P’(X)/Q(X). We can then evaluate  : and then: In the last step we simply replace  with the above equation

36. Normalized Importance Sampling We can now estimate the expectation of f(X) similarly to unnormalized importance sampling by sampling x[m] from Q(X) and then (hence the name “normalized”)

37. Importance Sampling Weaknesses Important to choose sampling distribution with heavy tails –Not to “miss” large values of f Many-dimensional I-S: –“Typical set” of P may take a long time to find, unless Q good approximation to P –Weights vary by factors exponential in N Similar for Likelihood Weighting

38. Today: Approximate Reasoning via Sampling 1.Monte Carlo techniques 1.Rejection sampling 2.Likelihood weighting 3.Importance sampling 2.Markov Chain Monte Carlo (MCMC) 1.Gibbs sampling 2.Metropolis-Hastings 3. Applications du jour: ?

39. Stochastic Sampling Previously: independent samples to estimate P(X = x |e ) Problem: difficult to sample from P(X 1, …. X n |e ) We had to use likelihood weighting –Introduces bias in estimation In some case, such as when the evidence is on leaves, these methods are inefficient

40. MCMC Methods Sampling methods that are based on Markov Chain –Markov Chain Monte Carlo (MCMC) methods Key ideas: –Sampling process as a Markov Chain Next sample depends on the previous one –Approximate any posterior distribution Next: review theory of Markov chains

41. Markov Chains Suppose X 1, X 2, … take some set of values –wlog. These values are 1, 2,... A Markov chain is a process that corresponds to the network: To quantify the chain, we need to specify –Initial probability: P(X 1 ) –Transition probability: P(X t+1 |X t ) A Markov chain has stationary transition probability: P(X t+1 |X t ) same for all times t X1X1 X2X2 X3X3 XnXn...

42. Irreducible Chains A state j is accessible from state i if there is an n such that P(X n = j | X 1 = i) > 0 –There is a positive probability of reaching j from i after some number steps A chain is irreducible if every state is accessible from every state

43. Ergodic Chains A state is positively recurrent if there is a finite expected time to get back to state i after being in state i –If X has finite number of states, then this is suffices that i is accessible from itself A chain is ergodic if it is irreducible and every state is positively recurrent

44. (A)periodic Chains A state i is periodic if there is an integer d such that when n is not divisible by d P(X n = i | X 1 = i ) = 0 Intuition: only every d steps state i may occur A chain is aperiodic if it contains no periodic state

45. Stationary Probabilities Thm: If a chain is ergodic and aperiodic, then the limit exists, and does not depend on i Moreover, let then, P * (X) is the unique probability satisfying

46. Stationary Probabilities The probability P * (X) is the stationary probability of the process Regardless of the starting point, the process will converge to this probability The rate of convergence depends on properties of the transition probability

47. Sampling from the stationary probability This theory suggests how to sample from the stationary probability: –Set X 1 = i, for some random/arbitrary i –For t = 1, 2, …, n Sample a value x t+1 for X t+1 from P(X t+1 |X t =x t ) –return x n If n is large enough, then this is a sample from P * (X)

48. Designing Markov Chains How do we construct the right chain to sample from? –Ensuring aperiodicity and irreducibility is usually easy Problem is ensuring the desired stationary probability

49. Designing Markov Chains Key tool: If the transition probability satisfies then, P * (X) = Q(X) This gives a local criteria for checking that the chain will have the right stationary distribution

50. MCMC Methods We can use these results to sample from P(X 1,…,X n |e) Idea: Construct an ergodic & aperiodic Markov Chain such that P * (X 1,…,X n ) = P(X 1,…,X n |e) Simulate the chain n steps to get a sample

51. MCMC Methods Notes: The Markov chain variable Y takes as value assignments to all variables that are consistent evidence For simplicity, we will denote such a state using the vector of variables

52. Gibbs Sampler One of the simplest MCMC method Each transition changes the state of one X i The transition probability defined by P itself as a stochastic procedure: –Input: a state x 1,…,x n –Choose i at random (uniform probability) –Sample x’ i from P(X i |x 1, …, x i-1, x i+1,…, x n, e) –let x’ j = x j for all j  i –return x’ 1,…,x’ n

53. Correctness of Gibbs Sampler How do we show correctness?

54. Correctness of Gibbs Sampler By chain rule P(x 1,…,x i-1, x i, x i+1,…,x n |e) = P(x 1,…,x i-1, x i+1,…,x n |e)P(x i |x 1,…,x i-1, x i+1,…,x n, e) Thus, we get Since we choose i from the same distribution at each stage, this procedure satisfies the ratio criteria Transition

55. Gibbs Sampling for Bayesian Network Why is the Gibbs sampler “easy” in BNs? Recall that the Markov blanket of a variable separates it from the other variables in the network –P(X i | X 1,…,X i-1,X i+1,…,X n ) = P(X i | Mb i ) This property allows us to use local computations to perform sampling in each transition

56. Gibbs Sampling in Bayesian Networks How do we evaluate P(X i | x 1,…,x i-1,x i+1,…,x n ) ? Let Y 1, …, Y k be the children of X i –By definition of Mb i, the parents of Y j are in Mb i  {X i } It is easy to show that

57. Metropolis-Hastings More general than Gibbs (Gibbs is a special case of M-H) Proposal distribution arbitrary q(x’|x) that is ergodic and aperiodic (e.g., uniform) Transition to x’ happens with probability  (x’|x)=min(1, P(x’)q(x|x’)/P(x)q(x’|x)) Useful when computing P(x) infeasible q(x’|x)=0 implies P(x’)=0 or q(x|x’)=0

58. Sampling Strategy How do we collect the samples? Strategy I: Run the chain M times, each for N steps –each run starts from a different state points Return the last state in each run M chains

59. Sampling Strategy Strategy II: Run one chain for a long time After some “burn in” period, sample points every some fixed number of steps “burn in” M samples from one chain

60. Comparing Strategies Strategy I: –Better chance of “covering” the space of points especially if the chain is slow to reach stationarity –Have to perform “burn in” steps for each chain Strategy II: –Perform “burn in” only once –Samples might be correlated (although only weakly) Hybrid strategy: –Run several chains, sample few times each –Combines benefits of both strategies

61. Summary: Approximate Inference Monte Carlo (sampling with positive and negative error) Methods: –Pos: Simplicity of implementation and theoretical guarantee of convergence –Neg: Can be slow to converge and hard to diagnose their convergence. Variational Methods – Your presentation Loopy Belief Propagation and Generalized Belief Propagation -- Your presentation

62. Next Time Combining Probabilities with Relations and Objects

63. THE END

64. Example: Naïve Bayesian Model A common model in early diagnosis: –Symptoms are conditionally independent given the disease (or fault) Thus, if –X 1,…,X p denote whether the symptoms exhibited by the patient (headache, high- fever, etc.) and –H denotes the hypothesis about the patients health then, P(X 1,…,X p,H) = P(H)P(X 1 |H)…P(X p |H), This naïve Bayesian model allows compact representation –It does embody strong independence assumptions

65. Elimination on Trees Formally, for any tree, there is an elimination ordering with induced width = 1 Thm Inference on trees is linear in number of variables

66. Importance Sampling to LW We want to compute P(Y=y|e)? (X is the set of random variables in the network and Y is some subset we are interested in) 1) Define a mutilated Bayesian network B Z=z to be a network where: all variables in Z are disconnected from their parents and are deterministically set to z all other variables remain unchanged 2) Choose Q to be B E=e convince yourself that P’(X)/Q(X) is exactly P(Y=y|X) 3) Choose f(x) to be 1(Y[m]=y)/M 4) Plug into the formula and you get exactly Likelihood Weighting  Likelihood weighting is correct!!!

67. A Word of Caution Deterministic nodes –Not ergodic in the simple sense –M-H cannot be used