1. Lecture 25: CS573 Advanced Artificial Intelligence Milind Tambe Computer Science Dept and Information Science Inst University of Southern California Tambe@usc.edu

2. Surprise Quiz II: Part I A B C AP(B) T0.9 F0.05 AP(C) T0.7 F0.01 P(A) = 0.05 Questions: Surprise

3. Markov

4. Markov

5. Dynamic Belief Nets XtXt EtEt In each time slice: Xt = Observable state variables Et = Observable evidence variables X t+1 E t+1 X t+2 E t+2

6. Types of Inference Filtering or monitoring: P(Xt | e1, e2…et) – Keep track of probability distribution over current states – Like POMDP belief state – P(@ISI | c1,c2….ct) and P(N@ISI | c1,c2…ct) Prediction: P(Xt+k | e1,e2…et) for some k > 0 – P(@ISI 3 hours from now | c1,c2…ct) Smoothing or hindsight: P(Xk | e1, e2…et) for 0 <= k < t – What is the state of the user at 11 Am, if observations at 9AM,10AM,11AM, 1pm, 2 pm Most likely explanation: Given a sequence of observations, find the sequence of states that is most likely to have generated the observations (speech recognition) – Argmax x1:t P(X 1:t |e 1:t )

7. Filtering: P(Xt+1 | e1,e2…et+1) P(Xt+1 | e 1:t+1) = f 1:t+1 = Norm * P(et+1 | Xt+1) *  P(Xt+1 | xt) * P(xt| e 1:t) xt e 1:t+1 = e1, e2…et+1 P(xt|e1:t) = f 1:t f 1:t+1 = Norm-const * FORWARD ( f 1:t, et+1) RECURSION

8. Computing Forward f 1:t+1 For our example of tracking user location: f 1:t+1 = Norm-const * FORWARD ( f 1:t, ct+1) Actually it is a vector, not a single quantity f 1:2 = P(L2 | c1, c2) implies computing for both Then normalize Hope you tried out all the computations from the last lecture at home!

9. Robotic Perception XtXt EtEt X t+1 E t+1 X t+2 E t+2 At-1 AtAt+1 At = action at time t (observed evidence) Xt = State of the environment at time t Et = Observation at time t (observed evidence)

10. Robotic Perception Similar to filtering task seen earlier Differences: Must take into account action evidence Norm * P(et+1 | Xt+1) *  P(Xt+1 | xt, at) * P(xt| e 1:t) xt POMDP belief update? Must note that the variables are continuous P(Xt+1 | e 1:t+1, a1:t) = Norm * P(et+1 | Xt+1) * ∫ P(Xt+1 | xt,at) * P(xt| e 1:t, a1:t-1)

11. Prediction Filtering without incorporating new evidence P(Xt+k | e1,e2…et) for some k > 0 – E.g., P( L3 | c1) =  P(L3 | L2) * P(L2 | c1) = (P(L3=@ISI|L2=@ISI)*P(L2=@ISI|c1) + P(L3=@ISI|L2=N@ISI)*P(L2=N@ISI|c1) = 0.7 * 0.6272 + 0.3 * 3728 = 0.43904 + 0.1118 = 0.55 – P(L4 | c1) =  P(L4 | L3) * P(L3 | c1) = 0.7 * 0.55 + 0.3 * 0.45 = 0.52 Computed in the last lecture Computed in the last lecture

12. Prediction – P(L5 | c1) = 0.7 * 0.52 + 0.3* 0.48 = 0.508 – P(L6 | c1) = 0.7 * 0.5 + 0.3 * 0.5 = 0.5… (converging to 0.5) Predicted distribution of user location converges to a fixed point – Stationary distribution of the markov process – Mixing time: Time taken to reach the fixed point Prediction useful if K << mixing time – The more uncertainty there is in the transition model – The shorter the mixing time; more difficult to make predictions

13. Smoothing P(Xk | e1, e2…et) for 0 <= k < t P(Lk | c1,c2…ct) = Norm * P(Lk | c1,c2..ck) * P(ck+1..ct | Lk) = Norm * f 1:k * b k+1:t b k+1:t is a backward message, like our earlier forward message Hence algorithm called forward-backward algorithm

14. b k+1:t backward message b k+1:t = P( e k+1:t | Xk) = P( e k+1, e k+2…. e t | Xk) =  P( e k+1, e k+2…. e t | Xk, Xk+1) P (xk+1 | Xk) xk+1 XkXk EkEk X k+1 E k+1 X k+2 E k+2

15. b k+1:t backward message b k+1:t = P( e k+1:t | Xk) = P( e k+1, e k+1…. e t | Xk) =  P( e k+1, e k+1…. e t | Xk, Xk+1) P (xk+1 | Xk) xk+1 =  P( e k+1, e k+1…. e t | Xk+1) P (xk+1 | Xk) xk+1 =  P( e k+1| Xk+1 ) P( e k+2:t | Xk+1) P (xk+1 | Xk) xk+1

16. b k+1:t backward message P ( e k+1:t | Xk) = b k+1:t =  P ( e k+1| Xk+1 ) P ( e k+2:t | Xk+1) P (xk+1 | Xk) xk+1 b k+1:t = BACKWARD( b k+2:t, e k+1:t) b k+1:t = P( e k+1:t | Xk) = P( e k+1, e k+1…. e t | Xk) =  P( e k+1| Xk+1 ) P( e k+2:t | Xk+1) P (xk+1 | Xk) xk+1

17. Example of Smoothing P(L1 = @ISI | c1, c2) = Norm * P(Lk | c1,c2..ck) * P(ck+1..ct | Lk) = Norm * P(L1 | c1) * P(c2 | L1) = Norm * 0.818 * P(c2 | L1) P(c2 | L1 = @ISI) = P ( e k+1:t | Xk) =  P ( e k+1| Xk+1 ) P ( e k+2:t | Xk+1) P (xk+1 | Xk) xk+1 =>  P(c2 | L2) * P(c3:2|L2) * P(L2 | L1) L2 = [ (0.9 * 1* 0.7) + (0.2 * 1* 0.3)] = 0.69

18. Example of Smoothing P(c2 | L1 = @ISI) =  P(c2 | L2) * P(L2 | L1) L2 = [ (0.9 * 0.7) + (0.2 * 0.3)] = 0.69 P(L1 = @ISI | c1, c2) = Norm * 0.818 * 0.69 = Norm * 0.56442 P(L1 = N@ISI | c1, c2) = Norm * 0.182 * 0.41 = Norm * 0.074 After normalization: P(L1 = @ISI | c1, c2) =.883 Smoothed estimate.883 > Filtered estimate P(L1=@ISI | c1)! WHY?

19. HMM

20. HMM Hidden Markov Models Speech recognition  perhaps the most popular application – Any speech recognition researcher in class? – Waibel and Lee – Dominance of HMMs in speech recognition from 1980s – For ideal isolated conditions they say 99% accuracy – Accuracy drops with noise, multiple speakers Find applications everywhere  just try putting in HMM in google First we gave Bellman update to AI (and other sciences) Now we make our second huge contribution to AI: Viterbi algorithm!

21. HMM Simple nature of HMM allow simple and elegant algorithms Transition model P(Xt+1 | Xt) for all values of Xt – Represented as a matrix |S| * |S| – For our example: Matrix “T” – Tij = P(Xt= j | Xt-1 = i) Sensor model also represented as a Diagonal matrix – Diagonal entries give P(et | Xt = i) – et is the evidence, e.g., ct = true – Matrix Ot

22. HMM f 1:t+1 = Norm-const * FORWARD ( f 1:t, ct+1) = Norm-const * P(ct+1 | Lt+1) *  P(Lt+1 | Lt) * P(Lt|c1,c2…ct) = Norm-const * O t+1 * T T * f 1:t f 1:2 = P (L2 | c1, c2) = Norm-const * O 2 * T T * f 1:1 = Norm-const * * *

23. Transpose

24. HMM f 1:2 = P (L2 | c1, c2) = Norm-const * O 2 * T T * f 1:1 = Norm-const * * * = Norm-const * * = Norm * = Norm * after normalization =

25. Backward in HMM P ( e k+1:t | Xk) = b k+1:t =  P ( e k+1| Xk+1 ) P ( e k+2:t | Xk+1) P (xk+1 | Xk) xk+1 = T * O k+1 * b k+2:t P(c2 | L1 = @ISI) = b 2:2 = * * b 3:2

26. Backward b k+1:t = T * O k+1 * b k+2:t b 3:2 = T * O 2 = * * = ( 0.69 0.41 )

27. Key Results for HMMs f 1:t+1 = Norm-const * O t+1 * T T * f 1:t b k+1:t = T * O k+1 * b k+2:t

28. Inference in DBN How to do inference in a DBN in general? Could unroll the loop forever… XtXt EtEt X t+1 E t+1 X t+2 E t+2 X t+3 E t+3 X t+1 E t+1 X t+2 E t+2 Slices added beyond the last observation have no effect on inference  WHY? So only keep slices within the observation period

29. Inference in DBN XtXt EtEt X t+1 E t+1 Alarm JOHN Mary E t+3 X t+1 E t+1 X t+2 E t+2 Slices added beyond the last observation have no effect on inference  WHY? P(Alarm | JohnCalls)  independent of MaryCalls

30. Complexity of inference in DBN Keep almost two slices in memory – Start with slice 0 – Add slice 1 – “Sum out” slice 0 (get a probability distribution over slice 1 state; don’t need to go back to slice 0 anymore – like POMDPs) – Add slice 2, sum out slice 1… Constant time and space per update Unfortunately, update exponential in the number of state variables Need approximate inference algorithms

31. Solving DBNs in General Exact methods: – Compute intensive – Variable elimination from Chapter 14 Approximate methods: – Particle filtering popularity – Run N samples together through slices of the DBN network – All N samples constitute the forward message – Highly efficient – Hard to provide theoretical guarantees

32. Next Lecture Continue with Chapter 15

33. Student Evaluations

34. Surprise Quiz II: Part II XtXt EtEt X t+1 E t+1 E’ t+1 Xt+1P(E’) T0.7 F0.01 Xt+1P(E) T0.8 F0.01 XtP(Xt+1) T0.5 F Question:

35. Most Likely Path Given a sequence of observations, find the sequence of states that most likely have generated these observations E.g., in the E-elves example, suppose [activity, activity, no-activity, activity, activity] What is the most likely explanation of the presence of the user at ISI over the course of the day? – Did the user step out at time = 3? – Was the user present all the time, but was in a meeting at time 3 Argmax x1:t P (X 1:t | e 1:t )

36. Not so simple… Use smoothing to find the posterior distribution at each time step E.g., compute P(L1=@ISI | c1:5), P(L1=N@ISI | c1:5), find max Do the same for P(L2=@ISI|c1:5) vs P(L2=N@ISI|c1:5) find max Find the maximum this way Why might this be different from computing what we want (the most likey sequence)? max x1:t+1 P (X 1:t+1 | e 1:t+1 ) via viterbi algorithm Norm * P(et+1 | Xt+1) * max (P(Xt+1 | xt) max P(x1….xt-1,xt|e1..et)) xt x1..xt-1