1. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 1 Jeff A. Bilmes University of Washington Department of Electrical Engineering EE512 Spring, 2006 Graphical Models Jeff A. Bilmes Lecture 8 Slides April 20 th, 2006

2. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 2 READING: –Chapter 11,12,15 in Jordan’s book Reminder: TA discussions and office hours: –Office hours: Thursdays 3:30-4:30, Sieg Ground Floor Tutorial Center –Discussion Sections: Fridays 9:30-10:30, Sieg Ground Floor Tutorial Center Lecture Room Reminder: take-home Midterm: May 5 th -8 th, you must work alone on this. Announcements

3. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 3 L1: Tues, 3/28: Overview, GMs, Intro BNs. L2: Thur, 3/30: semantics of BNs + UGMs L3: Tues, 4/4: elimination, probs, chordal I L4: Thur, 4/6: chrdal, sep, decomp, elim L5: Tue, 4/11: chdl/elim, mcs, triang, ci props. L6: Thur, 4/13: MST,CI axioms, Markov prps. L7: Tues, 4/18: Mobius, HC-thm, (F)=(G) L8: Thur, 4/20: phylogenetic trees, HMMs L9: Tue, 4/25 L10: Thur, 4/27 L11: Tues, 5/2 L12: Thur, 5/4 L13: Tues, 5/9 L14: Thur, 5/11 L15: Tue, 5/16 L16: Thur, 5/18 L17: Tues, 5/23 L18: Thur, 5/25 L19: Tue, 5/30 L20: Thur, 6/1: final presentations Class Road Map

4. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 4 L1: Tues, 3/28: L2: Thur, 3/30: L3: Tues, 4/4: L4: Thur, 4/6: L5: Tue, 4/11: L6: Thur, 4/13: L7: Tues, 4/18: L8: Thur, 4/20: Team Lists, short abstracts I L9: Tue, 4/25: L10: Thur, 4/27: short abstracts II L11: Tues, 5/2 L12: Thur, 5/4: abstract II + progress L13: Tues, 5/9 L14: Thur, 5/11: 1 page progress report L15: Tue, 5/16 L16: Thur, 5/18: 1 page progress report L17: Tues, 5/23 L18: Thur, 5/25: 1 page progress report L19: Tue, 5/30 L20: Thur, 6/1: final presentations L21: Tue, 6/6 4-page papers due (like a conference paper). Final Project Milestone Due Dates Team lists, abstracts, and progress reports must be turned in, in class and using paper (dead tree versions only). Final reports must be turned in electronically in PDF (no other formats accepted). Progress reports must report who did what so far!!

5. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 5 Factorization property on MRF, (F) When (F) = (G) = (L) = (P) inclusion-exclusion Möbius Inversion lemma Hammersley/Clifford theorem, when (G) => (F) Factorization and decomposability Factorization and junction tree Directed factorization (DF), and (G) Markov blanket Bayesian networks, moralization, and ancestral sets Summary of Last Time

6. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 6 d-Separation, (DL), (DO), and equivalence of all Markov properties on BNs. Phylogenetic Trees and Chordal Models Mixture Models Hidden Markov Models (HMMs) Forward (  ) recursion and elimination Backwards (  ) recursion and elimination Outline of Today’s Lecture

7. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 7 Books and Sources for Today M. Jordan: Chapters 11,12,15. Lauritzen, chapter 3. J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, 1988. T. McKee “Topics in Intersection Graph Theory”

8. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 8 Preservation of (DF) in ancestral sets

9. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 9 Example (DF) – (G)

10. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 10 Example (DF) – (G)

11. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 11 d-Separation revisited

12. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 12 d-Separation revisited

13. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 13 All Together Now

14. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 14 What else can chordal graphs do?

15. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 15 Phylogenetic Tree: example species characters resulting phylogenetic tree

16. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 16 Perfect Phylogeny

17. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 17 Examples

18. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 18 Phylogenetic Trees

19. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 19 Examples: G T, G I 1,1 1,2 1,3 3,1 3,2 3,3 2,1 2,3 2,2

20. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 20 Phylogenetic Trees

21. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 21 Phylogenetic Trees

22. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 22 Intersection Graphs

23. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 23 Intersection Graphs, Chordality, Phylogeny

24. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 24 Summary But triangulated graphs (really ``trees'') have many other properties as well. We are interested in them since they are exactly the class of models on which we can perform exact inference, which is the topic we will next spend some time on. Next topic: Morphing from mixture models to HMMs

25. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 25 Mixture Models

26. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 26 Mixture Models

27. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 27 Inference on Hidden Markov Models Hidden Markov Models (HMMs) are a ubiquitously used model in speech recognition, natural language processing, bioinformatics, financial markets, and many time-series problems. HMMs are rich enough to be interesting, but simple enough so that they are a perfect example to start with when performing exact inference. HMMs can be described either with a BN or an MRF –so this means they must be decomposable Since HMMs are already triangulated (after moralization if necessary), there is no triangulation step. Moreover, since the clique sizes are small, HMMs are easy to deal with (compexity only O(TN 2 )

28. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 28 HMMs and Bayesian Networks

29. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 29 HMMs and Markov Random Fields

30. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 30 HMMs

31. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 31 HMMs

32. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 32 HMMs, elimination orders, and forward recursion

33. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 33 HMMs, elimination orders, and forward recursion

34. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 34 HMMs, elimination orders, and backward recursion

35. Lec 8: April 20th, 2006EE512 - Graphical Models - J. BilmesPage 35 HMMs, elimination orders, and backward recursion