Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.

Slides:



Advertisements
Similar presentations
1 Undirected Graphical Models Graphical Models – Carlos Guestrin Carnegie Mellon University October 29 th, 2008 Readings: K&F: 4.1, 4.2, 4.3, 4.4,
Advertisements

Variational Methods for Graphical Models Micheal I. Jordan Zoubin Ghahramani Tommi S. Jaakkola Lawrence K. Saul Presented by: Afsaneh Shirazi.
CS498-EA Reasoning in AI Lecture #15 Instructor: Eyal Amir Fall Semester 2011.
Graphical Models - Inference - Wolfram Burgard, Luc De Raedt, Kristian Kersting, Bernhard Nebel Albert-Ludwigs University Freiburg, Germany PCWP CO HRBP.
1 Chapter 5 Belief Updating in Bayesian Networks Bayesian Networks and Decision Graphs Finn V. Jensen Qunyuan Zhang Division. of Statistical Genomics,
Lauritzen-Spiegelhalter Algorithm
Statistical Methods in AI/ML Bucket elimination Vibhav Gogate.
Exact Inference in Bayes Nets
1 Appendix B: Solving TSP by Dynamic Programming Course: Algorithm Design and Analysis.
Self Stabilizing Algorithms for Topology Management Presentation: Deniz Çokuslu.
Junction Trees And Belief Propagation. Junction Trees: Motivation What if we want to compute all marginals, not just one? Doing variable elimination for.
An Introduction to Variational Methods for Graphical Models.
Clique Trees Amr Ahmed October 23, Outline Clique Trees Representation Factorization Inference Relation with VE.
Hidden Markov Models M. Vijay Venkatesh. Outline Introduction Graphical Model Parameterization Inference Summary.
Junction Tree Algorithm Brookes Vision Reading Group.
Junction Trees: Motivation Standard algorithms (e.g., variable elimination) are inefficient if the undirected graph underlying the Bayes Net contains cycles.
From Variable Elimination to Junction Trees
Machine Learning CUNY Graduate Center Lecture 6: Junction Tree Algorithm.
GS 540 week 6. HMM basics Given a sequence, and state parameters: – Each possible path through the states has a certain probability of emitting the sequence.
CS774. Markov Random Field : Theory and Application Lecture 06 Kyomin Jung KAIST Sep
Anagh Lal Monday, April 14, Chapter 9 – Tree Decomposition Methods Anagh Lal CSCE Advanced Constraint Processing.
Global Approximate Inference Eran Segal Weizmann Institute.
Bayesian Networks Clique tree algorithm Presented by Sergey Vichik.
Belief Propagation, Junction Trees, and Factor Graphs
Graphical Models Lei Tang. Review of Graphical Models Directed Graph (DAG, Bayesian Network, Belief Network) Typically used to represent causal relationship.
. Inference I Introduction, Hardness, and Variable Elimination Slides by Nir Friedman.
Exact Inference: Clique Trees
PGM 2002/03 Tirgul5 Clique/Junction Tree Inference.
CSC2535 Spring 2013 Lecture 2a: Inference in factor graphs Geoffrey Hinton.
Undirected Models: Markov Networks David Page, Fall 2009 CS 731: Advanced Methods in Artificial Intelligence, with Biomedical Applications.
Exact Inference on Graphical Models Samson Cheung.
Belief Propagation. What is Belief Propagation (BP)? BP is a specific instance of a general class of methods that exist for approximate inference in Bayes.
1 Inferring structure to make substantive conclusions: How does it work? Hypothesis testing approaches: Tests on deviances, possibly penalised (AIC/BIC,
1 Variable Elimination Graphical Models – Carlos Guestrin Carnegie Mellon University October 11 th, 2006 Readings: K&F: 8.1, 8.2, 8.3,
Daphne Koller Variable Elimination Graph-Based Perspective Probabilistic Graphical Models Inference.
Readings: K&F: 11.3, 11.5 Yedidia et al. paper from the class website
Daphne Koller Message Passing Belief Propagation Algorithm Probabilistic Graphical Models Inference.
An Introduction to Variational Methods for Graphical Models
Intro to Junction Tree propagation and adaptations for a Distributed Environment Thor Whalen Metron, Inc.
1 Mean Field and Variational Methods finishing off Graphical Models – Carlos Guestrin Carnegie Mellon University November 5 th, 2008 Readings: K&F:
1 Use graphs and not pure logic Variables represented by nodes and dependencies by edges. Common in our language: “threads of thoughts”, “lines of reasoning”,
Exact Inference in Bayes Nets. Notation U: set of nodes in a graph X i : random variable associated with node i π i : parents of node i Joint probability:
Christopher M. Bishop, Pattern Recognition and Machine Learning 1.
Pattern Recognition and Machine Learning
Today Graphical Models Representing conditional dependence graphically
1 Variable Elimination Graphical Models – Carlos Guestrin Carnegie Mellon University October 15 th, 2008 Readings: K&F: 8.1, 8.2, 8.3,
Knowledge Representation & Reasoning Lecture #5 UIUC CS 498: Section EA Professor: Eyal Amir Fall Semester 2005 (Based on slides by Lise Getoor and Alvaro.
Instructor: Eyal Amir Grad TAs: Wen Pu, Yonatan Bisk
An introduction to chordal graphs and clique trees
Exact Inference Continued
CSCI 5822 Probabilistic Models of Human and Machine Learning
Clique Tree & Independence
Variable Elimination 2 Clique Trees
Cluster Graph Properties
Readings: K&F: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7 Markov networks, Factor graphs, and an unified view Start approximate inference If we are lucky… Graphical.
EE5900 Advanced Embedded System For Smart Infrastructure
Lecture 3: Exact Inference in GMs
Clique Tree Algorithm: Computation
Unifying Variational and GBP Learning Parameters of MNs EM for BNs
Junction Trees 3 Undirected Graphical Models
Readings: K&F: 11.3, 11.5 Yedidia et al. paper from the class website
Properties of BP Algorithm
Variable Elimination Graphical Models – Carlos Guestrin
BP in Practice Message Passing Inference Probabilistic Graphical
Advanced Machine Learning
Mean Field and Variational Methods Loopy Belief Propagation
Generalized Belief Propagation
Presentation transcript:

Junction tree Algorithm :Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati

Cluster Graphs A cluster graph K for a set of factors F is an undirected graph with the following properties: Each node i is associated with a subset C i ½ X Family preserving property: each factor  is such that scope[  ] µ C i Each edge between C i and C j is associated with a sepset S ij = C i Å C j Execution of variable elimination defines a cluster- graph Each factor used in elimination becomes a cluster-node An edge is drawn between two clusters if a message is passed between them in elimination Example: Next slide 

Variable Elimination to Junction Trees: Original graph DifficultyIntelligence Coherence Grade SAT Happy Letter Job

Variable Elimination to Junction Trees: Moralized graph DifficultyIntelligence Coherence Grade SAT Happy Letter Job

Variable Elimination to Junction Trees: Triangulated graph DifficultyIntelligence Coherence Grade SAT Happy Letter Job

Variable Elimination to Junction Trees: Elimination ordering: C, D, I, H, G, S, L DifficultyIntelligence Coherence Grade SAT Happy Letter Job C,D D

Elimination ordering: C, D, I, H, G, S, L DifficultyIntelligence Coherence Grade SAT Happy Letter Job C,D D,I,G D G,I Variable Elimination to Junction Trees:

Elimination ordering: C, D, I, H, G, S, L DifficultyIntelligence Coherence Grade SAT Happy Letter Job C,D D,I,G D G,I,S G,I G,S

Variable Elimination to Junction Trees: Elimination ordering: C, D, I, H, G, S, L DifficultyIntelligence Coherence Grade SAT Happy Letter Job C,D D,I,G D G,I,S G,I H,G,J G,S G,J

Variable Elimination to Junction Trees: Elimination ordering: C, D, I, H, G, S, L DifficultyIntelligence Coherence Grade SAT Happy Letter Job C,D D,I,G D G,I,S G,I H,G,J G,J G,J,S,L G,S J,S,L

Variable Elimination to Junction Trees: Elimination ordering: C, D, I, H, G, S, L DifficultyIntelligence Coherence Grade SAT Happy Letter Job C,D D,I,G D G,I,S G,I H,G,J G,J G,J,S,L G,S J,S,L L,J

Variable Elimination to Junction Trees: Elimination ordering: C, D, I, H, G, S, L DifficultyIntelligence Coherence Grade SAT Happy Letter Job C,D D,I,G D G,I,S G,I H,G,J G,J G,J,S,L G,S J,S,L L,J

Properties of Junction Tree Cluster-graph G induced by variable elimination is necessarily a tree Reason: each intermediate factor is used atmost once G satisfies Running Intersection Property (RIP) (X 2 C i & X in C j ) ) X 2 C K where C k is in the path of C i and C j If C i and C j are neighboring clusters, and C i passes message m ij to C j, then scope[m ij ] = S i,j Let F be set of factors over X. A cluster tree over F that satisfies RIP is called a junction tree One can obtain a minimal junction tree by eliminating the sub-cliques No redundancies C,D D,I,G D G,I,S G,I H,G,J G,J G,J,S,L G,S J,S,L L,J

Junction Trees to Variable elimination: Now we will assume a junction tree and show how to do variable elimination DifficultyIntelligence Coherence Grade SAT Happy Letter Job 1: C,D 2: G,I,D 3: G,S,I 4: G,J,S,L 5: H,G,J D G,I G,S G,J

Junction Trees to Variable Elimination: Initialize potentials first: DifficultyIntelligence Coherence Grade SAT Happy Letter Job 1: C,D 2: G,I,D 3: G,S,I 4:G,J,S,L 5:H,G,J D G,I G,S G,J  0 1 (C,D) = P(C)P(D|C)  0 2 (G,I,D) = P(G|D,I)  0 3 (G,S,I) = P(I)P(S|I)  0 4 (G,J,S,L) = P(L|G)P(J|S,L)  0 5 (H,G,J) = P(H|G,J)

Junction Trees to Variable Elimination: Pass messages: (C 4 is the root) 1: C,D 2: G,I,D 3: G,S,I 4:G,J,S,L 5:H,G,J D G,I G,S G,J  0 1 (C,D) = P(C)P(D|C)  0 2 (G,I,D) = P(G|D,I)  0 3 (G,S,I) = P(I)P(S|I)  0 4 (G,J,S,L) = P(L|G)P(J|S,L)  0 5 (H,G,J) = P(H|G,J)  1 ! 2 (D) =  C  0 1 (C,D)  2 ! 3 (G,I) =  D  0 2 (G,I,D)  1 ! 2 (D)  3 ! 4 (G,S) =  I  0 3 (G,S,I)  2 ! 3 (G,I)  5 ! 4 (G,J) =  H  0 5 (H,G,J)  4 (G,J,S,L) =  3 ! 4 (G,S)  5 ! 4 (G,J)  0 4 (G,J,S,L)

Junction Tree calibration Aim is to compute marginals of each node using least computation Similar to the 2-pass sum-product algorithm C i transmits a message to its neighbor C j after it receives messages from all other neighbors Called “Shafer-Shenoy” clique tree algorithm 1: C,D2: G,I,D 3: G,S,I 4:G,J,S,L5:H,G,J

Message passing with division Consider calibrated potential at node C i whose neighbor is C j Consider message from C i to C j Hence, one can write: CiCi CjCj

Message passing with division Belief-update or Lauritzen-Speigelhalter algorithm Each cluster C i maintains its fully updated current beliefs  i Each sepset s ij maintains  ij, the previous message passed between C i -C j regardless of direction Any new message passed along C i -C j is divided by  ij

Belief Update message passing Example 1: A,B 2: B,C3: C,D BC  12 =  1 ! 2 (B)  23 =  3 ! 2 (C)  2 ! 1 (B) This is what we expect to send in the regular message passing! Actual message

Belief Update message passing Another Example 1: A,B 2: B,C3: C,D BC  2 ! 3 (C) =  0 23  3 ! 2 (C) =  1 23 This is exactly the message C 2 would have received from C 3 if C 2 didn’t send an uninformed message: Order of messages doesn’t matter!

Belief Update message passing Junction tree invariance Recall: Junction Tree measure: A message from C i to C j changes only  j and  ij : Thus the measure remains unchanged for updated potentials too!

Junction trees from Chordal graphs Recall: A junction tree can be obtained by the induced graph from variable elimination Alternative approach: using chordal graphs Recall: Any chordal graph has a clique tree Can obtain chordal graphs through triangulation Finding a minimum triangulation, where largest clique has minimum size is NP-hard

Junction trees from Chordal graphs Maximum spanning tree algorithm Original Graph DifficultyIntelligence Coherence Grade SAT Happy Letter Job

Junction trees from Chordal graphs Maximum spanning tree algorithm Undirected moralized graph DifficultyIntelligence Coherence Grade SAT Happy Letter Job

Junction trees from Chordal graphs Maximum spanning tree algorithm Chordal (Triangulated) graph DifficultyIntelligence Coherence Grade SAT Happy Letter Job

Junction trees from Chordal graphs Maximum spanning tree algorithm Cluster graph DifficultyIntelligence Coherence Grade SAT Happy Letter Job C,D D,I,G 1 G,I,S 2 L,S,J 2 G,S,L 2 G,H

Junction trees from Chordal graphs Maximum spanning tree algorithm Junction tree DifficultyIntelligence Coherence Grade SAT Happy Letter Job C,D D,I,G D G,I,S G,I L,S,J S,L G,S,L G,S G,H G

Summary Junction tree data-structure for exact inference on general graphs Two methods Shafer-Shenoy Belief-update or Lauritzen-Speigelhalter Constructing Junction tree from chordal graphs Maximum spanning tree approach