Belief Propagation Revisited Adnan Darwiche. Graphical Models Battery Age Alternator Fan Belt Battery Charge Delivered Battery Power Starter Radio LightsEngine.

Slides:

Advertisements

Similar presentations

Cornell Accelerating Belief Propagation in Hardware Skand Hurkat and José Martínez Computer Systems Laboratory Cornell University

Advertisements

Graphical Models and Applications CNS/EE148 Instructors: M.Polito, P.Perona, R.McEliece TA: C. Fanti.

Variational Methods for Graphical Models Micheal I. Jordan Zoubin Ghahramani Tommi S. Jaakkola Lawrence K. Saul Presented by: Afsaneh Shirazi.

Bayesian Networks. Quiz: Probabilistic Reasoning 1.What is P(F), the probability that some creature can fly? 2.Creature b is a bumble bee. What’s P(F|B),

. Exact Inference in Bayesian Networks Lecture 9.

A. Darwiche Bayesian Networks. A. Darwiche Reasoning Systems Diagnostics: Which component failed? Information retrieval: What document to retrieve? On-line.

(speaker) Fedor Groshev Vladimir Potapov Victor Zyablov IITP RAS, Moscow.

Exact Inference in Bayes Nets

Junction Trees And Belief Propagation. Junction Trees: Motivation What if we want to compute all marginals, not just one? Doing variable elimination for.

Recursive Conditioning Adnan Darwiche Computer Science Department UCLA.

From Rumelhart (1977) "Toward Interactive Model of Reading"

A. Darwiche Inference in Bayesian Networks. A. Darwiche Query Types Pr: –Evidence: Pr(e) –Posterior marginals: Pr(x|e) for every X MPE: Most probable.

Convergent Message-Passing Algorithms for Inference over General Graphs with Convex Free Energies Tamir Hazan, Amnon Shashua School of Computer Science.

Introduction to Belief Propagation and its Generalizations. Max Welling Donald Bren School of Information and Computer and Science University of California.

EE462 MLCV Lecture Introduction of Graphical Models Markov Random Fields Segmentation Tae-Kyun Kim 1.

Belief Propagation by Jakob Metzler. Outline Motivation Pearl’s BP Algorithm Turbo Codes Generalized Belief Propagation Free Energies.

Belief Propagation on Markov Random Fields Aggeliki Tsoli.

CS774. Markov Random Field : Theory and Application Lecture 04 Kyomin Jung KAIST Sep

GS 540 week 6. HMM basics Given a sequence, and state parameters: – Each possible path through the states has a certain probability of emitting the sequence.

A. Darwiche Bayesian Networks. A. Darwiche Bayesian Network Battery Age Alternator Fan Belt Battery Charge Delivered Battery Power Starter Radio LightsEngine.

Extending Expectation Propagation for Graphical Models Yuan (Alan) Qi Joint work with Tom Minka.

Global Approximate Inference Eran Segal Weizmann Institute.

Understanding Belief Propagation and its Applications Dan Yuan June 2004.

Code and Decoder Design of LDPC Codes for Gbps Systems Jeremy Thorpe Presented to: Microsoft Research

Learning Programs Danielle and Joseph Bennett (and Lorelei) 4 December 2007.

A Trainable Graph Combination Scheme for Belief Propagation Kai Ju Liu New York University.

24 November, 2011National Tsin Hua University, Taiwan1 Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing.

Context-specific independence Graphical Models – Carlos Guestrin Carnegie Mellon University October 16 th, 2006 Readings: K&F: 4.1, 4.2, 4.3, 4.4,

Daphne Koller Message Passing Loopy BP and Message Decoding Probabilistic Graphical Models Inference.

Direct Message Passing for Hybrid Bayesian Networks Wei Sun, PhD Assistant Research Professor SFL, C4I Center, SEOR Dept. George Mason University, 2009.

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 9th Belief propagation Kazuyuki.

Tokyo Institute of Technology, Japan Yu Nishiyama and Sumio Watanabe Theoretical Analysis of Accuracy of Gaussian Belief Propagation.

Readings: K&F: 11.3, 11.5 Yedidia et al. paper from the class website

Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 7th~10th Belief propagation Kazuyuki Tanaka Graduate School of Information Sciences,

Daphne Koller Message Passing Belief Propagation Algorithm Probabilistic Graphical Models Inference.

Approximate Inference: Decomposition Methods with Applications to Computer Vision Kyomin Jung ( KAIST ) Joint work with Pushmeet Kohli (Microsoft Research)

CPSC 422, Lecture 11Slide 1 Intelligent Systems (AI-2) Computer Science cpsc422, Lecture 11 Oct, 2, 2015.

1 Mean Field and Variational Methods finishing off Graphical Models – Carlos Guestrin Carnegie Mellon University November 5 th, 2008 Readings: K&F:

Lecture 2: Statistical learning primer for biologists

Belief Propagation and its Generalizations Shane Oldenburger.

Wei Sun and KC Chang George Mason University March 2008 Convergence Study of Message Passing In Arbitrary Continuous Bayesian.

Exact Inference in Bayes Nets. Notation U: set of nodes in a graph X i : random variable associated with node i π i : parents of node i Joint probability:

Pattern Recognition and Machine Learning

Raptor Codes Amin Shokrollahi EPFL. BEC(p 1 ) BEC(p 2 ) BEC(p 3 ) BEC(p 4 ) BEC(p 5 ) BEC(p 6 ) Communication on Multiple Unknown Channels.

Markov Networks: Theory and Applications Ying Wu Electrical Engineering and Computer Science Northwestern University Evanston, IL 60208

Daphne Koller Introduction Motivation and Overview Probabilistic Graphical Models.

ED-BP: Belief Propagation via Edge Deletion UCLA Automated Reasoning Group Arthur Choi, Adnan Darwiche, Glen Lenker, Knot Pipatsrisawat Last updated 07/15/2010:

Many-Pairs Mutual Information for Adding Structure to Belief Propagation Approximations Arthur Choi and Adnan Darwiche University of California, Los Angeles.

ベーテ自由エネルギーに対するCCCPアルゴリズムの拡張

Compiling Bayesian Networks Using Variable Elimination

Extending Expectation Propagation for Graphical Models

Belief Propagation and Approximate Inference: Compensating for Relaxations Arthur Choi.

StatSense In-Network Probabilistic Inference over Sensor Networks

Exact Inference Continued

Arthur Choi and Adnan Darwiche UCLA

Generalized Belief Propagation

Exact Inference ..

Markov Random Fields Presented by: Vladan Radosavljevic.

Graduate School of Information Sciences, Tohoku University

Arthur Choi and Adnan Darwiche UCLA

Physical Fluctuomatics 7th~10th Belief propagation

Exact Inference Continued

Expectation-Maximization & Belief Propagation

Extending Expectation Propagation for Graphical Models

Probabilistic image processing and Bayesian network

Approximating the Partition Function by Deleting and then Correcting for Model Edges Arthur Choi and Adnan Darwiche University of California, Los Angeles.

Wellcome Centre for Neuroimaging at UCL

Readings: K&F: 11.3, 11.5 Yedidia et al. paper from the class website

Context-specific independence

Generalized Belief Propagation

Presentation transcript:

Belief Propagation Revisited Adnan Darwiche

Graphical Models Battery Age Alternator Fan Belt Battery Charge Delivered Battery Power Starter Radio LightsEngine Turn Over Gas Gauge Gas Fuel Pump Fuel Line Distributor Spark Plugs Engine Start

Graphical Models Battery Age Alternator Fan Belt Battery Charge Delivered Battery Power Starter Radio LightsEngine Turn Over Gas Gauge Gas Fuel Pump Fuel Line Distributor Spark Plugs Engine Start ON OFF OK WEAK DEAD Lights Battery Power

Graphical Models Battery Age Alternator Fan Belt Battery Charge Delivered Battery Power Starter Radio LightsEngine Turn Over Gas Gauge Gas Fuel Pump Fuel Line Distributor Spark Plugs Engine Start

Probabilistic Reasoning in… Diagnosis Planning Learning Channel coding Vision Speech recognition Language comprehension Bioinformatics …

Treewidth w

Larger Shift  Closer Object Smaller Shift  Further Object

Input: L&R Image Output: Depth Map Bayesian Network Images Define a Bayesian Network Reasoning in Bayesian Network Estimates Depth

Belief Propagation

Belief Propagation: What if there are loops? ? ?

Loopy Belief Propagation p.235

The Merit of Loopy Belief Propagation Revolutionary error correcting codes: Turbo Codes (Berrou&Glavieux 1993) LDPC Codes (MacKay&Neal 1995, Gallager 1962) Can closely reach the theoretical limit of communications in noisy channels Turbo & LDPC decoders: Loopy BP in BNs! (McEliece, MacKay & Cheng 1998)

Stereo Vision Two ImagesDepth Map

Stereo Vision Top 4 highest ranking are loopy BP or extend loopy BP

Edge Deletion Semantics (Joint work with Arthur Choi) Energy-Based Semantics (Statistical Physics)

The Idea A C B D

A C B D A C B D

A C B D A C B D X Y New Edge Parameters for each Query

Specifying the Approximation How do we parametrize edges? Quality of approximation Which edges do we delete? Quality of approximation Computational complexity

Parametrizing Edges: ED-BP U X U' s' Choose parameters that satisfy:

Parametrizing Edges Iteratively: ED-BP Iteration t = 0 Initialization

Parametrizing Edges Iteratively: ED-BP Iteration t = 1

Parametrizing Edges Iteratively: ED-BP Iteration t = 2

Parametrizing Edges Iteratively: ED-BP Iteration t c Convergence

Belief Propagation as Edge Deletion Iteration t

Which Edges To Delete? X U' s' U

Which Edges To Delete? X U' s' U

ED-BP: Improving on the Quality of IBP Exact Inference BP

ED-BP: Improving on the Quality of IBP Exact Inference BP

ED-BP: Potentially Bad Approximations Exact Inference BP Unimproved, but costly, approximation,

ED-BP: Improving on the Convergence Rate

ED-BP: Improving on Running Time

Edge Deletion in Undirected Models Original Network Approximate Network

Correcting the Partition Function I i j Theorem: If MI(X i,X j ) = 0 in ED-BP network M', then: where

Deleting Many Edges This will yield the Bethe free energy approximation!

Correcting the Partition Function II i j Theorem: For an ED-BP network M', we have: where

Which Edges Do We Recover? EC2? ij Recover edges with largest  kl MI(X i,X j ;X k X l ) !

Experiment: Random Grid x edges recovered |log Z - log Z'| EC1,rand EC2,rand EC1,MI EC2,MI EC1,2MI EC2,2MI Bethe exact yfq

Beyond Treewidth… Exact Inference: Exploit Non-structural Independence Approximate inference: Exact inference on an approximate network obtained by deleting edges

What next? Constant factors! Guarantees/bounds on approximations Edge recovery heuristics: getting the most out of the extra time Controlling tradeoff between quality & complexity Dynamic models Logical reasoning: Survey propagation