Arthur Choi and Adnan Darwiche UCLA {aychoi,darwiche}@cs.ucla.edu An Edge Deletion Semantics for Belief Propagation and its Practical Impact on Approximation Quality Slides used for oral presentation at AAAI-06. Updated 09/21/2006. Arthur Choi and Adnan Darwiche UCLA {aychoi,darwiche}@cs.ucla.edu

The Idea A C B D A B C D Approximate inference: Exact inference in an approximate model Approximate model: by deleting edges

The Idea A C B D A B Y X C D Approximate inference: Exact inference in an approximate model Approximate model: by deleting edges BP as approximate models Semantics lead to improved approximations.

The Idea Original Network Approximate Network

Deleting an Edge U X

Deleting an Edge: The Clone U U' X

Deleting an Edge: The Soft Evidence U New edge parameters for each new query. s' U' X

Deleting Edges: Specifying the Approximation How do we parametrize edges? Compensate for the missing edge Quality of approximation Which edges do we delete? Computational complexity

Belief Propagation in Polytrees C D E BP is exact in polytrees.

Belief Propagation in Polytrees U C D X E BP is exact in polytrees.

Belief Propagation in Polytrees U C D X E BP is exact in polytrees.

Belief Propagation in Polytrees U C D X E BP is exact in polytrees.

Belief Propagation in Loopy Networks U C D X E IBP is approximate in loopy networks.

Iterative Belief Propagation in Loopy Networks U C D X E Iteration t = 0, Initialization

Iterative Belief Propagation in Loopy Networks U C D X E Iteration t = 1

Iterative Belief Propagation in Loopy Networks U C D X E Iteration t = 2

Iterative Belief Propagation in Loopy Networks U C D X E Iteration tc Convergence

Iterative Belief Propagation Successfully applied in varied applications: Error-correcting codes Computer vision Satisfiability Etc. U X

Iterative Belief Propagation Successfully applied in varied applications: Error-correcting codes Computer vision Satisfiability Etc. U What does IBP do? Perspective based on statistical physics … Perspective based on deleting edges … X

Deleting Edges: Specifying the Approximation How do we parametrize edges? Compensate for the missing edge Quality of approximation Which edges do we delete? Computational complexity

Parametrizing Edges: ED-BP (Edge Deletion-Belief Propagation) ED-BP: Choose parameters that satisfy U s' U' Used as update equations: Initialize parameters Iterate until we reach a fixed point X

Parametrizing Edges Iteratively: ED-BP Iteration t = 0 Initialization

Parametrizing Edges Iteratively: ED-BP Iteration t = 1

Parametrizing Edges Iteratively: ED-BP Iteration t = 2

Parametrizing Edges Iteratively: ED-BP Iteration tc Convergence

Iterative Belief Propagation as Edge Deletion Theorem: IBP corresponds to ED-BP ED-BP: Iteration t Iteration t

Iterative Belief Propagation as Edge Deletion IBP is a disconnected approximation. IBP in the original network IBP is any polytree approximation.

Splitting the Network into Two Theorem: If deleting an edge splits a network into two independent subnetworks, marginals are exact in each subnetwork. U U' s' X What if deleting an edge doesn’t split a network into two?

Which Edges Do We Delete?

Summary: ED-BP (Edge Deletion-Belief Propagation) How do we parametrize edges? Local conditions: Global conditions: Agreement on parent marginals. Agreement on strengths of evidence. Which edges do we delete? Recover edges using mutual information.

ED-BP: Improving on the Quality of IBP Belief Propagation Exact Inference

ED-BP: Improving on the Quality of IBP Belief Propagation Exact Inference

ED-BP: Potentially Bad Approximations Belief Propagation Unimproved, but costly, approximation, Exact Inference

ED-BP: Improving on the Convergence Rate

ED-BP: Improving on Running Time

ED-BP: Global KL-Divergence Belief Propagation Exact Inference Choi & Darwiche, presented at UAI-06.

ED-BP: Global KL-Divergence Belief Propagation See UAI-06 Exact Inference Choi & Darwiche, presented at UAI-06.

ED-BP: Global KL-Divergence Belief Propagation See UAI-06 Exact Inference Choi & Darwiche, presented at UAI-06.

Summary: ED-BP (Edge Deletion-Belief Propagation) How do we parametrize edges? Characterizes BP: As a fully disconnected approximation, As a class of polytree approximations. Which edges do we delete? Recover edges using mutual information. Are there other ways to delete edges? Yes! Based on the KL-Divergence (see UAI-06).