1. Arthur Choi and Adnan Darwiche UCLA {aychoi,darwiche}@cs.ucla.edu
A Variational Approach for Approximating Bayesian Networks by Edge Deletion
Arthur Choi and Adnan Darwiche UCLA
Slides used for plenary presentation at UAI-06.
Updated 09/21/2006.

2. The Idea A C B D A B C D
Approximate inference: Exact inference in an approximate model
Approximate model: by deleting edges

3. The Idea A C B D A B Y X C D
Approximate inference: Exact inference in an approximate model
Approximate model: by deleting edges
Specifying Auxiliary Parameters
Method 1: BP
Method 2: KL

4. The Idea
Original Network
Approximate Network

5. Deleting an Edge U X

6. Deleting an Edge: The CloneU U' X

7. Deleting an Edge: The Soft EvidenceU
New edge parameters for each new query. s' U' X

8. Specifying the Approximation
How do we parametrize edges?
Compensate for the missing edge
Quality of approximation
Which edges do we delete?
Computational complexity

9. A First Approach: ED-BP (Edge Deletion-Belief Propagation)
Choose parameters that satisfy: U s' U' X
Can be used as update equations:
Initialize parameters randomly
Iterate until fixed point is reached
To be presented at AAAI-06.

10. Belief Propagation as Edge Deletion
Theorem:
IBP corresponds to ED-BP U s' U' X

11. Belief Propagation as Edge Deletion
IBP in the original network
ED-BP in a disconnected approximation
To be presented at AAAI-06.

12. Edge Recovery using Mutual Information
MI(U;U'|e') U s' U' X

13. A First Approach: ED-BP (Edge Deletion-Belief Propagation)
How do we parametrize edges?
Subsumes BP as a
degenerate case.
Which edges do we delete?
Recover edges using mutual information

14. A Second Approach Based on the KL-Divergence

15. An Simple Bound on The KL-DivergenceX U X U'
A Bayesian network
An approximation

16. An Simple Bound on The KL-DivergenceX U X U' U X U'
qu'|u = 1 iff u' = u
A Bayesian network
An extended network
An approximation

17. Identifying Edge Parameters: ED-KL
Theorem 1: Edge parameters
are a stationary point of the KL-divergence if and only if: U X U' s'

18. Identifying Edge Parameters: ED-KL
Theorem 1: Edge parameters
are a stationary point of the KL-divergence if and only if: U X U' s'
Theorem 2: Edge parameters
are a stationary point of the KL-divergence if and only if:

19. Deleting a Single Edge When a single edge is deleted, we can:
kl1
When a single edge is deleted, we can:
compute KL-divergence efficiently.
iterate efficiently.

20. Identifying Edges to Delete
kl4
kl1
kl2
kl5
kl3
kl6

21. Comparing ED-BP & ED-KL
ED-BP characterized by:
ED-KL characterized by:

22. Quality of Approximation
Disconnected Approximation
Exact Inference

23. Quality of Approximation
Belief Propagation

24. Quality of Approximation
Belief Propagation

25. Quality of Approximation

26. Quality of Approximation, Extreme Cases

27. Approximating MAP Consider the MAP explanation:
MAP is hard even when marginals are easy!
P(e): complexity in treewidth,
MAP: complexity in constrained treewidth.
Delete edges to reduce constrained treewidth!

28. Quality of MAP Approximations

29. Quality of MAP Approximations

30. Quality of MAP Approximations

31. Complexity of Approximation

32. Summary Approximate Inference Parametrizing Deleted Edges:
Exact inference in an approximate model.
Tradeoff approximation quality with computational resources by deleting edges.
Parametrizing Deleted Edges:
ED-BP: Subsumes belief propagation. (New understanding of belief propagation)
ED-KL: A variational approach.
Choosing Which Edges to Delete:
ED-BP: Edge recovery in terms of mutual information.
ED-KL: Delete edges by (single-edge) KL.
ED-BP + Delete edges by KL: surprisingly good!