1. Compiling Bayesian Networks Using Variable Elimination
Mark Chavira
Adnan Darwiche
UCLA

2. Outline Motivation Review ADD Compilation Experimental Results
Conclusion

3. Probabilistic Inference
Battery Age
Alternator
Fan Belt
Charge Delivered
Battery
Fuel Pump
Fuel Line
Starter
Gas
Distributor
Battery Power
Spark Plugs
Gas Gauge
Radio
Lights
Engine Turn Over
Engine Start

4. Multiple Queries
MAP
Sensitivity Analysis
State Estimation
Diagnosis …

5. Multiple Queries Diabetes Munin4 4255 queries
1.8 seconds using jointree
6.7 minutes using VE
Munin4
4580 queries
2.7s seconds using jointree
2.2 days using VE
Munin4: 1041 variables with average cardinality 5.4, 3.3 weeks using ADDs
Diabetes: 413 variables with average cardinality 11.3, 3.9 hours using ADDs
Mildew: 35 variables with average cardinality 17.6
581 queries
0.6 seconds using jointree
4.2 seconds using tables
1.5 hours using ADDs

6. Local Structure: CSI and Determinism
Battery Age
Alternator
Fan Belt
Charge Delivered
Battery
Fuel Pump
Fuel Line
Starter
Gas
Distributor
Battery Power
Spark Plugs
Gas Gauge
Radio
Lights
Engine Turn Over
Engine Start

7. Local Structure: CSI and Determinism
Battery Age
Alternator
Fan Belt
Charge Delivered
Battery
Fuel Pump
Fuel Line
Starter
Gas
Distributor
Battery Power
Spark Plugs
Gas Gauge
Radio
Lights
Engine Turn Over
Engine Start
Context Specific Independence (CSI)

8. Local Structure: CSI and Determinism
Battery Age
Alternator
Fan Belt
Charge Delivered
Battery
Fuel Pump
Fuel Line
Starter
Gas
Distributor
Battery Power ON
OFF OK
WEAK
DEAD
Lights
Battery Power
.99
.01
.20
.80 1
Spark Plugs
Gas Gauge
If Battery Power = Dead,
then Lights = OFF
Determinism
Radio
Lights
Engine Turn Over
Engine Start

9. Previous Work: VE and Local Structure
Bahar et al, 1993, ADDs
Sanner and McAllester, 2005, Affine ADDs
Poole and Zhang, 2003, Confactors
Larkin and Dechter, 2003, Sparse Representations …

10. Overhead Network Tabular (ms) ADD (ms) Times Worse barley 307 14,049
45.8
bm-5-3
4,892
658
0.1
diabetes
949
33,220
35.0
hailfinder 48
515
10.7
link
1,688
2,658
1.6
mm-3-8-3
2,166
843
0.4
mildew 72
92,602
1286.1
munin1
155
1,255
8.1
munin2
204
3,170
15.5
munin3
350
5,049
14.4
munin4
406
4,361
pathfinder 51
5,213
102.2
pigs 69
597
8.7
st-3-2
186
362
1.9
water 76
1,015
13.4

11. Outline Motivation Review ADD Compilation Experimental Results
Conclusion

12. Variable Elimination Elimination Order: A,B,C T1(A,B) T2(A,C) T3(B)
T4(C)
T5(A,B,C)
T6(B,C)
T7(B,C)
T8(C)
T9(C)
T10()

13. Algebraic Decision Diagrams (ADDs)X Y Z
f(.) x1 y1 z1
0.9 z2
0.1 y2 x2
0.5 X Z Y Z .1 .9 .5

14. Network MLF A B Set θx|u to Pr(x|u)
Set λx to 1(0) iff x is (is not) compatible with e

15. Circuit Evaluation and Differentiation.3 + 1 * * + + * * * * * * 1 .3
.03
.27 .7

16. Outline Motivation Review ADD Compilation Experimental Results
Conclusion

17. Goals Answer multiple queries Exploit local structure (w/o overhead)
Use VE as a basis for compiling a BN
Exploit local structure (w/o overhead)
Use structured factors rather than tables

18. Symbolic ADDs * * X X Y Y Y Y .4 .6 .9 .1 Normal ADD Symbolic ADD y2
y2 .5
y1
Normal ADD
Symbolic ADD

19. CPT Symbolic ADD X X Z Y Z Y Z Z .1 .9 .5 .1 .9 .5 X Y Z f(.) x1 y1 z1
0.9 z2
0.1 y2 x2
0.5 Z Y Z Y Z Z .1 .9 .5 .1 .9 .5

20. Indicator ADDs X
x1
x2

21. Symbolic ADD Operations

22. Changing a Single Line = * .1 .9
.09 = * * *
y2 *
y2 .5
y1 .5
y1

23. Multiplication ExampleY Y Y = Y * * * * *
y1
y2
y1 1
y2 .5
y1 .5
y2 1 .5

24. Local Structure * * * * * * X X Y Y Y Y y2 y1 1 y2 .5 y1 .5 y2 .5
y2
y1 1
y2 .5
y1 .5
y2 .5
y1

25. Local Structure * c1 c2
c1*c2 * α * 1 α + c1 c2
c1+c2 + α α

26. How Does it Work? Generate CPT and Indicator ADDs Perform VE as normal
Result is ADD sink labeled with pointer to compilation!

27. Other Details ADD Variable Order Unique Table for ADD nodes
Construction of CPT ADDs
Dealing with multi-valued variables

28. Example: Convert to Symbolic
Order: Y,X X
Pr(x) x1
0.1 x2
0.9 X X X X Y Y Y .1 .9
x1
x2
y1
y2 1 .5 X Y
Pr(y|x) x1 y1
0.0 y2
1.0 x2
0.5
CPT ADD
for X
Indicator
ADD for X
Indicator
ADD for Y
CPT ADD
for Y

29. Example: Multiply Y Order: Y,X X X X Y Y .1 .9 1 .5 CPT ADD for X1 .5
CPT ADD
for X
Indicator
ADD for X
Indicator
ADD for Y
CPT ADD
for Y

30. Example: Sum Out Y * * Order: Y,X X X X Y Y .1 .9 CPT ADD for X
Indicator
ADD for X
y2 .5
y1

31. Example: Multiply X + * * Order: Y,X X X X .1 .9 CPT ADD for X
Indicator
ADD for X
y1 .5
y2

32. Example: Multiply X * * + * * Order: Y,X X X .1 .9 CPT ADD for X x1
And then these two.
x1
x2 + * *
CPT ADD
for X
y1 .5
y2

33. Example: Sum Out X
Order: Y,X X * * .1 * * .9
x1 +
x2 * *
y1 .5
y2

34. Example: Compilation Complete!
Order: Y,X + * * .1 * * .9
x1 +
x2 * *
y1 .5
y2

35. Outline Motivation Review ADD Compilation Experimental Results
Conclusion

36. Offline Inference Network Ace (s) ADD (s) Times Better barley 8190.2
122.8
66.7
bm-5-3
0.8
6.0
0.1
diabetes
1710.0
110.3
15.5
hailfinder
0.7
1.2
0.5
link –
699.7
mm-3-8-3
1.5
11.9
mildew
3125.2
218.9
14.3
munin1
1005.1
316.7
3.2
munin2
198.4
31.7
6.3
munin3
188.4
17.6
10.7
munin4
205.0
37.8
5.4
pathfinder
4.9
5.8
0.9
pigs
23.1
10.0
2.3
st-3-2
2.4
0.2
water
3.0
20.7

37. Online Inference Network JT (ms) ADD (ms) Times Better barley 65,226
35,209
1.9
bm-5-3
89,593 83
1079.4
diabetes
29,316
20,421
1.4
hailfinder
245 70
3.5
link
223,542
175,769
1.3
mm-3-8-3
34,001
198
171.7
mildew
10,077
4,522
2.2
munin1
669,915
37,451
17.9
munin2
17,857
7,180
2.5
munin3
13,351
4,945
2.7
munin4
42,754
8,683
4.9
pathfinder
1,332
102
13.1
pigs
3,020
2,814
1.1
st-3-2
17,536 82
213.9
water
16,676
251
66.4

38. Conclusion VE limitations: VE as a basis for compilation
Multiple queries
Local structure
VE as a basis for compilation
Solves the multiple query problem
Solves the local structure (overhead) problem
Orders of magnitude more efficient than jointree
Makes body of research more effective
Implications beyond VE

39. Previous Work: VE and Multiple Queries
Cozman, 2000
Darwiche, 2000
There has been a limited amount of work to enhance VE to answer multiple queries simultaneously as does the jointree algorthm. For example, we might wish to compute probability of evidence and a posterior marginal on each variable simultaneously. But this work hasn’t improve performance over jointree. We’ll see over the course of this talk, that we can solve the multiple query problem while at the same time dramatically outperforming jointree local structure exists in the network.

40. Global Structure

41. Compiling Bayesian Networks
P(e) =
λa1λb1λc1θa1θb1θc1|a1 b1 +
λa1λb1λc2θa1θb1θc2|a1 b1 +
λa1λb1λc3θa1θb1θc3|a1 b1 +
λa1λb2λc1θa1θb2θc1|a1 b2 +
...
λa2λb2λc3θa2θb2θc3|a2 b2