4. Factoring distributionsV X1 X2 X3 X4 X5 X6 X7
Given random variables X1,…,Xn
Partition variables V into sets A and VnA as independent as possible
Formally: Want
A* = argminA I(XA; XVnA) s.t. 0<|A|<n
where I(XA,XB) = H(XB) - H(XB j XA) X1 X3 X4 X6 X2 X5 X7 A
VnA

5. Example: Mutual information
Given random variables X1,…,Xn
z(A) = I(XA; XVnA) = H(XVnA) – H(XVnA |XA)=z(V\A)
Lemma: Mutual information z(A) is submodular
z(A [ {s}) – z(A) = H(Xsj XA) – H(Xsj XVn(A[{s}) )
s(A) = z(A[{s})-z(A) monotonically nonincreasing  z submodular 
Nonincreasing in A: AµB ) H(Xs|XA) ¸ H(Xs|XB)
Nondecreasing in A

6. Queyranne’s algorithm [Queyranne ’98]
Theorem: There is a fully combinatorial, strongly polynomial algorithm for solving A* = argminA z(A) s.t. 0<|A|<n for symmetric submodular functions z
Runs in time O(n3) [instead of O(n8)…]

7. Why are pendent pairs useful?
Key idea: Let (t,u) pendent, A* = argmin z(A)
Then EITHER
t and u separated by A*, e.g., u2A*, tA*. But then A*={u}!! OR
u and t are not separated by A*
Then we can merge u and t… V A* u t V A* u t V A* u t