Factoring distributions V X1 X2 X3 X4 X5 X6 X7 Given random variables X1,…,Xn Partition variables V into sets A and VnA as independent as possible Formally: Want A* = argminA I(XA; XVnA) s.t. 0<|A|<n where I(XA,XB) = H(XB) - H(XB j XA) X1 X3 X4 X6 X2 X5 X7 A VnA
Example: Mutual information Given random variables X1,…,Xn z(A) = I(XA; XVnA) = H(XVnA) – H(XVnA |XA)=z(V\A) Lemma: Mutual information z(A) is submodular z(A [ {s}) – z(A) = H(Xsj XA) – H(Xsj XVn(A[{s}) ) s(A) = z(A[{s})-z(A) monotonically nonincreasing z submodular Nonincreasing in A: AµB ) H(Xs|XA) ¸ H(Xs|XB) Nondecreasing in A
Queyranne’s algorithm [Queyranne ’98] Theorem: There is a fully combinatorial, strongly polynomial algorithm for solving A* = argminA z(A) s.t. 0<|A|<n for symmetric submodular functions z Runs in time O(n3) [instead of O(n8)…]
Why are pendent pairs useful? Key idea: Let (t,u) pendent, A* = argmin z(A) Then EITHER t and u separated by A*, e.g., u2A*, tA*. But then A*={u}!! OR u and t are not separated by A* Then we can merge u and t… V A* u t V A* u t V A* u t