1. Symmetry Breaking Bifurcation of the Distortion Problem Albert E. Parker Complex Biological Systems Department of Mathematical Sciences Center for Computational Biology Montana State University Collaborators: Tomas Gedeon Alexander Dimitrov John P. Miller Zane Aldworth Bryan Roosien

2. We use Numerical continuation Bifurcation theory with symmetries to analyze a class of optimization problems of the form max F(q,  )=max (G(q)+  D(q)). The goal is to solve for  = B  (0,  ), where:. G and D are infinitely differentiable in . G is strictly concave. D is convex. G and D must be invariant under relabeling of the classes. The hessian of F is block diagonal with N blocks {B  } and B  =B  if q(z  |y)= q(z  |y) for every y  Y. A Class of Problems

3. Deterministic Annealing (Rose 1998) max H(Z|Y) +  D(Y,Z) Clustering Algorithm Rate Distortion Theory (Shannon ~1950) max –I(Y,Z) +  D(Y,Z) Optimal Source Coding Information Distortion (Dimitrov and Miller 2001) max H(Z|Y) +  I(X,Z) Used in neural coding. Information Bottleneck Method (Tishby, Pereira, Bialek 2000) max –I(Y,Z) +  I(X,Z) Used for document classification, gene expression, neural coding and spectral analysis Problems in this class

4. Solution q * of max F(q) when p(X,Y):= 4 gaussian blobs p(X,Y)I(X,Z) vs. N X YZ q(z|y) Q(Y |X) Q * (Z|X ) The optimal quantizer q * (Z|Y) Z Z

5. Some nice properties of the problem The feasible region , a product of simplices, is nice. Lemma  is the convex hull of vertices (  ). The optimal quantizer q* is DETERMINISTIC. Theorem The extrema of lie generically on the vertices of .. Corollary The optimal quantizer is invariant to small perturbations in the model.

6. Annealing G(q)+  D(q)

7. At  = B  (0,  ) Annealing G(q)+  D(q)

8.  q* Conceptual Bifurcation Structure

9. Bifurcations of q * (  ) Observed Bifurcations for the 4 Blob Problem Conceptual Bifurcation Structure  q*  (Y N |Y)

10. Goal: To efficiently solve max q  (G(q) +  D(q)) for each , incremented in sufficiently small steps, as   B. Method: Study the equilibria of the of the flow The Jacobian wrt q of the K constraints {  z q(z|y)-1} is J = (I K I K … I K ). The first equilibrium is q*(  0 = 0)  1/N.. determines stability and location of bifurcation. The Dynamical System

11. Assumptions: Let q * be a local solution to and fixed by S M. Call the M identical blocks of  q F (q *,  ): B. Call the other N-M blocks of  q F (q *,  ): {R  }. At a singularity (q *, *,  * ), B has a single nullvector v and R  is nonsingular for every . If M<N, then B  R  -1 + MI K is nonsingular. Theorem: (q *, *,  * ) is a bifurcation of equilibria of if and only if  q, L (q *, *,  * ) is singular. Theorem: If  q, L (q *, *,  * ) is singular then  q F (q *,  * ) is singular. Theorem: If (q *, *,  * ) is a bifurcation of equilibria of, then  *  1. Theorem: dim (ker  q F (q *,  * )) = M with basis vectors w 1,w 2, …, w M Theorem: dim (ker  q, L (q *, *,  * )) = M-1 with basis vectors Some theoretical results

12. Bifurcations with symmetry To better understand the bifurcation structure, we capitalize on the symmetries of the optimization function F(q,  ). The “obvious” symmetry is that F(q,  ) is invariant to relabeling of the N classes of Z The symmetry group of all permutations on N symbols is S N. The action of S N on and  q, L (q,,  ) is represented by the finite Lie Group where P is a “block permutation” matrix.

13. The Equivariant Branching Lemma gives the existence of bifurcating solutions for every subgroup of  which fixes a one dimensional subspace of ker  q, L (q *,,  ). Theorem: Let (q *, *,  * ) be a singular point of the flow such that q * is fixed by S M. Then there exists M bifurcating solutions, (q *, *,  * ) + (tu k,0,  (t)), each fixed by group S M-1, where What do the bifurcations look like? 

14. For the 4 Blob problem: The subgroups and bifurcating directions of the observed bifurcating branches subgroups: S 4 S 3 S 2 1 bif direction: (-v,-v,3v,-v,0) T (-v,2v,0,-v,0) T (-v,0,0,v,0) T … No more bifs!

15. Partial lattice of the isotropy subgroups of S 4 (and associated bifurcating directions)

16. Let T(q*,  *) = Transcritical or Degenerate? Theorem: If T(q*,  *)  0 and M>2, then the bifurcation at (q*,  *) is transcritical. If T(q*,  *) = 0, it is degenerate. Branch Orientation? Theorem: If T(q*,  *) > 0 or if T(q*,  *) < 0, then the branch is supercritical or subcritical respectively. If T(q*,  *) = 0, then  4 qqqq F(q,  ) dictates orientation. Branch Stability? Theorem: If T(q*,  *)  0, then all branches fixed by S M-1 are unstable. Bifurcation Structure

17. The Smoller-Wasserman Theorem ascertains the existence of bifurcating branches for every maximal isotropy subgroup. Theorem: If M is a composite number, then there exists bifurcating solutions with isotropy group for every element  of order M in  and every prime p|M. The bifurcating direction is in the p-1 dimensional subspace of ker  q, L (q *,,  ) which is fixed by. We have never numerically observed solutions fixed by and so perhaps they are unstable. Other Branches

18. Lattice of the maximal isotropy subgroups in S 4

19. The efficient algorithm Let q 0 be the maximizer of max q G(q),  0 =1 and  s > 0. For k  0, let (q k,  k ) be a solution to max q G(q) +  D(q ). Iterate the following steps until  K =  max for some K. 1.Perform  -step: solve for and select  k+1 =  k + d k where d k =  s /(||   q k || 2 + ||   k || 2 +1) 1/2. 2.The initial guess for q k+1 at  k+1 is q k+1 (0) = q k + d k   q k. 3.Optimization: solve max q G(q) +  k+1 D(q) to get the maximizer q k+1, using initial guess q k+1 (0). 4.Check for bifurcation: compare the sign of the determinant of an identical block of each of  q [G(q k ) +  k D(q k )] and  q [G(q k+1 ) +  k+1 D(q k+1 )]. If a bifurcation is detected, then set q k+1 (0) = q k + d_k u where u is in Fix(H) and repeat step 3.