1. Continuation and Symmetry Breaking Bifurcation of the Information Distortion Function September 19, 2002 Albert E. Parker Complex Biological Systems Department of Mathematical Sciences Center for Computational Biology Montana State University Collaborators: Tomas Gedeon Alexander Dimitrov John P. Miller Zane Aldworth Bryan Roosien

2. Outline  Our Problem  A Class of Problems  Continuation  Bifurcation with Symmetries  How we can efficiently solve the Class of Problems

3. The Neural Coding Problem : We want to understand the neural code. We seek an answer to the question: How does neural activity represent information about environmental stimuli? “The little fly sitting in the fly’s brain trying to fly the fly”

4. The mathematical problem Optimizing the Information Distortion Function max F(q,  )= max (H(Z|Y)+  I(X,Z)) We are really just interested in max I(X,Z).. H(Z|Y) is the conditional entropy of Z|Y. I(X,Z) is the mutual information between X and Z. X Y Q(Y |X) environmental stimuli neural responses Z q(Z |Y) neural responses in N clusters

5. Annealing

7. Random clusters Application of the method to 4 gaussian blobs

8. Information Bottleneck Method (Tishby, Pereira, Bialek 2000) max –I(Y,Z) +  I(X,Z) Used for document classification, gene expression, neural coding and spectral analysis Deterministic Annealing (Rose 1998) max H(Z|Y) +  D(Y,Z) Clustering Algorithm Rate Distortion Theory (Shannon ~1950) max –I(Y,Z) +  D(Y,Z) Similar Problems

9. The Class of Problems max F(q,  )=max(G(q)+  D(q)) To apply the bifurcation theory, the above problem must satisfy: G and D are infinitely differentiable in . G is strictly concave. G and D must be invariant under relabeling of the classes. The hessian of F is block diagonal with N blocks and B  =B  if q(z  |y)= q(z  |y) for every y  Y.

10. The Dynamical System Goal: To efficiently solve max q  (G(q) +  D(q)) for each , incremented in sufficiently small steps, as . Method: Study the equilibria of the of the flow The Jacobian wrt q of the K constraints {  z q(z|y)-1} is J=(I K I K … I K ). If w T  q F(q *,  ) w < 0 for every w  ker J, then q * (  ) is a maximizer of. The first equilibrium is q*(  0 = 0)  1/N. If w T  q F(q *,  ) w < 0 for every w  ker J, then q * (  ) is a maximizer of. The first equilibrium is q*(  0 = 0)  1/N.

11. The Dynamical System How: Use numerical continuation in a constrained system to choose  and to choose an initial guess to find the equilibria q*(  ). Use bifurcation theory with symmetries to understand bifurcations of the equilibria. Investigating the Dynamical System

12. In our dynamical system the hessian determines the stability of equilibria and the location of bifurcation. Theorem: (q *,  * ) is a bifurcation of equilibria of if and only if  q, L (q *,  * ) is singular. Theorem: Let (q *,  * ) be a local solution to. If  q, L (q *,  * ) is singular, then  q F (q *,  * ) is singular. Properties of the Dynamical System

13. Continuation A local maximum q k * (  k ) of is an equilibrium of the gradient flow. Initial condition q k+1 (0) (  k+1 (0) ) is sought in tangent direction   q k, which is found by solving the matrix system The continuation algorithm used to find q k+1 * (  k+1 ) is based on Newton’s method. We cannot use Newton’s method directly, since we have a constrained problem. Instead, we use an Augmented Lagrangian or an implicit solution method.

14.  q* Bifurcation! What to do? How to continue after bifurcation?

15. Do not want to get stuck on a local max  global max 

16. Bifurcations of q * (  ) Observed Bifurcations for the 4 Blob Problem Conceptual Bifurcation Structure  q*  (Y N |Y)

17. Questions … How to detect bifurcation? What kinds of bifurcations do we expect? And how many bifurcating solutions are there? How to choose a direction to take after bifurcation is detected? Explain the nice bifurcation structure that is observed numerically. 1.Why are there only N-1 bifurcations observed? 2.Why are there no bifurcations observed after all N classes have resolved?

18. Bifurcations with symmetry To better understand the bifurcation structure, we capitalize on the symmetries of the optimization function F(q,  ). The “obvious” symmetry is that F(q,  ) is invariant to relabeling of the N classes of Z The symmetry group of all permutations on N symbols is S N. We need to define the action of S N on q and F(q,  ) …

19. The Groups Let P be the finite group of n ×n “block” permutation matrices which represents the action of S N on q and F(q,  ). For example, if N=3, permutes q(z 1 |y) with q(z 2 |y) for every y F(q,  ) is P -invariant means that for every   P, F(  q,  ) = F(q,  ) Let  be the finite group of (n+K) × (n+K) block permutation matrices which represents the action of S N on and  q, L (q,,  ):  q, L (q,,  ) is  -equivariant means that for every     q, L (q,,  ) =  q, L ( ,  )

20. Bifurcations with symmetry The symmetry of is measured by its isotropy subgroup An isotropy subgroup  is a maximal isotropy subgroup of  if there does not exist an isotropy subgroup  of  such that     . At bifurcation, the fixed point subspace of  q*, * is

21. Bifurcations with symmetry One of the tools we use to describe a bifurcation in the presence of symmetries is the Equivariant Branching Lemma (Vanderbauwhede and Cicogna 1980-1). Idea: The bifurcation structure of local solutions is described by the isotropy subgroups of  which have dim Fix(  )=1. System:. r(x,  ) is G -equivariant for some compact Lie Group G Fix( G )={0} Let H be an isotropy subgroup of G such that dim Fix ( H ) = 1. Assume   r(0,0) x 0  0 for nontrivial x 0  Fix ( H ) (crossing condition). Then there is a unique smooth solution branch (tx 0,  (t)) to r = 0 such that x 0  Fix ( H ) and the isotropy subgroup of each solution is H.

22. Bifurcations with symmetry The other tool: Smoller-Wasserman Theorem (1985-6) For variational problems where there is a bifurcating solution tangential to Fix( H ) for every maximal isotropy subgroup H, not only those with dim Fix( H ) = 1. dim Fix( H ) =1 implies that H is a maximal isotropy subgroup

23. From bifurcation, the Equivariant Branching Lemma shows that the following solutions emerge: An equilibria q * is called M-uniform if  q F (q *,  ) has M blocks that are identical. The M classes of Z corresponding to these M identical blocks are called unresolved classes. The classes of Z that are not unresolved are called resolved. The first equilibria, q *  1/N, is N-uniform. Theorem: q * is M-uniform if and only if q * is fixed by S M. What do the bifurcations look like?

24. Theorem: dim ker  q F (q *,  )=M with basis vectors {v i } M i=1 Theorem: dim ker  q, L (q *,,  )=M-1 with basis vectors Point: Since the bifurcating solutions whose existence is guaranteed by the EBL and the SW Theorem are tangential to ker  q, L (q *,,  ), then we know the explicit form of the bifurcating directions. What do the bifurcations look like?

25. Assumptions: Let q * be M-uniform Call the M identical blocks of  q F (q *,  ): B. Call the other N-M blocks of  q F (q *,  ): {R  }. We assume that B has a single nullvector v and that R  is nonsingular for every . If M<N, then B  R  -1 + MI K is nonsingular. Theorem: Let (q *, *,  * ) be a singular point of the flow such that q * is M-uniform. Then there exists M bifurcating (M-1)- uniform solutions (q *, *,  * ) + (tu k,0,  (t)), where What do the bifurcations look like?

26. Example: Some of the bifurcating branches when N=4 are given by the following isotropy subgroup lattice for S 4

27. For the 4 Blob problem: The isotropy subgroups and bifurcating directions of the observed bifurcating branches isotropy group: S 4 S 3 S 2 1 bif direction: (-v,-v,3v,-v,0) T (-v,2v,0,-v,0) T (-v,0,0,v,0) T … No more bifs!

28. The Smoller-Wasserman Theorem shows that (under the same assumptions as before) if M is composite, then there exists bifurcating solutions with isotropy group for every element  of order M in  and every prime p|M. Furthermore, dim (Fix )=p-1 We have never numerically observed solutions fixed by and so perhaps they are unstable. Are there other branches?

29. Bifurcating branches from a 4-uniform solution are given by the following isotropy subgroup lattice for S 4

30. Maximal isotropy subgroup for S 4

31. Issues: S M The full lattice of subgroups of the group S M is not known for arbitrary M.

32. Bifurcation Type? Pitchfork? Conjecture: There are only pitchfork bifurcations. (show that for  ’(t), which depends on  3 qqq F(q,  ), that  ’(0)=0 for any M – have this result for M=2,3) Subcritical or Supercritical? If not pitchfork,  ’(0) >0 or  ’(0) <0 answers this question. If pitchfork, one needs to examine  ’’(0), depends on  4 qqqq F(q,  ). Stability? (Is the bifurcating solution a maximum to ?)

33. Answered Questions How to detect bifurcation? Look for singularity of B. What kinds of bifurcations do we expect? And how many bifurcating solutions are there? M bifurcating (M-1)-uniform solutions How to choose a direction to take after bifurcation is detected? ((M-1)v, -v, -v, -v, … -v, 0) T Explain the nice bifurcation structure that is observed numerically. 1.Why are there only N bifurcations observed. There are only N different types of M-uniform solutions for M  N. 2.Why are there no bifurcations observed after all N classes have resolved. For 1-uniform solutions,  q, L (q *,,  ) is nonsingular.

34. The efficient algorithm Let q 0 be the maximizer of max q G(q),  0 =1 and  s > 0. For k  0, let (q k,  k ) be a solution to max q G(q) +  D(q ). Iterate the following steps until  K =  max for some K. 1.Perform  -step: solve for and select  k+1 =  k + d k where d k =  s /(||   q k || 2 + ||   k || 2 +1) 1/2. 2.The initial guess for q k+1 at  k+1 is q k+1 (0) = q k + d k   q k. 3.Optimization: solve max q G(q) +  k+1 D(q) to get the maximizer q k+1, using initial guess q k+1 (0). 4.Check for bifurcation: compare the sign of the determinant of an identical block of each of  q [G(q k ) +  k D(q k )] and  q [G(q k+1 ) +  k+1 D(q k+1 )]. If a bifurcation is detected, then set q k+1 (0) = q k + d_k u where u is in Fix(H) and repeat step 3.