1. Architecture and Equilibra 结构和平衡 Chapter 6

2. 2002.12.4 2 Chapter 6 Architecture and Equilibria Perface lyaoynov stable theorem

3. 2002.12.4 3 Chapter 6 Architecture and Equilibria 6.1 Neutral Network As Stochastic Gradient system Classify Neutral network model By their synaptic connection topolgies and by how learning modifies their connection topologies synaptic connection topolgies how learning modifies their connection topologies

4. 2002.12.4 4 Chapter 6 Architecture and Equilibria 6.1 Neutral Network As Stochastic Gradient system

5. 2002.12.4 5 Chapter 6 Architecture and Equilibria 6.2 Global Equilibra:convergence and stability Neural network :synapses, neurons three dynamical systems: synapses dynamical systems neuons dynamical systems joint synapses-neurons dynamical systems Historically,Neural engineers study the first or second neural network.They usually study learning in feedforward neural networks and neural stability in nonadaptive feedback neural networks. RABAM and ART network depend on joint equilibration of the synaptic and neuronal dynamical systems.

6. 2002.12.4 6 Chapter 6 Architecture and Equilibria 6.2 Global Equilibra:convergence and stability Equilibrium is steady state (for fixed-point attractors) Convergence is synaptic equilibrium. Stability is neuronal equilibrium. More generally neural signals reach steady state even though the activations still change. We denote steady state in the neuronal field Stability - Equilibrium dilemma : Neuron fluctuate faster than synapses fluctuate. Convergence undermines stability

7. 2002.12.4 7 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids:AVQ Algorithms We shall prove that: Competitve AVQ synaptic vector converge to pattern-class centroid. They vibrate about the centroid in a Browmian motion Competitve learning adpatively qunatizes the input pattern space charcaterizes the continuous distributions of pattern.

8. 2002.12.4 8 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids:AVQ Algorithms The Random Indicator function Supervised learning algorithms depend explicitly on the indicator functions.Unsupervised learning algorthms don’t require this pattern-class information. Centriod Comptetive AVQ Stochastic Differential Equations

9. 2002.12.4 9 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids:AVQ Algorithms The Stochastic unsupervised competitive learning law: We want to show that at equilibrium We assume The equilibrium and convergence depend on approximation (6-11),so 6-10 reduces :

10. 2002.12.4 10 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids:AVQ Algorithms Competitive AVQ Algorithms 1. Initialize synaptic vectors: 2.For random sample,find the closet(“winning”)synaptic vector 3.Update the wining synaptic vectors by the UCL,SCL,or DCL learning algorithm.

11. 2002.12.4 11 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids:AVQ Algorithms Unsupervised Competitive Learning (UCL) defines a slowly deceasing sequence of learning coefficient Supervised Competitive Learning (SCL)

12. 2002.12.4 12 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids:AVQ Algorithms Differential Competitive Learning (DCL) denotes the time change of the jth neuron’s competitive signal. In practice we only use the sign of (6-20) Stochastic Equilibrium and Convergence Competitive synaptic vector coverge to decsion-class centrols. May coverge to locally mixima.

13. 2002.12.4 13 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids:AVQ Algorithms AVQ centroid theorem: if a competitive AVQ system converges,it converge to the centroid of the sampled decision class. Proof. Suppose the jth neuron in Fy wins the actitve competition. Suppose the jth synaptic vector codes for decision class Suppose the synaptic vector has reached equilibrium

14. 2002.12.4 14 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids:AVQ Algorithms

15. 2002.12.4 15 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids:AVQ Algorithms Arguments: The sptial and temporal integrals are approximate equal. The AVQ centriod theorem assumes that convergence occurs. The AVQ centroid convergence theorem ensure : exponential convergence

16. 2002.12.4 16 Chapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem AVQ Convergence Theorem: Competitive synaptic vectors converge exponentially quikly to pattern-class centroids. Proof.Consider the random quadratic form L The pattern vectors x do not change in time. (still valid if the pattern vector x change in time)

17. 2002.12.4 17 Chapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem The average E[L] as Lyapunov function for the sochastic competitice dynamical system. Assume: Noise process is zero-mean and independence of the noise process with “signal”process

18. 2002.12.4 18 Chapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem So,on average by the learning law 6-12, If any synaptic vector move along its trajetory. So, the competitive AVQ system is asymtotically stabel,and in gereral converges exponentially quickly to a locally equilibrium. Suppose IfThen every synaptic vector has Reached equilibrium and is constant.

19. 2002.12.4 19 Chapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem Since p(x) is a nonnegative weigth function. The weighted integral of the learning difference must equal zero : So equilibrium synaptic vector equal centroids. Q.E.D

20. 2002.12.4 20 Chapter 6 Architecture and Equilibra 6.4 AVQ Convergence Theorem Argument Total mean-squared error of vector quantization for the partition So the AVQ convergence theorem implies that the class centroid, and asymptotically,competitive synaptic vector-total mean-squared error. By The Synaptic vectors perform stochastic gradient desent on the mean-squared-error in pettern-plus-error space In the sense :competitive learning reduces to stochostic gradient descent

21. 2002.12.4 21 Chapter 6 Architecture and Equilibria 6.5 Global stability of feedback neural networks Global stability is jointly neuronal-synaptics steady state. Global stability theorems are powerful but limited. Their power: their dimension independence nonlinear generality their exponentially fast convergence to fixed points. Their limitation: do not tell us where the equilibria occur in the state space.

22. 2002.12.4 22 Chapter 6 Architecture and Equilibra 6.5 Global stability of feedback neural networks Stability-Convergence Dilemma Stability-Convergence Dilemma arise from the asymmetry in neounal and synaptic fluctuation rates. Neurons change faster than synapses change. Neurons fluctuate at the millisecond level. Synapses fluctuate at the second or even minute level. The fast-changing neurons must balance the slow-changing synapses.

23. 2002.12.4 23 Chapter 6 Architecture and Equilibria 6.5 Global stability of feedback neural networks Stability-Convergence Dilemma 1.Asymmetry:Neurons in and fluctuate faster than the synapses in M. 2.stability: (pattern formation). 3.Learning: 4.Undoing: the ABAM theorem offers a general solution to stability- convergence dilemma.

24. 2002.12.4 24 Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem The ABAM Theorem The Hebbian ABAM and competitive ABAM models are globally stabel. Hebbian ABAM model: Competitive ABAM model, replacing 6-35 with 6-36

25. 2002.12.4 25 Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem If the positivity assumptions Then, the models are asymptotically stable, and the squared activation and synaptic velocities decrease exponentially quickly to their equilibrium values: Proof. the proof uses the bounded lyapunov function L

26. 2002.12.4 26 Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem Make the difference to 6-37:

27. 2002.12.4 27 Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem To prove global stability for the competitve learning law 6-36 We prove the stronger asymptotic stable of the ABAM models with the positivity assumptions.

28. 2002.12.4 28 Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem Along trajectories for any nonzero change in any neuronal activation or any synapse. Trajectories end in equilibrium points. Indeed 6-43 implies: The squared velocities decease exponentially quickly because of the strict negativity of (6-43) and,to rule out pathologies. Q.E.D

29. 2002.12.4 29 Chapter 6 Architecture and Equilibria 6.7 structural stability of unsuppervised learning and RABAM Is unsupervised learning structural stability? Structural stability is insensivity to small perturbations Structural stability ignores many small perturbations. Such perturbations preserve qualitative properties. Basins of attractions maintain their basic shape.

30. 2002.12.4 30 Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsuppervised learning and RABAM Random Adaptive Bidirectional Associative Memories RABAM Browian diffusions perturb RABAM model. The differential equations in 6-33 through 6-35 now become stochastic differential equations, with random processes as solutions. The diffusion signal hebbian law RABAM model:

31. 2002.12.4 31 Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsuppervised learning and RABAM With the stochastic competitives law:

32. 2002.12.4 32 Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsuppervised learning and RABAM With the stochastic competitives law:

33. 2002.12.4 33 Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsuppervised learning and RABAM With noise (independent zero-mean Gaussian white- noise process). the signal hebbian noise RABAM model:

34. 2002.12.4 34 Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsuppervised learning and RABAM RABAM Theorem. The RABAM model (6-46)-(6-48) or (6-50)-(6-54), is global stable.if signal functions are strictly increasing and ampligication functions and are strictly postive, the RABAM model is asympotically stable. Proof. The ABAM lyapunov function L in (6-37) now defines a random process. At each time t,L(t) is a random variable. The expected ABAM lyapunov function E(L) is a lyapunov function for the RABAM.

35. 2002.12.4 35 Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsuppervised learning and RABAM

36. 2002.12.4 36 Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsuppervised learning and RABAM 【 Reference 】 [1] “Neural Networks and Fuzzy Systems -Chapter 6” P.221-261 Bart kosko University of Southern California.