Architecture and Equilibria 结构和平衡 学生：郑巍 导师：刘三阳 2002.12.4 Chapter 6 Architecture and Equilibria 结构和平衡 学生：郑巍 导师：刘三阳

2002.12.4 Chapter 6 Architecture and Equilibria 6.1 Neutral Network As Stochastic Gradient system Classify Neutral network model By their synaptic connection topologies and by how learning modifies their connection topologies synaptic connection topologies how learning modifies their connection topologies 2006.11.10

2002.12.4 Chapter 6 Architecture and Equilibria 6.1 Neutral Network As Stochastic Gradient system Attention :the taxonomy boundaries are fuzzy because the defining terms are fuzzy. 2006.11.10

Three stochastic gradient systems represent the three main categories Chapter 6 Architecture and Equilibria 6.1 Neutral Network As Stochastic Gradient system Three stochastic gradient systems represent the three main categories Backpropagation (BP) Adaptive vector quantization (AVQ) Random adaptive bidirectional associative memory (RABAM) 2006.11.10

2002.12.4 Chapter 6 Architecture and Equilibria 6.2 Global Equilibria: convergence and stability Neural network :synapses , neurons three dynamical systems synapses dynamical systems neurons dynamical systems joint synapses-neurons dynamical systems Historically,Neural engineers study the first or second neural network independently .They usually study learning in feedforward neural networks and neural stability in nonadaptive feedback neural networks. RABAM and ART network depend on joint equilibration of the synaptic and neuronal dynamical systems. 2006.11.10

Convergence undermines stability Chapter 6 Architecture and Equilibria 6.2 Global Equilibria: convergence and stability Equilibrium is steady state (for fixed-point attractors) Convergence is synaptic equilibrium. Stability is neuronal equilibrium. We denote steady state in the neuronal field Another forms with noise Stability - Equilibrium dilemma : Neuron fluctuate faster than synapses fluctuate. Convergence undermines stability 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms Competitive learning adaptively quantizes the input pattern space characterizes the continuous distributions of pattern. We shall prove that: Competitive AVQ synaptic vector converge exponentially to pattern-class centroid. They vibrate about the centroid in a Brownian motion 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms Competitive AVQ Stochastic Differential Equations The Random Indicator function Supervised learning algorithms depend explicitly on the indicator functions.Unsupervised learning algorithms don’t require this pattern-class information. Centriod 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms The Stochastic unsupervised competitive learning law: We assume The equilibrium and convergence depend on approximation (6-11) ,so 6-10 reduces : 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms Competitive AVQ Algorithms 1. Initialize synaptic vectors: 2.For random sample ,find the closet(“winning”)synaptic vector 3.Update the wining synaptic vectors by the UCL ,SCL,or DCL learning algorithm. 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms Unsupervised Competitive Learning (UCL) defines a slowly decreasing sequence of learning coefficient Supervised Competitive Learning (SCL) 2006.11.10

2002.12.4 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms Differential Competitive Learning (DCL) denotes the time change of the jth neuron’s competitive signal . In practice we only use the sign of (6-20) Stochastic Equilibrium and Convergence Competitive synaptic vector converge to decision-class centroids. May converge to locally maxima. 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms AVQ centroid theorem: if a competitive AVQ system converges,it converge to the centroid of the sampled decision class. Proof. Suppose the jth neuron in Fy wins the activity competition. Suppose the jth synaptic vector codes for decision class Suppose the synaptic vector has reached equilibrium 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms Arguments: The spatial and temporal integrals are approximate equal. The AVQ centriod theorem assumes that convergence occurs. The AVQ centroid convergence theorem ensure : exponential convergence 2006.11.10

Chapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem Competitive synaptic vectors converge exponentially quickly to pattern-class centroids. Proof.Consider the random quadratic form L The pattern vectors x do not change in time. (still valid if the pattern vector x change slowly relative to synaptic changes.) 2006.11.10

Chapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem The average E[L] as Lyapunov function for the stochastic competitive dynamical system. Assume: Noise process is zero-mean and independence of the noise process with “signal”process 2006.11.10

Chapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem So ,on average by the learning law 6-12, iff any synaptic vector move along its trajectory. So, the competitive AVQ system is asymptotically stable and in general converges exponentially quickly to a locally equilibrium. Suppose If Then every synaptic vector has Reached equilibrium and is constant . 2006.11.10

Chapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem Since p(x) is a nonnegative weight function. The weighted integral of the learning difference must equal zero : So equilibrium synaptic vector equal centroids. Q.E.D 2006.11.10

Chapter 6 Architecture and Equilibra 6.4 AVQ Convergence Theorem Argument Total mean-squared error of vector quantization for the partition So the AVQ convergence theorem implies that the class centroid, and asymptotically ,competitive synaptic vector-total mean-squared error. By The Synaptic vectors perform stochastic gradient descent on the mean-squared-error surface in pattern-plus-error In the sense :competitive learning reduces to stochastic gradient descent 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.5 Global stability of feedback neural networks Global stability is jointly neuronal-synaptics steady state. Global stability theorems are powerful but limited. Their power: their dimension independence nonlinear generality their exponentially fast convergence to fixed points. Their limitation: do not tell us where the equilibria occur in the state space. 2006.11.10

Chapter 6 Architecture and Equilibra 6 Chapter 6 Architecture and Equilibra 6.5 Global stability of feedback neural networks Stability-Convergence Dilemma Stability-Convergence Dilemma arise from the asymmetry in neuronal and synaptic fluctuation rates. Neurons change faster than synapses change. Neurons fluctuate at the millisecond level. Synapses fluctuate at the second or even minute level. The fast-changing neurons must balance the slow-changing synapses. 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.5 Global stability of feedback neural networks Stability-Convergence Dilemma 1.Asymmetry:Neurons in and fluctuate faster than the synapses in M. 2.stability: (pattern formation). 3.Learning: 4.Undoing: the ABAM theorem offers a general solution to stability-convergence dilemma. 2006.11.10

Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem The ABAM Theorem(证明的关键是找到一个合适的Lyapunov函数） The Hebbian ABAM and competitive ABAM models are globally stable. Hebbian ABAM model: Competitive ABAM model , replacing 6-35 with 6-36 2006.11.10

Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem If the positivity assumptions Then, the models are asymptotically stable, and the squared activation and synaptic velocities decrease exponentially quickly to their equilibrium values: Proof. the proof uses the bounded lyapunov function L 2006.11.10

Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem Make the difference to 6-37: 2006.11.10

Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem To prove global stability for the competitive learning law 6-36 We prove the stronger asymptotic stable of the ABAM models with the positivity assumptions. 2006.11.10

Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem Along trajectories for any nonzero change in any neuronal activation or any synapse. Trajectories end in equilibrium points. Indeed 6-43 implies: The squared velocities decease exponentially quickly because of the strict negativity of (6-43) and ,to rule out pathologies . Q.E.D because of the second-order assumption of nondegenerate Hessian matrix. 2006.11.10

2002.12.4 Chapter 6 Architecture and Equilibria 6.7 structural stability of unsupervised learning and RABAM Is unsupervised learning structural stability? Structural stability is insensitivity to small perturbations Structural stability ignores many small perturbations. Such perturbations preserve qualitative properties. Basins of attractions maintain their basic shape. 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsupervised learning and RABAM Random Adaptive Bidirectional Associative Memories RABAM Brownian diffusions perturb RABAM model. (也就是加进一种噪声） The differential equations in 6-33 through 6-35 now become stochastic differential equations, with random processes as solutions. The diffusion signal hebbian law RABAM model: 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsupervised learning and RABAM With the stochastic competitive law: If is sufficiently steep 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsupervised learning and RABAM With noise (independent zero-mean Gaussian white-noise process). the signal hebbian noise RABAM model: 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsupervised learning and RABAM RABAM Theorem. The RABAM model (6-46)-(6-48) or (6-50)-(6-54), is global stable.if signal functions are strictly increasing and amplification functions and are strictly positive, the RABAM model is asymptotically stable. Proof. The ABAM lyapunov function L in (6-37) now defines a random process. At each time t,L(t) is a random variable. The expected ABAM lyapunov function E(L) is a lyapunov function for the RABAM. 2006.11.10

Chapter 6 Architecture and Equilibria 6 Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsupervised learning and RABAM 2006.11.10

2002.12.4 Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsupervised learning and RABAM 【Reference】 [1] “Neural Networks and Fuzzy Systems -Chapter 6” P.221-261 Bart kosko University of Southern California. 2006.11.10