1. Architecture and Equilibria 结构和平衡 学生：郑巍 导师：刘三阳
Chapter 6
Architecture and Equilibria 结构和平衡 学生：郑巍 导师：刘三阳

2.
Chapter 6 Architecture and Equilibria 6.1 Neutral Network As Stochastic Gradient system
Classify Neutral network model By their synaptic connection topologies and by how learning modifies their connection topologies
synaptic connection topologies
how learning modifies their connection topologies

3.
Chapter 6 Architecture and Equilibria 6.1 Neutral Network As Stochastic Gradient system
Attention :the taxonomy boundaries are fuzzy because the defining terms are fuzzy.

4. Three stochastic gradient systems represent the three main categories
Chapter 6 Architecture and Equilibria 6.1 Neutral Network As Stochastic Gradient system
Three stochastic gradient systems represent the three main categories
Backpropagation (BP)
Adaptive vector quantization (AVQ)
Random adaptive bidirectional associative memory (RABAM)

5.
Chapter 6 Architecture and Equilibria 6.2 Global Equilibria: convergence and stability
Neural network :synapses , neurons
three dynamical systems
synapses dynamical systems
neurons dynamical systems
joint synapses-neurons dynamical systems
Historically,Neural engineers study the first or second neural network independently .They usually study learning in feedforward neural networks and neural stability in nonadaptive feedback neural networks. RABAM and ART network depend on joint equilibration of the synaptic and neuronal dynamical systems.

6. Convergence undermines stability
Chapter 6 Architecture and Equilibria 6.2 Global Equilibria: convergence and stability
Equilibrium is steady state (for fixed-point attractors)
Convergence is synaptic equilibrium.
Stability is neuronal equilibrium.
We denote steady state in the neuronal field
Another forms with noise
Stability - Equilibrium dilemma :
Neuron fluctuate faster than synapses fluctuate.
Convergence undermines stability

7. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms
Competitive learning adaptively quantizes the input pattern space
characterizes the continuous distributions of pattern.
We shall prove that:
Competitive AVQ synaptic vector converge exponentially to pattern-class centroid. They vibrate about the centroid in a Brownian motion

8. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms
Competitive AVQ Stochastic Differential Equations
The Random Indicator function
Supervised learning algorithms depend explicitly on the indicator functions.Unsupervised learning algorithms don’t require this pattern-class information.
Centriod

9. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms
The Stochastic unsupervised competitive learning law:
We assume
The equilibrium and convergence depend on approximation (6-11) ,so reduces :

10. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms
Competitive AVQ Algorithms
1. Initialize synaptic vectors:
2.For random sample ,find the closet(“winning”)synaptic
vector
3.Update the wining synaptic vectors by the UCL ,SCL,or DCL learning algorithm.

11. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms
Unsupervised Competitive Learning (UCL)
defines a slowly decreasing sequence of learning coefficient
Supervised Competitive Learning (SCL)

12.
Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms
Differential Competitive Learning (DCL)
denotes the time change of the jth neuron’s competitive
signal . In practice we only use the sign of (6-20)
Stochastic Equilibrium and Convergence
Competitive synaptic vector converge to decision-class centroids.
May converge to locally maxima.

13. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms
AVQ centroid theorem:
if a competitive AVQ system converges,it converge to the centroid of the sampled decision class.
Proof. Suppose the jth neuron in Fy wins the activity competition.
Suppose the jth synaptic vector codes for decision class
Suppose the synaptic vector has reached equilibrium

14. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms

15. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.3 Synaptic convergence to centroids: AVQ Algorithms
Arguments:
The spatial and temporal integrals are approximate equal.
The AVQ centriod theorem assumes that convergence occurs.
The AVQ centroid convergence theorem ensure :
exponential convergence

16. Chapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem
Competitive synaptic vectors converge exponentially quickly to pattern-class centroids.
Proof.Consider the random quadratic form L
The pattern vectors x do not change in time.
(still valid if the pattern vector x change slowly relative to synaptic changes.)

17. Chapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem
The average E[L] as Lyapunov function for the stochastic competitive dynamical system.
Assume: Noise process is zero-mean and independence of the noise process with “signal”process

18. Chapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem
So ,on average by the learning law 6-12,
iff any synaptic vector move along its trajectory.
So, the competitive AVQ system is asymptotically stable and in general converges exponentially quickly to a locally equilibrium.
Suppose If
Then every synaptic vector has
Reached equilibrium and is constant .

19. Chapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem
Since p(x) is a nonnegative weight function.
The weighted integral of the learning difference
must equal zero :
So equilibrium synaptic vector equal centroids. Q.E.D

20. Chapter 6 Architecture and Equilibra 6.4 AVQ Convergence Theorem
Argument
Total mean-squared error of vector quantization for the partition
So the AVQ convergence theorem implies that
the class centroid, and asymptotically ,competitive synaptic vector-total mean-squared error. By
The Synaptic vectors perform stochastic gradient descent on the mean-squared-error surface in pattern-plus-error
In the sense :competitive learning reduces to stochastic gradient descent

21. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.5 Global stability of feedback neural networks
Global stability is jointly neuronal-synaptics steady state.
Global stability theorems are powerful but limited.
Their power:
their dimension independence
nonlinear generality
their exponentially fast convergence to fixed points.
Their limitation:
do not tell us where the equilibria occur in the state space.

22. Chapter 6 Architecture and Equilibra 6
Chapter 6 Architecture and Equilibra 6.5 Global stability of feedback neural networks
Stability-Convergence Dilemma
Stability-Convergence Dilemma arise from the asymmetry in neuronal and synaptic fluctuation rates.
Neurons change faster than synapses change.
Neurons fluctuate at the millisecond level.
Synapses fluctuate at the second or even minute level.
The fast-changing neurons must balance the slow-changing synapses.

23. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.5 Global stability of feedback neural networks
Stability-Convergence Dilemma
1.Asymmetry:Neurons in and fluctuate faster than the synapses in M.
2.stability: (pattern formation).
3.Learning:
4.Undoing:
the ABAM theorem offers a general solution to stability-convergence dilemma.

24. Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem
The ABAM Theorem(证明的关键是找到一个合适的Lyapunov函数）
The Hebbian ABAM and competitive ABAM models are globally stable.
Hebbian ABAM model:
Competitive ABAM model , replacing 6-35 with 6-36

25. Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem
If the positivity assumptions
Then, the models are asymptotically stable,
and the squared activation and synaptic velocities decrease exponentially quickly to their equilibrium values:
Proof. the proof uses the bounded lyapunov function L

26. Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem
Make the difference to 6-37:

27. Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem
To prove global stability for the competitive learning law 6-36
We prove the stronger asymptotic stable of the ABAM models
with the positivity assumptions.

28. Chapter 6 Architecture and Equilibria 6.6 The ABAM Theorem
Along trajectories for any nonzero change in any neuronal activation or any synapse.
Trajectories end in equilibrium points.
Indeed 6-43 implies:
The squared velocities decease exponentially quickly because of the strict negativity of (6-43) and ,to rule out pathologies .
Q.E.D because of the second-order assumption of nondegenerate Hessian matrix.

29.
Chapter 6 Architecture and Equilibria 6.7 structural stability of unsupervised learning and RABAM
Is unsupervised learning structural stability?
Structural stability is insensitivity to small perturbations
Structural stability ignores many small perturbations.
Such perturbations preserve qualitative properties.
Basins of attractions maintain their basic shape.

30. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsupervised learning and RABAM
Random Adaptive Bidirectional Associative Memories RABAM
Brownian diffusions perturb RABAM model. (也就是加进一种噪声）
The differential equations in 6-33 through now become stochastic differential equations, with random processes as solutions.
The diffusion signal hebbian law RABAM model:

31. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsupervised learning and RABAM
With the stochastic competitive law:
If is sufficiently steep

32. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsupervised learning and RABAM
With noise (independent zero-mean Gaussian white-noise process).
the signal hebbian noise RABAM model:

33. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsupervised learning and RABAM
RABAM Theorem.
The RABAM model (6-46)-(6-48) or (6-50)-(6-54),
is global stable.if signal functions are strictly increasing and amplification functions and are strictly positive, the RABAM model is asymptotically stable.
Proof. The ABAM lyapunov function L in (6-37) now defines
a random process. At each time t,L(t) is a random variable.
The expected ABAM lyapunov function E(L) is a lyapunov function for the RABAM.

34. Chapter 6 Architecture and Equilibria 6
Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsupervised learning and RABAM

35.
Chapter 6 Architecture and Equilibria 6.7 Structural stability of unsupervised learning and RABAM
【Reference】
[1] “Neural Networks and Fuzzy Systems -Chapter 6” P Bart kosko University of Southern California.