Artificial Neural Networks Ch15

2 Objectives Grossberg network is a self-organizing continuous-time competitive network.  Continuous-time recurrent networks  The foundation for the adaptive resonance theory The biological motivation for the Grossberg network: the human visual system

3 Biological Motivation: Vision Eyeball and Retina

Leaky Integrator In mathematics, a leaky integrator equation is a specific differential equation, used to describe a component or system that takes the integral of an input, but gradually leaks a small amount of input over time. It appears commonly in hydraulics, electronics, and neuroscience where it can represent either a single neuron or a local population of neurons 4

5 Basic Nonlinear Model Leaky Integrator: : the system time constant  (input) is constant and  n(t):response

P(t)=1 n(0)=0, = 1, 0.5, 0.25,

7 Shunting Model Excitatory: the input that causes the response to increase –– p +. Inhibitory: the input that causes the response to decrease –– p . Biases b + and b  (nonnegative) determine the upper and lower limits on the response. Linear decay term Nonlinear gain control

8 Grossberg Network Three components: Layer 1, Layer 2 and the adaptive weights. The network includes short-term memory (STM) and long-term memory (LTM) mechanisms, and performs adaptation, filtering, normalization and contrast enhancement.

9 Layer 1 Receives external inputs and normalizes the intensity of the input pattern.

10 On-Center / Off-Surround Excitatory input: Inhibitory input: This type of connection pattern produces a normalization of the input pattern.

11 Normalization Inhibitory bias, Excitatory bias Steady state neuron output where is the relative intensity of input i. The input vector is normalized

12 Layer 1 Response p = p =

13 Characteristics of Layer 1 The network is sensitive to relative intensities of the input pattern, rather than absolute intensities. The output of Layer 1 is a normalized version of the input pattern. The on-center/off-surround connection pattern and the nonlinear gain control of the shunting model produce the normalization effect. The operation of Layer 1 explains the brightness constancy and brightness contrast characteristics of the human visual system.

14 Layer 2 A layer of continuous-time instar performs several functions.

15 Functions of Layer 2 It normalizes the total activity in the layer. It contrast enhances its pattern, so that the neuron that receives the largest input will dominate the response (like winner-take-all competition in the Hamming network). It operates as a short-term memory by storing the contrast-enhanced pattern.

16 Feedback Connections The feedback enables the network to store a pattern, even after the input has been removed. The feedback also performs the competition that causes the contrast enhancement of the pattern.

17 Equation of Layer 2 provides on-center feedback connection provides off-surround feedback connection W 2 consists of adaptive weights. Its rows, after training, will represent the prototype patterns. Layer 2 performs a competition between the neurons, which tends to contrast enhance the output pattern – maintaining large outputs while attenuating small outputs.

18 Layer 2 Example Correlation between prototype 1 and input. Correlation between prototype 2 and input.

19 Layer 2 Response t w 2 1  T a 1 w 2 2  T a 1 n 1 2 t() n 2 2 t() Contrast Enhancement and Storage Input to neuron 1: Input to neuron 2: a 1 (the steady state resulted obtained from Layer 1 example) is applied for 0.25 sec. and then removed. Input vector:

20 Characteristics of Layer 2 Even before the input is removed, some contrast enhancement is performed. After the input has been set to zero, the network further enhances the contrast and stores the patterns.  It is the nonlinear feedback that enables the network to store pattern, and the on- center/off-surround connection patterns that causes the contrast enhancement.

21 Transfer Functions See Fig Linear: perfect storage of any pattern, but amplify noise. Slower than linear: amplify noise, reduce contrast. Faster than linear: winner-take-all, suppress noise, quantize total activity. Sigmoid: suppress noise, contrast enhance, not quantized.

22 Learning Law The rows of the adaptive weights W 2 will represent patterns that have been stored and that the network will be able to recognize.  Long term memory (LTM) Learning law #1: Learning law #2: turn off learning (and forgetting) when is NOT active. decay term Hebbian-type learning

23 Response of Adaptive Weights For Pattern 1: For Pattern 2: w 11  2 t() w 12  2 t() w 21  2 t() w 22  2 t() The first row of the weight matrix is updated when n 1 2 (t) is active, and the second row of the weight matrix is updated when n 2 2 (t) is active. Two different input patterns are alternately presented to the network for periods of 0.2 seconds at a time.

24 Relation to Kohonen Law Grossberg learning law (continuous-time): Euler approximation for the derivative: Discrete-time approximation to Grossberg law:

25 Relation to Kohonen Law Rearrange terms: Assume that a faster-than-linear transfer function (winner-take-all) is used in Layer 2. Kohonen law:

26 Three Major Differences The Grossberg network is a continuous-time network. Layer 1 of Grossberg network automatically normalizes the input vectors. Layer 2 of Grossberg network can perform a “soft” competition, rather than the winner-take-all competition of the Kohonen network.  This soft competition allows more than one neuron in Layer 2 to learn.  Cause the Grossberg network to operate as a feature map.