1. NEURAL NETWORK THEORY NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS 欢迎大家提出意见建议！ 2003.10.15

2. 2 NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS NEURONS AS FUNCTIONS Neurons behave as functions. Neurons transduce an unbounded input activation x(t) at time t into a bounded output signal S(x(t)).

3. 3 NEURONS AS FUNCTIONS The transduction description: a sigmoidal or S-shaped curve the logistic signal function: The logistic signal function is sigmoidal and strictly increases for positive scaling constant c >0. NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

4. 4 NEURONS AS FUNCTIONS S(x) x 0 -∞-++∞ Fig.1 s(x) is a bounded monotone-nondecreasing function of x If c → +∞ ， we get threshold signal function (dash line), Which is piecewise differentiable NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

5. 5 SIGNAL MONOTONICITY In general, signal functions are monotone nondecreasing S’>=0. This means signal functions have an upper bound or saturation value. The staircase signal function is a piecewise-differentiable Monotone-nondecreasing signal function. NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

6. 6 SIGNAL MONOTONICITY An important exception: bell-shaped signal function or Gaussian signal functions The sign of the signal-activation derivation s’ is opposite the sign of the activation x. We shall assume signal functions are monotone nondecreasing unless stated otherwise. NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

7. 7 SIGNAL MONOTONICITY Generalized Gaussian signal function define potential or radial basis function : input activation vector: variance: mean vector: NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS we shall consider only scalar-input signal functions:

8. 8 SIGNAL MONOTONICITY neurons are nonlinear but not too much so ---- a property as semilinearity Linear signal functions - make computation and analysis comparatively easy - do not suppress noise - linear network are not robust Nonlinear signal functions - increases a network ’ s computational richness - increases a network ’ s facilitates noise suppression - risks computational and analytical intractability - favors dynamical instability NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

9. 9 SIGNAL MONOTONICITY Signal and activation velocities the signal velocity: =dS/dt Signal velocities depend explicitly on action velocities NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

10. 10 BIOLOGICAL ACTIVATIONS AND SIGNALS Fig.2 Neuron anatomy 神经元 (Neuron) 是由细胞核 (cell nucleus) ，细胞体 (soma) ，轴 突 (axon) ，树突 (dendrites) 和突触 (synapse) 所构成的 NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

11. 11 X= （ x1 ， x2 ， … ， xn ） W= （ w1 ， w2 ， … ， wn ） net=∑xiwi net=XW x 2 w 2 ∑f o=f （ net ） x n w n … net=XW x 1 w 1 BIOLOGICAL ACTIVATIONS AND SIGNALS NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

12. 12 BIOLOGICAL ACTIVATIONS AND SIGNALS Competitive Neuronal Signal logical signal function ( Binary  Bipolar ) The neuron “ wins ” at time t if, “ loses ” if and otherwise possesses a fuzzy win-loss status between 0 an 1. a. Binary signal functions : [0,1] b. Bipolar signal functions : [-1,1] McCulloch—Pitts (M—P) neurons NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

13. 13 NEURON FIELDS Neurons within a field are topologically ordered, often by proximity. zeroth-order topology : lack of topological structure Denotation:,, neural system samples the function m times to generate the associated pairs,..., The overall neural network behaves as an adaptive filter and sample data changed network parameters. NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

14. 14 NEURONAL DYNAMICAL SYSTEMS Description:a system of first-order differential or difference equations that govern the time evolution of the neuronal activations or membrane potentials Activation differential equations: in vector notation: (1) (2) (3) (4) NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

15. 15 NEURONAL DYNAMICAL SYSTEMS Neuronal State spaces So the state space of the entire neuronal dynamical system is: Augmentation: NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

16. 16 NEURONAL DYNAMICAL SYSTEMS Signal state spaces as hyper-cubes The signal state of field at time t: The signal state space: an n-dimensional hypercube The unit hypercube : or, The relationship between hyper-cubes and the fuzzy set :, subsets of correspond to the vertices of NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

17. 17 NEURONAL DYNAMICAL SYSTEMS Neuronal activations as short-term memory Short-term memory(STM) : activation Long-term memory(LTM) : synapse NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

18. 18 1 、 Liner Function S(x) = cx + k, c>0 SIGNAL FUNCTION (ACTIVATION FUNCTION) x S o k NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

19. 19 2. Ramp Function r if x≥θ S(x)= cx if |x|<θ -rif x≤-θ r>0, r is a constant. SIGNAL FUNCTION (ACTIVATION FUNCTION) r -r θ -θ x S NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

20. 20 SIGNAL FUNCTION (ACTIVATION FUNCTION) 3 、 threshold linear signal function: a special Ramp Function Another form: NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

21. 21 SIGNAL FUNCTION (ACTIVATION FUNCTION) 4 、 logistic signal function: Where c>0. So the logistic signal function is monotone increasing. NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

22. 22 SIGNAL FUNCTION (ACTIVATION FUNCTION) 5 、 threshold signal function: Where T is an arbitrary real-valued threshold,and k indicates the discrete time step. NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

23. 23 SIGNAL FUNCTION (ACTIVATION FUNCTION) 6 、 hyperbolic-tangent signal function: Another form: NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

24. 24 SIGNAL FUNCTION (ACTIVATION FUNCTION) 7 、 threshold exponential signal function: When, NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

25. 25 SIGNAL FUNCTION (ACTIVATION FUNCTION) 8 、 exponential-distribution signal function: When, NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

26. 26 SIGNAL FUNCTION (ACTIVATION FUNCTION) 9 、 the family of ratio-polynomial signal function: An example For, NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

27. 27 SIGNAL FUNCTION (ACTIVATION FUNCTION) NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

28. 28 SIGNAL FUNCTION (ACTIVATION FUNCTION) NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

29. 29 PULSE-CODED SIGNAL FUNCTION Definition: (5) (6) where (7) NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

30. 30 PULSE-CODED SIGNAL FUNCTION Pulse-coded signals take values in the unit interval [0,1]. Proof: when NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

31. 31 PULSE-CODED SIGNAL FUNCTION Velocity-difference property of pulse-coded signals The first-order linear inhomogenous differential equation: (8) (9) The solution to this differential equation: A simple form for the signal velocity: (10) (11) NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS (5)

32. 32 PULSE-CODED SIGNAL FUNCTION The central result of pulse-coded signal functions: The instantaneous signal-velocity equals the current pulse minus the current expected pulse frequency. ------------- the velocity-difference property of pulse- coded signal functions NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS (10)