1. NEURAL NETWORK THEORY NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS 2004.10.13

2. ACTIVATIONS AND SIGNALS NEURONS AS FUNCTIONS SIGNAL MONOTONICITY BIOLOGICAL ACTIVATIONS AND SIGNALS NEURON FIELDS NEURONAL DYNAMICAL SYSTEMS COMMON SIGNAL FUNCTION PULSE-CODED SIGNAL FUNCTION

3. NEURONS AS FUNCTIONS Neurons behave as functions. Neurons transduce an unbounded input activation x(t) at time t into a bounded output signal S(x(t)).

4. NEURONS AS FUNCTIONS S(x) x 0 -∞-++∞ Fig.1 s(x) is a bounded monotone-nondecreasing function of x If c → +∞ ， we get threshold signal function (dash line), Which is piecewise differentiable

5. NEURONS AS FUNCTIONS The transduction description: a sigmoidal or S-shaped curve the logistic signal function: The logistic signal function is sigmoidal and strictly increases for positive scaling constant c >0.

6. NEURONS AS FUNCTIONS S(x) x 0 -∞-++∞ Fig.1 s(x) is a bounded monotone-nondecreasing function of x If c → +∞ ， we get threshold signal function (dash line), Which is piecewise differentiable

7. NEURONS AS FUNCTIONS S would transduce the four-neuron vector of activations (-6 350 49 – 689) to the four- dimensional bit vector of signal (0 1 1 0) Zero activations to unity,zero,or the previous signal

8. SIGNAL MONOTONICITY In general, signal functions are monotone nondecreasing S’>=0. This means signal functions have an upper bound or saturation value. S(x) x 0 -∞-++∞

9. SIGNAL MONOTONICITY An important exception: bell-shaped signal function or Gaussian signal functions The sign of the signal-activation derivation s’ is opposite the sign of the activation x. We shall assume signal functions are monotone nondecreasing unless stated otherwise.

10. SIGNAL MONOTONICITY Generalized Gaussian signal function define potential or radial basis function : input activation vector: variance: mean vector: we shall consider only scalar-input signal functions:

11. SIGNAL MONOTONICITY A property of signal monotonicity: semi-linearity Comparation: a. Linear signal functions: computation and analysis is comparatively easy; do not suppress noise. b. Nonlinear signal functions: increases a network ’ s computational richness and facilitates noise suppression; risks computational and analytical intractability;

12. SIGNAL MONOTONICITY Signal and activation velocities the signal velocity: =dS/dt Signal velocities depend explicitly on action velocities

13. BIOLOGICAL ACTIVATIONS AND SIGNALS Fig.2 Neuron anatomy 神经元 (Neuron) 是由细胞核 (cell nucleus) ，细胞体 (soma) ，轴 突 (axon) ，树突 (dendrites) 和突触 (synapse) 所构成的

14. BIOLOGICAL ACTIVATIONS AND SIGNALS Dendrites Synapse Axon

15. BIOLOGICAL ACTIVATIONS AND SIGNALS X= （ x1 ， x2 ， … ， xn ） W= （ w1 ， w2 ， … ， wn ） net=∑xiwi net=XW x 2 w 2 ∑f o=f （ net ） x n w n … net=XW x 1 w 1

16. BIOLOGICAL ACTIVATIONS AND SIGNALS Competitive Neuronal Signal logical signal function ( Binary  Bipolar ) The neuron “ wins ” at time t if, “ loses ” if and otherwise possesses a fuzzy win-loss status between 0 an 1. a. Binary signal functions : [0,1] b. Bipolar signal functions : [-1,1] McCulloch—Pitts (M—P) neurons

17. NEURON FIELDS Neurons within a field are topologically ordered, often by proximity. In the simplest case, neuron are not topologically. (zeroth-order topology)

18. NEURON FIELDS Denotation:,, neural system samples the function m times to generate the associated pairs,..., The overall neural network behaves as an adaptive filter and sample data changed network parameters.

19. NEURON FIELDS

20. NEURONAL DYNAMICAL SYSTEMS Description:a system of first-order differential or difference equations that govern the time evolution of the neuronal activations or membrane potentials Activation differential equations: in vector notation: (1) (2) (3) (4)

21. NEURONAL DYNAMICAL SYSTEMS Autonomous Nonautonomous automomous systems are usually easier to analyze than nonautomous systems.

22. NEURONAL DYNAMICAL SYSTEMS Neuronal State spaces So the state space of the entire neuronal dynamical system is: Augmentation:

23. NEURONAL DYNAMICAL SYSTEMS Signal state spaces as hypercubes The signal state of field at time t: The signal state space: an n-dimensional hypercube The unit hypercube : or, The relationship between hyper-cubes and the fuzzy set :, subsets of correspond to the vertices of

24. NEURONAL DYNAMICAL SYSTEMS Neuronal activations as short-term memory Short-term memory(STM) : activation Long-term memory(LTM) : synapse

25. COMMON SIGNAL FUNCTION 1 、 Liner Function S(x) = cx + k, c>0 x S o k

26. COMMON SIGNAL FUNCTION 2. Ramp Function r if x≥θ S(x)= cx if |x|<θ -rif x≤-θ r>0, r is a constant. r -r-r θ -θ -θ x S

27. COMMON SIGNAL FUNCTION 3 、 threshold linear signal function: a special Ramp Function Another form:

28. COMMON SIGNAL FUNCTION 4 、 logistic signal function: Where c>0. So the logistic signal function is monotone increasing.

29. COMMON SIGNAL FUNCTION 5 、 threshold signal function: Where T is an arbitrary real-valued threshold,and k indicates the discrete time step.

30. COMMON SIGNAL FUNCTION 6 、 hyperbolic-tangent signal function:

31. COMMON SIGNAL FUNCTION 7 、 threshold exponential signal function: When,

32. COMMON SIGNAL FUNCTION 8 、 exponential-distribution signal function: When,

33. COMMON SIGNAL FUNCTION 9 、 the family of ratio-polynomial signal function: An example For,

34. PULSE-CODED SIGNAL FUNCTION Definition: where

35. PULSE-CODED SIGNAL FUNCTION Pulse-coded signals take values in the unit interval [0,1]. Proof: when

36. PULSE-CODED SIGNAL FUNCTION Velocity-difference property of pulse-coded signals The first-order linear inhomogenous differential equation: The solution to this differential equation: A simple form for the signal velocity:

37. PULSE-CODED SIGNAL FUNCTION The central result of pulse-coded signal functions: The instantaneous signal-velocity equals the current pulse minus the current expected pulse frequency. ------------- the velocity-difference property of pulse- coded signal functions

38. End Thank you ! Li Wei 2004.10.13