1. 1 Neural networks 2

2. 2 Introduction: Neural networks The nervous system contains 10^12 interconnected neurons.

3. 3 Introduction: Neural networks Dr. Robert Hecht-Nielsen defines a neural network as: “...a computing system made up of a number of simple, highly interconnected processing elements, which process information by their dynamic state response to external inputs.”

4. 4 History and applications Classification Divided objects into different classes Quantitative data --> qualitative information Pattern recognition, language recognition, etc. Operations Research Solve hard problems Associative memory Reproduce data based on incomplete and /or noisy information. Machine learning

5. 5 Artificial neuron model

6. 6 We divide the neural network into three phases: architecture – represents a structure of the network, how its neurons are connected. It can be simply imagined as a view on the network from outside. active phase It describes the inwards of the network – what happens from the moment, when the input enters the network till the computations reach its output. adaptation – is a networks reaction on the ongoing computations. It denotes alterations of the neurons weights.

7. 7 Learning The idea is to take a large number of handwritten digits, known as training and then develop a system which can learn from those training examples.

8. 8 Learning Learning is a phase of the development of a neural network in which the behavior of the network is altered until getting the desired behavior. It consists of changing the network weights to give the network response to an examples and experiences. It is difficult to decide values ​​ of the weights connection in neural network for a given application. There are two main classes of learning algorithms: –Supervised learning. –Unsupervised Learning.

9. 9 Supervised learning supervisor Network desired results Obtained results Error Inputs

10. 10 Unsupervised Learning Network Obtained network Inputs

11. 11 Learning rules Learning means changing the weights of connections between neurons. There are many rules to change to alter the weights: –Hebb :  w ij (t) = Rx i.x j –Widrow-Hoff (delta rule) :  w ij =R(d - a j )a i –Grossberg :  w ij =R(a j - w ij )a i ij W ij

12. 12 Hebb rule A change in the strength of a connection is a function of the pre – and post-synaptic neural activities. It is called the “hebbian learning rule” after D. Hebb (“When neuron A repeatedly participates in firing neuron B, the strength of the action of A onto B increases”.) x i x j δw ij 0 0 0 0 1 0 1 0 0 1 1 + A more general form of a Hebbian learning rule would be: w ij (t+1) = w ij (t) +δw ij (t) (New) Where δw ij (t) = x i.x j

13. 13 Hebb rule limitation if the set of input patterns are not mutually orthogonal, interference may occur and the network may not be able to learn associations. This limitation of Hebbian learning (unsupervised) can be overcome by using Perceptron algorithm and the Delta rule.

14. 14 Delta rule Widrow-Hoff (1960) proposes the delta rule. The delta rule is a local rule, i.e. each output cell learns independently of the other cells. An output cell changes the links’ weights only : –If it is active whereas it should be passive then it decreases the weights of active retina inputs. –If it is passive whereas it should be active then it increases the weights of active retina inputs.

15. 15 Delta rule a i = 0a i = 1 d = 0  W ij = 0  W ij = -R d = 1  W ij = R  W ij = 0 a i activation produced by the network. d i desired result (by humain expert). For example: if the output is lower than the desired result, we increase the weight of the link when the output of neurone (j) is equal to 1 (we suppose that we use binary units {0,1}).

16. 16 Delta rule We write the Delta rule (Widrow-Hoff rule) as follows: w i,j (t + 1) = w i,j (t) + R * (d i - a i ) * a j = w i,j (t) +  w i,j –w i,j (t + 1) = new weight of i -> j link at time (t + 1) –w i,j (t) = weight of link i -> j at time (t) –a j = output (0 ou 1) of retina cell j (output of the input neurone j) –a i = response of output cell i (output result) –d i = Desired response (desired result) of output cell i. –R = positive constant (between 0.0 and 1.0):

17. 17 Perceptron The Perceptron is a classic learning algorithm for the neural model of learning. It is an algorithm for supervised classification of an input into one of several possible non-binary outputs. It is a type of linear classifier, i.e. a classification algorithm that makes its predictions based on a linear predictor function combining a set of weights with the feature vector. It consists of two neural layers entirely interconnected: The first layer, called retina consisting of binary cell (OUT = 0 ou 1). 1)The second layer provide the response (the result). 2)The learning is supervised and the weights are altered using the Delta rule.

18. 18 Supervised learning Perceptron Retina output input

19. 19 Perceptron For example if we have 5 * 6 input cells and 10 outputs to recognize a digital (among 0,1,2,3,4,5,6,7,8,9). If digital 4 is the input, the fifth output cell is active and the other outputs must be passive. All the input cells are interconnected to each output cell with a different weight. If the output of retina cells are indexed by j (ex: from 0 to 29) and output cells are indexed by i (ex: from 0 to 9), the entry a j of the output cell (j) is weighted sum of all output of the retina cells (i).

20. 20 Perceptron Optic input retina image Three Output Retina e6 e5 e4 e3 e2 e1 Wij*ei

21. 21 Perceptron a j = the input of the output cell j. e i = the output (0 ou 1) of retina cell i. w i,j = weight of link i-> j. The threashold function allow to determine the output of the cell j: We can replace the threshold S by a thereshold equal to 0 ( we add new input cell always active with a weight equal to S.

22. 22 Perceptron learning algorithm 1.Initialize the weights and threshold value S to a small random values. 2.Introduce the combination E = (e 1,... e n ) to the learning base. 3.Compute the output entry aj (calculated by the inner potential or the weighted sum) and the output x for this entry: –a = Σ(w i. e i ) - S –x = sign (a) (si a>0 alors x = +1 sinon a≤0 alors x = -1) 4.If the output x of Perceptron is different than the desired result d i than for this combination E we modify the weight (μ is the learning rate): w i (t+1) = w i (t) + μ.((d - x).e i ) –Note: d = +1 if E belongs to class1, d = -1 if E belongs to class 2 and (d - x) is the error estimation. 5.Repeat untill all combinations are processed (i.e. repeat from step 2)

23. Perceptron learning algorithm Example of Perceptron learning algorithm : Learning base example is below: e1e1 e2e2 d 111(1) 1 (2) (3) 1 (4) Initial conditions : w 1 = -0.2, w 2 = +0.1, S = 0.2, (μ = +0.1)

24. Perceptron learning algorithm Example 1.Initiales conditions : w 1 = -0.2, w 2 = +0.1, S = 0.2, (μ = +0.1) 2.a = Σ(w i. e i ) – S a(1) = (-0.2x1) + (0.1x1) - 0.2 = -0.3 a≤0 then x = -1 3.x = -1 (The desired output d = +1), hence, weighth modification: 4.w i (t+1) = w i (t) + μ.((d i – x i ).e i ) w 1 = -0.2 +[0.1(1-(-1)).(+1)] = 0 w 2 = 0.1+[0.1(1-(-1)).(+1)] = 0.3 e1e1 e2e2 d 111(1) 1 (2) (3) 1 (4) Back to step 2 X

25. Perceptron learning algorithm Example 1.Initiales conditions : w 1 = 0, w 2 = 0.3, S = 0.2, (μ= +0.1) 2.a = Σ(w i. e i ) – S a(1) = (0x1) + (0.3x1) - 0.2 = 0.1 a>0 then x = 1 We pass to the second combinaison: 2.a = Σ(w i. e i ) – S a(2) = (0x-1) + (0.3x1) - 0.2 = 0.1 a>0 then x = 1 3.x 2 =1 et d 2 =-1 then we modifie the weight. 4.w i (t+1) = w i (t) + μ.((d 2 – x 2 ).e i ) w 1 = 0 +[0.1(-1-1).(-1)] = 0.2 w 2 = 0.3+[0.1(-1-1).(+1)] = 0.1 e1e1 e2e2 d 111(1) 1 (2) (3) 1 (4) Back to step 2 e1e1 e2e2 d 111(1) 1 (2) (3) 1 (4) X

26. Perceptron learning algorithm Exemple 1.Initiales conditions : w 1 = 0.2, w 2 = 0.1, S = 0.2, (μ= +0.1) 2.a = Σ(w i. e i ) – S a(1) = (0.2x1) + (0.1x1) - 0.2 = 0.1 a>0 then x = 1 We pass to the next combinaison: 2.a = Σ(w i. e i ) – S a(2) = (0.2x-1) + (0.1x1) - 0.2 = -0.3 a≤0 then x = -1 We pass to the next combination: 2.a = Σ(w i. e i ) – S a(3) = (0.2x-1) + (0.1x-1) - 0.2 = -0.5 a≤0 then x = -1 We pass to the next combination: 2.a = Σ(w i. e i ) – S a(4) = (0.2x1) + (0.1x-1) - 0.2 = -0.1 a≤0 then x = -1 e1e1 e2e2 d 111(1) 1 (2) (3) 1 (4) e1e1 e2e2 d 111(1) 1 (2) (3) 1 (4) e1e1 e2e2 d 111(1) 1 (2) (3) 1 (4) e1e1 e2e2 d 111(1) 1 (2) (3) 1 (4)

27. Perceptron learning algorithm The final solution using perceptron learning algorithm: Learning base : Perceptron fullfilled the input space partitio n into 2 classes (1 et 2) depending on the value of the output (+1 ou -1). e1e1 e2e2 d 111(1) 1 (2) (3) 1 (4) Final conditions : w 1 = 0.2, w 2 = 0.1, S = 0.2, (μ = +0.1)

28. Perceptron learning algorithm The separation of the two zones is performed using hyperplane. The equation of the separation line is given by: w 1.e 1 +w 2.e 2 -S = 0 The input partition of dimension 2 achieved by Perceptron behive as the binary AND. The 4 combinations of the learning base represent 4 square’s edges. Perceptron parameters are : w1 = 0.2, w2 = 0.1 et S = 0.2. e1e1 e2e2 d 111(1) 1 (2) (3) 1 (4) Classe 1 Classe 2

29. Perceptron learning algorithm Notes: It is possible to consider that the threshold S is a weight for input cell. The value of the threshold S is altered with the other weights, this makes the convergence of the algorithm to the final solution more rapid (if the solution exists). We call one iteration of learning when we pass all the combinations one times into the learning algorithm.

30. 30 Perceptron learning algorithm Exercices Question1 : Use Hebb rule to solve the example below: Find the weights values. Initial conditions (w1 = w2 = w3 = w4 = S = 0) Discussion!!! e1e1 e2e2 e3e3 e4e4 xdxd 11 1(1) 11111(2) 111 (3) 1 1 (4)

31. Perceptron learning algorithm Exercices: Question2 : Use Perceptron learning algorithm to solve the previous question. (Do not change S for this quastion). Solution: w 1 = -0.2, w 2 = -0.2, w 3 = 0.6, w 4 = 0.2 Question3 : Find the weight to learn the binary OR Initial condition: w 1 = -0.2, w 2 = +0.1, S = 0.2, (μ = +0.1) e1e1 e2e2 d 111(1) 11(2) (3) 11(4)

32. 32 Perceptron learning algorithm limitation Perceptron is unable to distinguish patterns which are nor laniary separable. input P0 1 0 1 input Q1 0 0 1 AND0 0 0 1 XOR1 1 0 0 Binary AND Binary XOR lineary separable non-lineary separable

33. 33 Next Perceptron multi-layer.