1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 23 Nov 2, 2005 Nanjing University of Science & Technology

2 Happy Halloween

3 Lecture 23 Topics 1.Review the Backpropagation Algorithm 2.Give Preliminary Assignments for design of Artificial Neural Networks. 3.Discuss the flow diagram for Backpropagation algorithm in more detail. 4. Answer Frequently Asked Questions (FAQ) 5. Specify some Guidelines for selection of parameters and structure

4 Backpropagation Training Algorithm for Feedforward Neural networks

5 Input pattern sample x k

6 Calculate Outputs First Layer

7 Calculate Outputs Second Layer

8 Calculate Outputs Last Layer

9 Check Performance E TOTAL (p)  ½  (d[x(p-i)] – f( w T (p-i)  x(p-i) ) 2 i = 0 N s - 1 E TOTAL (p+1) = E TOTAL (p) + E p+1 (p+1) – E p-Ns (p-N s ) Single Sample Error Over all Samples Error Can be computed recursively

10 Change Weights Last Layer using Rule #1

11 Change Weights previous Layer using Rule #2

12 Change Weights previous Layer using Modified Rule #2

13 Input pattern sample x k+1 Continue Iterations Until

14 Repeat process until performance is satisfied or maximum number of iterations are reached. If performance not satisfied at maximum number of iterations the algorithm stops and NO design is obtained. If performance is satisfied then the current weights and structure provide the required design.

15 Freeze Weights to get Acceptable Neural Net Design

16 Definitions Convergence: Occurs when the tolerance parameter has been satisfied thus giving an acceptable design. Failure to Converge: Occurs when the tolerance level is not satisfied in N MAX iterations thus giving no design. Lock UP: Occurs when the algorithm produces saturated outputs that will only allow very small corrections to the weights.

17 (a) Select a feedforward neural network structure- Number of layers, Number of nodes per layer, types of nonlinearities (b) Select learning and momentum parameters acceptable error, maximum number of iterations (c) Construct a Training set with correct classification and normalize samples (d) Select Training type – Supervised, By Sample or By Epoch (e) Select Target values Design Preliminary Assignments

18 Discussion of Backpropagation Training Algorithm for Feedforward Neural Networks Training samples with classification Computer Flow Diagram Further Discussion of each step

19 Training Set: { ( x (k), d x (k) ), k=1,2,…,N s } Training Patterns Desired output for each training pattern

20 Backpropagation Algorithm for Training Feedforward Artificial Neural Networks Flow Chart

21 (Step 0) Initialization Randomly assign the weights to all Layers: (Method 1) Weights selected from [-1, 1] (Method 2) Ng-Widrow Procedure Set the iteration number p=1 and go to Step(1)

22 (Step 1) Select one of the training samples Method 1- Fixed order: Define order that the training samples are to be drawn and keep order through successive samples and all passes through data. Method 2- Multiple Fixed orders: Use different specific ordering through different passes through data Method 3- Fully Random Order. Randomly select order through each pass through the data Using one of the methods above select a training sample x(p) which includes the pattern vector and the class assignment. Then go to Step (2)

23 (Step 2) Calculate outputs of each layer Using selected sample x(p) and weights matrices for each layer W (k) (p), k=1 to N L calculate outputs y (1) (p) of all nodes in layer 1, and then calculate all outputs in successive layers y (k) (p), k = 2, 3, …, N L. Store all nodal outputs in memory. Then Go To step(3)

24 E TOTAL (p)  ½  (d[x(p-i)] – f( w T (p-i)  x(p-i) ) 2 i = 0 N s - 1 Single Sample Error Over all Samples Error (Step 3) Calculation of performance Usually use the following Approximation to reduce computational complexity

25 E TOTAL (p+1) = E TOTAL (p) + E p+1 (p+1) – E p-Ns (p-N s ) This approximation can be computed recursively taking care to avoid roundoff errors by When error gets small this approximation becomes pretty good! Then GO To Step(4)

26 (Step 4) Branch on Error Tolerance If performance not satisfied, E TOTAL (p) > ε (acceptable total error) Then GO TO Step(5) If performance is satisfied then the current weights and structure provide the required design and the algorithm is stopped.

27 If p = N MAX then algorithm stops and NO acceptable design is obtained (Step 5) Branch on Maximum number of Itterations If p < N MAX ( This does not mean that an acceptable design is not possible, only that it was not found). Then GO TO Step(6) (algorithm continues)

28 (Step 6) Calculation of weight changes using weight update equations Calculate the weight changes and new weights for the last (L)th layer using the outputs from step (3) and the current input sample x(p) using Rule #1 for each nonlinearity. Continue calculating new weights in reverse order for preceding L-1 Layers to the first layer using Rule #2 and modified Rule #2 If training by epoch, the weight changes will be accumulated for each sample though one entire pass through the data before changing weights Go to Step (7)

29 (Step 7) Update the weight matrices Update all weight matrices for all layers to get new weight matrices W (k) (p+1) Then go to step (1)

30 Questions Concerning Neural Net Design 1.How do we select the Training parameters and structure of NN? See later discussion. 2. Why do we Normalize the Training Samples? To avoid saturation of nonlinarities. 3. What effect does sample order have on training? Can possibly speed up convergence.

31 4. Should we train by epoch or by sample? Training by sample may speed up convergence ( but not significantly). 5. How do we select the Training Pattern set? Training set patterns should be representative of all patterns in each of the classes.

32 Guidelines for the “Art” of Neural Net Design

33 Guidelines for Neural Net Design 1. Choice of training parameter η usually 0 < η < 10 η too big could cause lockup higher η can give faster training Nominal value η = 0.8

34 Guidelines for Neural Net Design 2. Choice of momentum parameter m usually 0 < m < 1 m to big could cause lockup higher m can give faster training Nominal value m = 0.3

35 Guidelines for Neural Net Design 3. Choice of activation functions for neuron nonlinearities: Tanh, Logistic, Linear, Step Function, Signum function, continuous, discrete Bipolar continuous usually gives faster design with saturation at 1 and -1 Design relatively insensitive to type of nonlinearity. Choose Tanh nonlinearities

36 Guidelines for Neural Net Design 4. Choice of Maximum number of Iterations N MAX usually 1000 < N MAX < 10 8 No harm in selecting as large as possible to give time for the algorithm to converge. However too large of a value could mask unsuitable design parameters wasting resources and time! Nominal value N MAX = 10,000

37 Guidelines for Neural Net Design 5. Choice of error tolerance E TOT usually < E TOT < 0.5 E TOT too big could cause poor design and give poor generalization E TOT to small can result in excessive number of iterations and “grandmothering” Nominal value E TOT = 0.05

38 Guidelines for Neural Net Design 6. Choice of number of Layers L usually 0 < L < 5 L too big can cause slow convergence or no convergence and result in an unnecessarily complex design if it converges. L to small may not give enough flexibility for the required design and thus not converge. Nominal value L = 3

39 Approximation theory theorems imply that two layers are all that is needed but the theorems do not give the nonlinear functions only that they exist. Our results on Hyperplane-AND-OR structure gives us reason to believe that maybe three layers are sufficient. 6. Choice of number of Layers L. (cont) Guidelines for Neural Net Design

40 Guidelines for Neural Net Design 7. Choice of number nodes in each Layer usually N L = 1 or N C (last layer) Use N L = N C for large number of classes(N C >6) No need to use any more than the number of classes. Use L=1 for small number of classes. This gives enough dynamic range to discriminate between output values(sensitivity). Nominal value L = 1 or N C

41 8. Selection of Target values Two class case- single output neuon Bipolar Activation Nonlinearity Unipolar Activation Nonlinearity Guidelines for Neural Net Design

42 Selection of Target values (Continued) Two class case- two output neurons Bipolar Activation Nonlinearity Unipolar Activation Nonlinearity 0.1

43 Selection of Target values (Continued) K class case- one output neuron t i selected as center of R i

44 Selection of Target values (Continued) K class case- K output neurons Bipolar Activation Nonlinearity Decision Rule

45 Summary Lecture 23 1.Review the Backpropagation Algorithm 2.Give Preliminary Assignments for design of Artificial Neural Networks. 3.Discuss the flow diagram for Backpropagation algorithm in more detail. 4. Answer Frequently Asked Questions (FAQ) 5. Specify some Guidelines for selection of parameters and structure

46 A few Jack-O-Lanterns

47 End of Lecture 23