1. Chapter 7 Introduction to Back Propagation Neural Networks BPNN
KH Wong
Neural Networks NN ver. 4h

2. Neural Networks NN ver. 4h
Introduction
Very Popular
A high performance Classifier (multi-class)
Successful in handwritten optical character OCR recognition, speech recognition, image noise removal etc.
Easy to implementation
Slow in learning
Fast in classification
Neural Networks NN ver. 4h

3. Neural Networks NN ver. 4h
Overview
Back Propagation Neural Networks (BPNN)
Part 1: Feed forward processing (classification or Recognition)
Part 2: Feed backward processing (Training the network), also include forward processing
Appendix:
A MATLAB example is explained
%source :
Neural Networks NN ver. 4h

4. Theory of Back Propagation Neural Net (BPNN)
Use many samples to train the weights (W) & Biases (b), so it can be used to classify an unknown input into different classes
Will explain
How to use it after training: forward pass (classify / or called recognize input )
How to train it: how to train the weights and biases (using forward and backward passes)
Neural Networks NN ver. 4h

5. Neural Networks NN ver. 4h
Motivation
Biological findings inspire the development of Neural Net
Input weights Logic function output
Biological relation
Input
Dendrites
Output
Neuron(Logic function)
X=inputs
W=weights
Output
Neural Networks NN ver. 4h

6. Optical character recognition OCR example
Training: Train the system first by presenting a lot of samples to the network
Recognition: When an image is input to the system, it will tell what character it is
Training up the network:
weights (W) and bias (b)
Neural Net
Neural Net
Output3=‘1’, other outputs=‘0’
Neural Networks NN ver. 4h

7. Part 1 (classification in action /or called Recognition process)
Forward pass of Back Propagation Neural Net (BPNN)
Assume weights (W) and bias (b) are found by training already (to be discussed in part2)
Neural Networks NN ver. 4h

8. Recognition: assume weight (W) bias (b) are found earlier
Output
Output0=0
Output1=0
Output2=0
Output3=1 :
Each pixel is X(u,v)
Neural Networks NN ver. 4h

9. Neural Networks NN ver. 4h
Neurons in BPNN
In side each neuron :
Inputs
Output neurons
Neural Networks NN ver. 4h

10. Multi-layer structure of a BP neural network
Input
layer
Neural Networks NN ver. 4h

11. Neurons in the Multi-layer structure
In between any neighboring 2 layers, a set of neurons can be found
Each Neuron
Neural Networks NN ver. 4h

12. Neural Networks NN ver. 4h
BPNN Forward pass
Forward pass is to find the output when an input is given. For example:
Assume we have used N=60,000 images to train a network to recognize c=10 numerals.
When an unknown image is given to the input, the output neuron corresponds to the correct answer will give the highest output level. 1
Input
image
10 output neurons for 0,1,2,..,9
Neural Networks NN ver. 4h

13. Neural Networks NN ver. 4h
Architecture (exercise: write formulas for A1(i=4) and A2(k=3) How many inputs, hidden neurons, outputs, weights in each layer?
P(j=1)
l=1(j=1,i=1) A1
 2(i=1,k=1)
A1(i=1)
Neuron i=1
Bias=b1(i=1)
 2(i=2,k=1)
Neuron k=1
Bias=b2(k=1)
P(j=2)
l=1(j=2,i=1) A2
 2(i=5,k=1)
P(j=9)
l=1(j=9,i=1) A5
A2(k=2)
 l=1(j=1,i=1)
P(j=1)
P(j=2)
P(j=3) :
P(j=9)
A1(i=1)
 l=2(i=1,k=1)
 l=2(i=2,k=1)
 l=1(j=2,i=1)
A1(i=2)
 l=2(i=2,k=2)
l=1(j=3,i=4)
 l=2(i=5,k=3)
A2:layer2, 3 Output neurons
indexed by k
Wl=2=5x3
bl=2=3x1
A1(i=5)
 l=1(j=9,i=5)
A1: Hidden layer1 =5 neurons, indexed by i
Wl=1=9x5
bl=1=5x1
Input:
P=9x1
Indexed by j
Layer l=2
Layer l=1
S2 generated
Neural Networks NN ver. 4h
S1 generated

14. Answer (exercise: write values for A1(i=4) and A2(k=3)
P=[ ]
Wl=1=[ ]
bl=1=
%Find A1(i=4)
A1_i_is_4=1/(1+exp[-(l=1*P+bl=1))]
=0.49
How many inputs, hidden neurons, outputs, weights and biases in each layer?
Answer: Inputs=9, hidden neurons=5, outputs=3, weights in hidden layer (layer1) =9x5, neurons in output layer (layer2)= 5x3, 5 biases in hidden layer (layer1), 3 biases in output layer (layer2)
Neural Networks NN ver. 4h

15. Numerical Example : Architecture of the example
Input
Layer
9x1 pixels
output
Layer 3x1
Neural Networks NN ver. 4h

16. Part 2: feed backward processing (Training the network)
Backward pass of Back Propagation Neural Net (BPNN) (Training)
Ref:
Neural Networks NN ver. 4h

17. Neural Networks NN ver. 4h
Feed backward stage
Part1:Feed
Forward (studied before)
Part2: Feed
backward
For training we need to find , why?
We will explain why and prove equations in the following slides
Neural Networks NN ver. 4h

18. The criteria to train a network
Based on the overall error function, there are ‘N’ samples and ‘c’ classes to be learned
Neural Networks NN ver. 4h

19. Neural Networks NN ver. 4h
Theory
Neuron j
wkj 19
K=1,2,..,n
n inputs to neuron j
Output of neuron j is yj
Neural Networks NN ver. 4h

20. Learning by gradient decent
Neural Networks NN ver. 4h

21. Neural Networks NN ver. 4h
We need to find , why?
Using Taylor series
Neural Networks NN ver. 4h

22. Neural Networks NN ver. 4h
neuron j
wij
Theory xi ui yj
Neural Networks NN ver. 4h

23. Case 1: if neuronj is at the output layer
We want to see
True
Target
Class=tj
wij xi
Output yj uj yj
Neuron j as an output neuron
Neural Networks NN ver. 4h

24. Neural Networks NN ver. 4h
Case2 : if neuronj is at the hidden layer Output yj affects all neurons connected to it in next layer
neuron j
Input xi to the hidden neuron j, P(:,) in program
For this hidden neuron j, this df1 in the program
Neural Networks NN ver. 4h

25. Neural Networks NN ver. 4h
After all are found
Neural Networks NN ver. 4h

26. Neural Networks NN ver. 4h
Training
How to train the neurons: how to train the weights (W) and biases (b) (use forward, backward passes)
Initialize W and b randomly
Iter=1: all_epochs (or break when E is very small)
For all training samples {
Forward pass (same as the recognition process in part1) for each output neuron:
Use training samples: Xclass_t : feed forward to find y.
E=error_function(y-t)
Backward pass:
From the output layer find  and b to reduce Error E. Find all s(of the output)
Calculate  and b of all hidden layers, }
Neural Networks NN ver. 4h

27. Neural Networks NN ver. 4h
Summary
Learn what is Back Propagation Neural Networks (BPNN)
Learn the forward pass
Learn the backward pass and the training of the BPNN network
Neural Networks NN ver. 4h

28. Neural Networks NN ver. 4h
References
Wiki
Matlab programs
Neural Network for pattern recognition- Tutorial
CNN Matlab example
Neural Networks NN ver. 4h

29. Neural Networks NN ver. 4h
Appendices
Neural Networks NN ver. 4h

30. Appendix 1:Sigmod function f(u) and its derivative f’(u)
Neural Networks NN ver. 4h

31. Derivation (for the output layer , in each neuron)
Alternative
Derivation (for the output layer , in each neuron)
Output
(last layer)
t=target (teacher)
y=output
Feeding back to the previous layer
Neural Networks NN ver. 4h

32. Neural Networks NN ver. 4h
derivation
Neural Networks NN ver. 4h

33. Neural Networks NN ver. 4h
BNPP example in matlab
Based on Neural Network for pattern recognition- Tutorial
Neural Networks NN ver. 4h

34. Neural Networks NN ver. 4h
Example: a simple BPNN
Number of classes (no. of output neurons)=3
Input 9 pixels: each input is a 3x3 image
Training samples =3 for each class
Number of hidden layers =1
Number of neurons in the hidden layer =5
Neural Networks NN ver. 4h

35. Display of testing patterns
Neural Networks NN ver. 4h

36. Neural Networks NN ver. 4h
Architecture
P(j=1)
l=1(j=1,i=1) A1
 2(i=1,k=1)
A1(i=1)
Neuron i=1
Bias=b1(i=1)
 2(i=2,k=1)
Neuron k=1
Bias=b2(k=1)
P(j=2)
l=1(j=2,i=1) A2
 2(i=5,k=1)
P(j=9)
l=1(j=9,i=1) A5
A2(k=2)
 l=1(j=1,i=1)
P(j=1)
P(j=2)
P(j=3) :
P(j=9)
A1(i=1)
 l=2(i=1,k=1)
 l=2(i=2,k=1)
 l=1(j=2,i=1)
A1(i=2)
 l=2(i=2,k=2)
l=1(j=3,i=4)
 l=2(i=5,k=3)
A2:layer2, 3 Output neurons
indexed by k
Wl=2=5x3
bl=2=3x1
A1(i=5)
 l=1(j=9,i=5)
A1: Hidden layer1 =5 neurons, indexed by i
Wl=1=9x5
bl=1=5x1
Input:
P=9x1
Indexed by j
Layer l=2
Layer l=1
S2 generated
Neural Networks NN ver. 4h
S1 generated

37. Neural Networks NN ver. 4h
%source :
clear memory %commented added by kh wong
clear all
clc
nump=3; % number of classes
n=3; % number of images per class
% training images reshaped into columns in P
% image size (3x3) reshaped to (1x9)
% training images
P=[ ; ...
; % class 1
; ...
; ...
; % class 2
; ...
; ...
; % class 3
]';
% testing images
N=[ ; ...
; % class 1
; ...
; ...
; % class 2
; ...
; ...
; % class 3
]';
% Normalization
P=P/256;
N=N/256;
Neural Networks NN ver. 4h

38. Neural Networks NN ver. 4h
% display the training images
figure(1),
for i=1:n*nump
im=reshape(P(:,i), [3 3]);
%remove theline below to reflect the truth data input
% im=imresize(im,20); % resize the image to make it clear
subplot(nump,n,i),imshow(im);…
title(strcat('Train image/Class #', int2str(ceil(i/n))))
end
% display the testing images
figure,
im=reshape(N(:,i), [3 3]);
% remove theline below to reflect the truth data input
% im=imresize(im,20); % resize the image to make it clear
subplot(nump,n,i),imshow(im);title(strcat('test image #', int2str(i)))
Neural Networks NN ver. 4h

39. Neural Networks NN ver. 4h
% targets
T=[ ];
S1=5; % numbe of hidden layers
S2=3; % number of output layers (= number of classes)
[R,Q]=size(P);
epochs = 10000; % number of iterations
goal_err = 10e-5; % goal error
a=0.3; % define the range of random variables
b=-0.3;
W1=a + (b-a) *rand(S1,R); % Weights between Input and Hidden Neurons
W2=a + (b-a) *rand(S2,S1); % Weights between Hidden and Output Neurons
b1=a + (b-a) *rand(S1,1); % Weights between Input and Hidden Neurons
b2=a + (b-a) *rand(S2,1); % Weights between Hidden and Output Neurons
n1=W1*P;
A1=logsig(n1); %feedforward the first time
n2=W2*A1;
A2=logsig(n2);%feedforward the first time
e=A2-T; %actually e=T-A2 in main loop
error =0.5* mean(mean(e.*e)); % better say e=T-A2 , but no harm to error here
nntwarn off
Neural Networks NN ver. 4h

40. Neural Networks NN ver. 4h
for itr =1:epochs
if error <= goal_err
break
else
for i=1:Q %i is index to a column in P(9x9), for each column of P ( P:,i)
% is a training sample image, 9 training samples, 3 for each class
%A1=5x9, A1 =outputs of hidden layer and input to output layer
% A2=3x9, A2=Outputs of output layer
%T=true class, each column in T is for 1 training sample
% hidden_layer =1, output_layer =2,
df1=dlogsig(n1,A1(:,i)); %df1 is 5x1 for 5 neurons in hidden layer
df2=dlogsig(n2,A2(:,i)); %df2 is 3x1 for output neurons
% s2 is sigma2=sensitvity2 from the output layer , equation(2)
s2 = -1*diag(df2) * e(:,i); %e=T-A2; df2=f’=f(1-f) of layer2
Neural Networks NN ver. 4h

41. Neural Networks NN ver. 4h
%s1=5x1
s1 = diag(df1)* W2'* s2; % eq(3),feedback, from s2 to S1
%dW= -n*s2*df(u)*x in ppt, =0.1, S2 is found, x is A1
%W2 is 3x5 , each output neuron receives, update W2
% 5 inputs from 5 hidden neurons in the hidden layer
%sigma2=s2 = -1*diag(df2) * e(:,i); %e=T-A2; df2=f’=f(1-f) of layer2
%delta_W2 = -learning_rate*sigma2*input_to_output_layer
%delta_W2 = -0.1*sigma2*A1
W2 = W2-0.1*s2*A1(:,i)'; %learning rate=0.1, equ(2) output case
%3x5 =3x5- (3x1*1x5),
%A1=5 hidden neuron outputs (5 hidden neurons)
%A1(:,i)’=1x5=outputs of hidden layer,
b2 = b2-0.1*s2; %threshold
% 3x1=3x1- 3x1
%P1(:,i)=1x9 =input t o hidden,
% s1=5x1 because each hidden note has 1 sensitivity (sigma)
W1 = W1-0.1*s1*P(:,i)';% update W1 in layer 1, see equ(3) hidden case
%5x9 = 5x9-(5x1* 1x9), since P is 9x9 and for an i, P(:,i)' =1x9
Neural Networks NN ver. 4h

42. Neural Networks NN ver. 4h
b1 = b1-0.1*s1;%threshold
%5x1=5x1-5x1
A1(:,i)=logsig(W1*P(:,i)+b1);%forward
%5x1 = 5x1
A2(:,i)=logsig(W2*A1(:,i)+b2);%forward
%3x1=3x1
end
e = T - A2; % for this e, put -ve sign for finding s2
error =0.5*mean(mean(e.*e));
disp(sprintf('Iteration :%5d mse :%12.6f%',itr,error));
mse(itr)=error;
Neural Networks NN ver. 4h

43. Neural Networks NN ver. 4h
threshold=0.9; % threshold of the system (higher threshold = more accuracy)
% training images result
%TrnOutput=real(A2)
TrnOutput=real(A2>threshold)
% applying test images to NN , TESTING BEGINS HERE
n1=W1*N;
A1=logsig(n1);
n2=W2*A1;
A2test=logsig(n2);
% testing images result
%TstOutput=real(A2test)
TstOutput=real(A2test>threshold)
% recognition rate
wrong=size(find(TstOutput-T),1);
recognition_rate=100*(size(N,2)-wrong)/size(N,2)
% end of code
Neural Networks NN ver. 4h