1. Backpropagation Learning Algorithm

2. However, we will concentrate on nets with units arranged in layersx1 xn
The backpropagation algorithm was used to train the multi layer perception MLP
MLP used to describe any general Feedforward (no recurrent connections) Neural Network FNN
However, we will concentrate on nets with units arranged in layers

3. Architecture of BP Nets
Multi-layer, feed-forward networks have the following characteristics:
-They must have at least one hidden layer
Hidden units must be non-linear units (usually with sigmoid activation functions)
Fully connected between units in two consecutive layers, but no connection between units within one layer.
For a net with only one hidden layer, each hidden unit receives input from all input units and sends output to all output units
Number of output units need not equal number of input units
Number of hidden units per layer can be more or less than input or output units

4. Other Feedforward Networks
Madaline
Multiple adalines (of a sort) as hidden nodes
Adaptive multi-layer networks
Dynamically change the network size (# of hidden nodes)
Networks of radial basis function (RBF)
e.g., Gaussian function
Perform better than sigmoid function (e.g., interpolation in function approximation

5. Introduction to Backpropagation
In 1969 a method for learning in multi-layer network, Backpropagation (or generalized delta rule) , was invented by Bryson and Ho.
It is best-known example of a training algorithm. Uses training data to adjust weights and thresholds of neurons so as to minimize the networks errors of prediction.
Slower than gradient descent .
Easiest algorithm to understand
Backpropagation works by applying the gradient descent rule to a feedforward network.

6. How many hidden layers and hidden units per layer?
Theoretically, one hidden layer (possibly with many hidden units) is sufficient for any L2 functions
There is no theoretical results on minimum necessary # of hidden units (either problem dependent or independent)
Practical rule :
n = # of input units; p = # of hidden units
For binary/bipolar data: p = 2n
For real data: p >> 2n
Multiple hidden layers with fewer units may be trained faster for similar quality in some applications

7. Training a BackPropagation Net
Feedforward training of input patterns
each input node receives a signal, which is broadcast to all of the hidden units
each hidden unit computes its activation which is broadcast to all output nodes
Back propagation of errors
each output node compares its activation with the desired output
based on these differences, the error is propagated back to all previous nodes Delta Rule
Adjustment of weights
weights of all links computed simultaneously based on the errors that were propagated back

8. Three-layer back-propagation neural network

9. Generalized delta rule
Delta rule only works for the output layer.
Backpropagation, or the generalized delta rule, is a way of creating desired values for hidden layers

10. Description of Training BP Net: Feedforward Stage
1. Initialize weights with small, random values
2. While stopping condition is not true
for each training pair (input/output):
each input unit broadcasts its value to all hidden units
each hidden unit sums its input signals & applies activation function to compute its output signal
each hidden unit sends its signal to the output units
each output unit sums its input signals & applies its activation function to compute its output signal

11. Training BP Net: Backpropagation stage
3. Each output computes its error term, its own weight correction term and its bias(threshold) correction term & sends it to layer below
4. Each hidden unit sums its delta inputs from above & multiplies by the derivative of its activation function; it also computes its own weight correction term and its bias correction term

12. Training a Back Prop Net: Adjusting the Weights
Each output unit updates its weights and bias
6. Each hidden unit updates its weights and bias
Each training cycle is called an epoch. The weights are updated in each cycle
It is not analytically possible to determine where the global minimum is. Eventually the algorithm stops in a low point, which may just be a local minimum.

13. How long should you train?
Goal: balance between correct responses for training patterns & correct responses for new patterns (memorization v. generalization)
In general, network is trained until it reaches an acceptable error rate (e.g. 95%)
If train too long, you run the risk of overfitting

14. Graphical description of of training multi-layer
neural network using BP algorithm
To apply the BP algorithm to the following FNN

15. To teach the neural network we need training data set
To teach the neural network we need training data set. The training data set consists of input signals (x1 and x2 ) assigned with corresponding target (desired output) z.
The network training is an iterative process. In each iteration weights coefficients of nodes are modified using new data from training data set.
After this stage we can determine output signals values for each neuron in each network layer.
Pictures below illustrate how signal is propagating through the network, Symbols w(xm)n represent weights of connections between network input xm and neuron n in input layer. Symbols yn represents output signal of neuron n.

17. Propagation of signals through the hidden layer
Propagation of signals through the hidden layer. Symbols wmn represent weights of connections between output of neuron m and input of neuron n in the next layer.

18. Propagation of signals through the output layer.
In the next algorithm step the output signal of the network y is compared with the desired output value (the target), which is found in training data set. The difference is called error signal d of output layer neuron.

19. It is impossible to compute error signal for internal neurons directly, because output values of these neurons are unknown. For many years the effective method for training multilayer networks has been unknown.
Only in the middle eighties the backpropagation algorithm has been worked out. The idea is to propagate error signal d (computed in single teaching step) back to all neurons, which output signals were input for discussed neuron.

20. The weights' coefficients wmn used to propagate errors back are equal to this used during computing output value. Only the direction of data flow is changed (signals are propagated from output to inputs one after the other). This technique is used for all network layers. If propagated errors came from few neurons they are added. The illustration is below:

21. When the error signal for each neuron is computed, the weights coefficients of each neuron input node may be modified. In formulas below df(e)/de represents derivative of neuron activation function (which weights are modified).

24. Coefficient h affects network teaching speed
Coefficient h affects network teaching speed. There are a few techniques to select this parameter. The first method is to start teaching process with large value of the parameter. While weights coefficients are being established the parameter is being decreased gradually.
The second, more complicated, method starts teaching with small parameter value. During the teaching process the parameter is being increased when the teaching is advanced and then decreased again in the final stage.

25. Training Algorithm 1
Step 0: Initialize the weights to small random values
Step 1: Feed the training sample through the network and determine the final output
Step 2: Compute the error for each output unit, for unit k it is:
Actual output
dk =
(tk – yk)f’(y_ink)
Required output
Derivative of f

26. Training Algorithm 2 Dwjk = hdkzj
Step 3: Calculate the weight correction term for each output unit, for unit k it is:
Hidden layer signal
Dwjk = hdkzj
A small constant

27. S Training Algorithm 3 d_inj = dkwjk
Step 4: Propagate the delta terms (errors) back through the weights of the hidden units where the delta input for the jth hidden unit is:
d_inj = dkwjk S
k=1 m
The delta term for the jth hidden unit is:
dj = d_injf’(z_inj)
where
f’(z_inj)= f(z_inj)[1- f(z_inj)]

28. Training Algorithm 4 Dwij = hdjxi wjk(new) = wjk(old) + Dwjk
Step 5: Calculate the weight correction term for the hidden units:
Step 6: Update the weights:
Step 7: Test for stopping (maximum cylces, small changes, etc)
Dwij = hdjxi
wjk(new) = wjk(old) + Dwjk

29. Options
There are a number of options in the design of a backprop system
Initial weights – best to set the initial weights (and all other free parameters) to random numbers inside a small range of values
(say: – 0.5 to 0.5)
Number of cycles – tend to be quite large for backprop systems
Number of neurons in the hidden layer – as few as possible

30. Example
The XOR function could not be solved by a single layer perceptron network
The function is:
X Y F

31. XOR Architecture x y f
v21 S
v11
v31
v22
v12
v32
w21
w11
w31 1

32. Initial Weights Randomly assign small weight values: f S 1 x y -.3 .21
.15
-.4
.25 .1
-.2 .3 1

33. Feedfoward – 1st Pass f S y_in1 = -.3(1) + .21(0) + .25(0) = -.31 x y f
.21 S
-.3
.15
-.4
.25 .1
-.2 .3 1
y_in1 = -.3(1) + .21(0) + .25(0) = -.3
f = .43 1
y_in3 = -.4(1) - .2(.43) (.56) = -.318
f = .42
y_in2 = .25(1) -.4(0) + .1(0)
(not 0)
f = .56
f(yj )=
y_ini 1
1 + e
Activation function f:
Training Case: (0 0 0)

34. Backpropagate d3 = (t3 – y3)f’(y_in3) =(t3 – y3)f(y_in3)[1- f(y_in3)]f
.21 S
-.3
.15
-.4
.25 .1
-.2 .3 1
d_in1 = d3w13 = -.102(-.2) = .02
d1 = d_in1f’(z_in1) = .02(.43)(1-.43)
= .005
d3 =
(0 – .42).42[1-.42]
= -.102
d_in2 = d3w12 = -.102(.3) = -.03
d2 = d_in2f’(z_in2) = -.03(.56)(1-.56)
= -.007

35. Update the Weights – First Pass
(t3 – y3)f’(y_in3)
=(t3 – y3)f(y_in3)[1- f(y_in3)] f
.21 S
-.3
.15
-.4
.25 .1
-.2 .3 1
d_in1 = d3w13 = -.102(-.2) = .02
d1 = d_in1f’(z_in1) = .02(.43)(1-.43)
= .005
d3 =
(0 – .42).42[1-.42]
= -.102
d_in2 = d3w12 = -.102(.3) = -.03
d2 = d_in2f’(z_in2) = -.03(.56)(1-.56)
= -.007

36. Final Result
After about 500 iterations: x y f 1 S
-1.5
-.5 -2

37. gradient descent method + MLP to whom may have interest
More details for
gradient descent method + MLP
to whom may have interest

38. In the perceptron/single layer nets, we used gradient descent on the error function to find the correct weights:
D wji = (tj - yj) xi
We see that errors/updates are local to the node ie the change in the weight from node i to output j (wji) is controlled by the input that travels along the connection and the error signal from output j x1
(tj - yj) x1 x2 ?
But with more layers how are the weights for the first 2 layers found when the error is computed for layer 3 only?
There is no direct error signal for the first layers!!!!!

39. Credit assignment problem
Problem of assigning ‘credit’ or ‘blame’ to individual elements
involved in forming overall response of a learning system
(hidden units)
In neural networks, problem relates to deciding which weights
should be altered, by how much and in which direction.
Analogous to deciding how much a weight in the early layer contributes to the output and thus the error
We therefore want to find out how weight wij affects the error ie we want:

40. Forward pass phase: computes ‘functional signal’, feedforward
Backpropagation learning algorithm ‘BP’
Solution to credit assignment problem in MLP
Rumelhart, Hinton and Williams (1986)
BP has two phases:
Forward pass phase: computes ‘functional signal’, feedforward
propagation of input pattern signals through network

41. Forward pass phase: computes ‘functional signal’, feedforward
Backpropagation learning algorithm ‘BP’
Solution to credit assignment problem in MLP. Rumelhart, Hinton and Williams (1986) (though actually invented earlier in a PhD thesis relating to economics)
BP has two phases:
Forward pass phase: computes ‘functional signal’, feedforward
propagation of input pattern signals through network
Backward pass phase: computes ‘error signal’, propagates the error backwards through network starting at output units (where the error is the difference between actual and desired output values)

42. Two-layer networks x1 Outputs of 1st layer zi x2 y1 Inputs xi
Outputs yj ym
2nd layer weights wij from j to i xn
1st layer weights vij from j to i

43. We will concentrate on three-layer, but could easily
generalize to more layers
zi (t) = g( S j vij (t) xj (t) ) at time t
= g ( ui (t) )
yi (t) = g( S j wij (t) zj (t) ) at time t
= g ( ai (t) )
a/u known as activation, g the activation function
biases set as extra weights

44. Forward pass
Weights are fixed during forward and backward pass at time t
1. Compute values for hidden units
2. compute values for output units yk
wkj(t) zj
vji(t) xi

45. Backward Pass
Will use a sum of squares error measure. For each training pattern we have:
where dk is the target value for dimension k. We want to know how to modify weights in order to decrease E. Use gradient descent ie
both for hidden units and output units

46. The partial derivative can be rewritten as product of two terms using chain rule for partial differentiation
both for hidden units and output units
How error for pattern changes as function of change
in network input to unit j
Term A
How net input to unit j changes as a function of
change in weight w
Term B

47. Term B first:
Term A Let
(error terms). Can evaluate these by chain rule:

48. For output units we therefore have:

49. For hidden units must use the chain rule:

50. Backward Pass
wki
wji Dk Dj
Weights here can be viewed as providing
degree of ‘credit’ or ‘blame’ to hidden units di
di = g’(ai) Sj wji Dj

51. vij(t+1)-vij(t) = h d i (t) xj (n) wij(t+1)-wij(t) = h D i (t) zj (t)
Combining A+B gives
So to achieve gradient descent in E should change weights by
vij(t+1)-vij(t) = h d i (t) xj (n)
wij(t+1)-wij(t) = h D i (t) zj (t)
Where h is the learning rate parameter (0 < h <=1)

52. Summary
Weight updates are local
output unit
hidden unit

53. End of slides

54. x1 x2 Z1 Z2 Y 1 2 -2
4.3
9.2
8.8
-0.1
5.3
-4.5
-0.8