CS 621 Artificial Intelligence Lecture 25 – 14/10/05 Prof. Pushpak Bhattacharyya Training The Feedforward Network; Backpropagation Algorithm

Multilayer Feedforward Network - Needed for solving problems which are not linearly separable. - Hidden layer neurons: assist computation.

Forward connection; no feedback connection …….. Output layer …….. Hidden layer …….. …….. Input layer Forward connection; no feedback connection

TOTAL SUM SQUARE ERROR(TSS) Gradient Descent Rule j ΔWji α - δE/ δWji Wji fed feeding i P M E = error = ½ Σ Σ( tm – om) 2 p=1 m=1 TOTAL SUM SQUARE ERROR(TSS)

Gradient Descent For a Single Neuron y n Net input = Σ WiXi i=0 …. W0 = 0 Wn Wn-1 Xn X0 = -1 Xn-1

Characteristic function y= f(net) Characteristic function f = sigmoid = 1 / ( 1+ e-net ) df f = = f(1-f) dnet y net

α ΔWi - δE/ δWi E = ½( t- o)2 Y = 0 …. observed target Wn W0 Wn-1 Xn

α W = <Wn, ……, W0> randomly initialized ΔWi - δE/ δWi = - η δE/ δWi , η is the learning rate 0 <= η <=1 α

E ΔWi = - η δE / δWi δE / δWi = δ(1/2(t - o)2) / δWi = (δE / δo) * (δo / δWi ); chain rule = - (t - o) * (δo / δnet) * ( δnet / δWi)

δo / δnet = δ f(net) / δnet = f (net) = f ( 1 - f ) = o ( 1 - o )

δnet / δWi = xi y W …. …. Wn Wi W0 Xn X0 Xi n net = ΣWiXi i = 0

E = ½ (t - o)2 ΔWi = η (t - o) (1 - o) o Xi o δE / δo W δnet / δWi …. …. δf / δnet Wn Wi W0 Xn X0 Xi

o E = ½( t - o) 2 ΔWi = η (t - o) (1 - o) o Xi Obs: Xi = 0 , ΔWi = 0 If Xi is more, so is the ΔWi BLAME/CREDIT ASSIGNMENT …. …. Wn Wi W0 Xn X0 Xi

More the difference ( t – o ), more is Δw. If( t – o ) is +ve , so is Δw If( t – o ) is –ve, so is Δw

If o is 0/1 , Δw = 0 o is 0/1 when net = - ∞ or + ∞ Δw  0 because of o  0/1. It is called “saturation” or “paralysis’ of the network. It happens due to sigmoid. o 1 net

Solution to network saturation 1. y = k / (1+e–x) k 2. y = tanh(x) x - k

Solution to network saturation (Contd) 3. Scale the inputs Reduced the values Problem of floating/fixed number representation error.

ΔWi = η ( t - o) o ( 1 – o) Xi Smaller η  smaller ΔW

Start with large η, gradually decrease it. op. pt Wi Global minimum Start with large η, gradually decrease it.

Gradient Descent training is typically slow: First parameter: η ; learning rate Second parameter: β; Momentum factor 0 <= β <= 1

Momentum Factor Use a part of previous weight Change into the current weight change. (ΔWi)n = η (t - o) o (1 – o) Xi + β(ΔWi)n-1 Iteration

Effect of β If (ΔWi)n and (ΔWi)n-1 are of same sign then (ΔWi)n is enhanced. If (ΔWi)n and (ΔWi)n-1 are of opposite sign then effective (ΔWi)n is reduced.

Accelerates movement at A. 2) Dampens oscillation near global minimum. op. pt Q R S W Accelerates movement at A. 2) Dampens oscillation near global minimum.

(ΔWi)n = η (t - o) o (1 – o) Xi + β(ΔWi )n-1 Relation between η and β ? Pure gradient descent momentum

Relation between η and β η >> β ? η << β ? (ΔWi)n = η (t - o) o (1 – o) Xi + β(ΔWi)n-1

Relation between η and β (Contd) If η << β (ΔWi)n = β(ΔWi)n-1 recurrence Relation (ΔWi )n = β(ΔWi)n-1 = β[β(ΔWi)n-2] = β2[β(ΔWi)n-3] . = βn(ΔWi)0

Relation between η and β (Contd) β is typically 1/10 th of η Empirical Practice If β is very large compared to η, no effect of output error, input or neuron characteristics is felt. Also (ΔW) goes on decreasing since β is a fraction.