Backpropagation

Last time…

Correlational learning: Hebb rule What Hebb actually said: When an axon of cell A is near enough to excite a cell B and repeatedly and consistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficacy, as one of the cells firing B, is increased. The minimal version of the Hebb rule: When there is a synapse between cell A and cell B, increment the strength of the synapse whenever A and B fire together (or in close succession). The minimal Hebb rule as implemented in a network:

Limitations of Hebbian learning Many association problems it cannot solve Especially where similar input patterns must produce quite different outputs. Without further constraints, weights grow without bound Each weight is learned independently of all others Weight changes are exactly the same from pass to pass.

Predictive error-driven learning with linear units…

1 2 3 w1 w2

Why don’t we just use two-layer perceptrons?

aj = bw + ai * wij i b j w1 bw

1 2 3 w1 w2 b Input 1 Input 2 Output 1 𝑎 3 = 𝑏 𝑤 +𝑎 1 𝑤 1 + 𝑎 2 𝑤 2 1

1 2 3 w1 w2 b Input 1 Input 2 Output 1 𝑎 3 = 𝑏 𝑤 +𝑎 1 𝑤 1 + 𝑎 2 𝑤 2 1

Linear separability problem: with linear units (and thresholded linear units), *no solutions exist* for input/output mappings that are not linearly separable. What if we add an extra layer between input and output?

5 𝑎 5 = 𝑎 3 𝑤 5 + 𝑎 4 𝑤 6 w5 w6 = (𝑎 1 𝑤 1 + 𝑎 2 𝑤 3 ) 𝑤 5 + (𝑎 1 𝑤 2 + 𝑎 2 𝑤 4 ) 𝑤 6 3 4 = 𝑎 1 ( 𝑤 1 𝑤 5 + 𝑤 2 𝑤 6 )+ 𝑎 2 ( 𝑤 3 𝑤 5 + 𝑤 4 𝑤 6 ) = 𝑎 1 𝑤 𝑥 + 𝑎 2 𝑤 𝑦 w2 w3 w1 w4 1 2 Same as a linear network without any hidden layer!

What if we use thresholded units?

5 𝑛𝑒𝑡 𝑗 = 𝑖 𝑎 𝑖 𝑤 𝑖𝑗 w5 w6 If netj > thresh, aj = 1 Else aj = 0 3 4 w2 w3 w1 w4 1 2

𝑛𝑒𝑡 𝑗 = 𝑖 𝑎 𝑖 𝑤 𝑖𝑗 5 If netj > 9.9, aj = 1 Else aj = 0 10 -10 3 4 00 11 01 10 Unit 3 Unit 4 10 10 5 5 1 2

00 11 01 10 Unit 3 Unit 4 Input space 00 11 01 10 Input space Hidden space 11 01 00 10

So with thresholded units and a hidden layer, solutions exist… …and solutions can be viewed as “re-representing” the inputs, so as to make the mapping to the output unit learnable. BUT, how can we learn the correct weights instead of just setting them by hand?

But what if: 𝐸= 1 2 𝑗 ( 𝑡 𝑗 − 𝑎 𝑗 ) 2 𝑑𝐸 𝑑 𝑤 𝑖𝑗 =− ( 𝑡 𝑗 −𝑎 𝑗 )× 𝑎 𝑖 𝑑𝐸 𝑑 𝑤 𝑖𝑗 = 𝑑𝐸 𝑑 𝑎 𝑗 𝑑 𝑎 𝑗 𝑑 𝑤 𝑖𝑗 Simple delta rule: ∆ 𝑤 𝑖𝑗 =𝛼( 𝑡 𝑗 − 𝑎 𝑗 )× 𝑎 𝑖 𝑎 𝑗 = 𝑖 𝑎 𝑖 𝑤 𝑖𝑗 𝑛𝑒𝑡 𝑗 = 𝑖 𝑎 𝑖 𝑤 𝑖𝑗 𝑎 𝑗 =𝑓( 𝑛𝑒𝑡 𝑗 ) 𝑑𝐸 𝑑 𝑤 𝑖𝑗 = 𝑑𝐸 𝑑 𝑎 𝑗 𝑑 𝑎 𝑗 𝑑 𝑛𝑒𝑡 𝑗 𝑑 𝑛𝑒𝑡 𝑗 𝑑 𝑤 𝑖𝑗 …What function should we use for aj?

Can’t use a threshold function---why not? Derivative = 0 here Derivative is infinite here Activation Net i

𝑎 𝑗 = 1 1+ 𝑒 − 𝑛𝑒𝑡 𝑗 Sigmoid function: Net input Change in activation 1.00 0.90 0.80 0.70 Change in activation 0.60 0.50 0.40 Activation 0.30 0.20 0.10 0.00 -10 -5 5 10 Net input

𝐸= 1 2 𝑗 ( 𝑡 𝑗 − 𝑎 𝑗 ) 2 𝑑𝐸 𝑑 𝑤 𝑖𝑗 =− ( 𝑡 𝑗 −𝑎 𝑗 )× 𝑎 𝑖 𝑑𝐸 𝑑 𝑤 𝑖𝑗 = 𝑑𝐸 𝑑 𝑎 𝑗 𝑑 𝑎 𝑗 𝑑 𝑤 𝑖𝑗 Simple delta rule: ∆ 𝑤 𝑖𝑗 =𝛼( 𝑡 𝑗 − 𝑎 𝑗 )× 𝑎 𝑖 𝑎 𝑗 = 𝑖 𝑎 𝑖 𝑤 𝑖𝑗 𝑎 𝑗 = 1 1+ 𝑒 − 𝑛𝑒𝑡 𝑗 𝑑 𝑎 𝑗 𝑑 𝑛𝑒𝑡 𝑗 = 𝑎 𝑗 (1− 𝑎 𝑗 ) 𝑑𝐸 𝑑 𝑤 𝑖𝑗 = 𝑑𝐸 𝑑 𝑎 𝑗 𝑑 𝑎 𝑗 𝑑 𝑛𝑒𝑡 𝑗 𝑑 𝑛𝑒𝑡 𝑗 𝑑 𝑤 𝑖𝑗 𝑑𝐸 𝑑 𝑤 𝑖𝑗 =− ( 𝑡 𝑗 −𝑎 𝑗 )× 𝑎 𝑗 (1− 𝑎 𝑗 )×𝑎 𝑖 ∆ 𝑤 𝑖𝑗 =𝛼 ( 𝑡 𝑗 −𝑎 𝑗 )× 𝑎 𝑗 (1− 𝑎 𝑗 )×𝑎 𝑖

𝛿 5 𝛿 5 * w5 𝑑𝐸 𝑑 𝑤 5 = 𝑑𝐸 𝑑 𝑎 5 𝑑 𝑎 5 𝑑 𝑛𝑒𝑡 5 𝑑 𝑛𝑒𝑡 5 𝑑 𝑤 5 5 𝑑𝐸 𝑑 𝑤 5 = 𝑑𝐸 𝑑 𝑎 5 𝑑 𝑎 5 𝑑 𝑛𝑒𝑡 5 𝑑 𝑛𝑒𝑡 5 𝑑 𝑤 5 5 𝛿 5 𝑛𝑒𝑡 𝑗 = 𝑖 𝑎 𝑖 𝑤 𝑖𝑗 𝑑𝐸 𝑑 𝑤 1 w5 w6 3 4 w2 w3 𝑑 𝑛𝑒𝑡 5 𝑑 𝑎 3 = 𝑤 5 𝑑𝐸 𝑑 𝑎 3 = 𝛿 5 * w5 w1 w4 1 2 𝑑 𝑎 3 𝑑 𝑛𝑒𝑡 3 = 𝑎 3 (1− 𝑎 3 ) 𝑑 𝑛𝑒𝑡 3 𝑑 𝑤 1 = 𝑎 1 𝑑𝐸 𝑑 𝑤 1 = 𝛿 5 × 𝑑 𝑛𝑒𝑡 5 𝑑 𝑎 3 𝑑 𝑎 3 𝑑 𝑛𝑒𝑡 3 𝑑 𝑛𝑒𝑡 3 𝑑 𝑤 1

5 6 Targets For outputs delta computed directly based on error. Delta is stored at each unit and also used directly to adjust each incoming weight. 3 4 5 1 2 6 Output For hidden units, there are no targets; “error” signal is instead the sum of the output unit deltas. These are used to compute deltas for the hidden units, which are again stored with unit and used to directly change incoming weights. Hidden Input Deltas, and hence error signal at output, can propagate backward through network through many layers until it reaches the input.

Alternative error functions.

𝐸= 1 2 (𝑡−𝑎) 2 Sum-squared error: 𝑑𝐸 𝑑𝑎 =−(𝑡−𝑎) 𝑑𝐸 𝑑𝑛𝑒𝑡 =− 𝑡−𝑎 𝑎(1−𝑎) 5 w5 w6 3 4 w2 w3 w1 w4 1 2 𝐸=−𝑡 ln 𝑎 −(1−𝑡) ln (1−𝑎) Cross-entropy error: 𝑑𝐸 𝑑𝑛𝑒𝑡 =𝑎−𝑡

5 w5 w6 3 4 w2 w3 w1 w4 1 2

Input 1 Input 2 New input Output 1 1 3 w1 w2 1 2 2