Supervised Hebbian Learning CHAPTER 7 Supervised Hebbian Learning Ming-Feng Yeh

Objectives The Hebb rule, proposed by Donald Hebb in 1949, was one of the first neural network learning laws. A possible mechanism for synaptic modification in the brain. Use the linear algebra concepts to explain why Hebbian learning works. The Hebb rule can be used to train neural networks for pattern recognition. Ming-Feng Yeh

Hebb’s Postulate Hebbian learning (The Organization of Behavior) When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it; some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased. 當細胞A的軸突到細胞B的距離進到足夠刺激它，且反覆或持續地刺激B，則在這兩個細胞或其中一個將會發生某種成長過程或代謝反應，以增加A對B的刺激效果。 Ming-Feng Yeh

Linear Associator W SR p n a a = Wp R1 S1 R S  The linear associator is an example of a type of neural network called an associator memory.  The task of an associator is to learn Q pairs of prototype input/output vectors: {p1,t1}, {p2,t2},…, {pQ,tQ}.  If p = pq, then a = tq. q = 1,2,…,Q. If p = pq + , then a = tq + . Ming-Feng Yeh

Hebb Learning Rule If two neurons on either side of a synapse are activated simultaneously, the strength of the synapse will increase.  The connection (synapse) between input pj and output ai is the weight wij.  Unsupervised learning rule Not only do we increase the weight when pj and ai are positive, but we also increase the weight when they are both negative. Supervised learning rule Ming-Feng Yeh

Supervised Hebb Rule Assume that the weight matrix is initialized to zero and each of the Q input/output pairs are applied once to the supervised Hebb rule. (Batch operation)  Ming-Feng Yeh

Performance Analysis  Assume that the pq vectors are orthonormal (orthogonal and unit length), then  If pq is input to the network, then the network output can be computed  If the input prototype vectors are orthonormal, the Hebb rule will produce the correct output for each input.  Ming-Feng Yeh

Performance Analysis  Assume that each pq vector is unit length, but they are not orthogonal. Then  error The magnitude of the error will depend on the amount of correlation between the prototype input patterns.  Ming-Feng Yeh

Orthonormal Case Success!! Ming-Feng Yeh

Not Orthogonal Case The outputs are close, but do not quite match the target outputs. Ming-Feng Yeh

Solved Problem P7.2 i. Orthogonal, not orthonormal, ii. Ming-Feng Yeh

Solutions of Problem P7.2 iii. Hamming dist. = 1 Hamming dist. = 2 Ming-Feng Yeh

Pseudoinverse Rule   Performance index:  Goal: choose the weight matrix W to minimize F(W).  When the input vectors are not orthogonal and we use the Hebb rule, then F(W) will be not be zero, and it is not clear that F(W) will be minimized.   If the P matrix has an inverse, the solution is  Ming-Feng Yeh

Pseudoinverse Rule P matrix has an inverse iff P must be a square matrix. Normally the pq vectors (the column of P) will be independent, but R (the dimension of pq, no. of rows) will be larger than Q (the number of pq vectors, no. of columns). P does not exist any inverse matrix.  The weight matrix W that minimizes the performance index is given by the pseudoinverse rule .  where P+ is the Moore-Penrose pseudoinverse. Ming-Feng Yeh

Moore-Penrose Pseudoinverse The pseudoinverse of a real matrix P is the unique matrix that satisfies  When R (no. of rows of P) > Q (no. of columns of P) and the columns of P are independent, then the pseudoinverse can be computed by .  Note that we do NOT need normalize the input vectors when using the pseudoinverse rule.  Ming-Feng Yeh

Example of Pseudoinverse Rule Ming-Feng Yeh

Autoassociative Memory The linear associator using the Hebb rule is a type of associative memory ( tq  pq ). In an autoassociative memory the desired output vector is equal to the input vector ( tq = pq ).  An autoassociative memory can be used to store a set of patterns and then to recall these patterns, even when corrupted patterns are provided as input.  W 3030 30 p 301 a n Ming-Feng Yeh

Corrupted & Noisy Versions Recovery of 50% Occluded Patterns  Recovery of 67% Occluded Patterns  Recovery of Noisy Patterns  Ming-Feng Yeh

Variations of Hebbian Learning Many of the learning rules have some relationship to the Hebb rule.  The weight matrices of Hebb rule have very large elements if there are many prototype patterns in the training set.  Basic Hebb rule:  Filtered learning: adding a decay term, so that the learning rule behaves like a smoothing filter, remembering the most recent inputs more clearly.  Ming-Feng Yeh

Variations of Hebbian Learning Basic Hebb rule:  Delta rule: replacing the desired output with the difference between the desired output and the actual output. It adjusts the weights so as to minimize the mean square error.  The delta rule can update the weights after each new input pattern is presented.  Unsupervised Hebb rule:  Ming-Feng Yeh

Solved Problem P7.6 + p a W n 1 b Wp = 0 p2 p1 i. 11 n 1 b W 2 p 21 p1 p2 Wp = 0 Why is a bias required to solve this problem? The decision boundary for the perceptron network is Wp + b = 0. If these is no bias, then the boundary becomes Wp = 0 which is a line that must pass through the origin. No decision boundary that passes through the origin could separate these two vectors. i. Ming-Feng Yeh

Solved Problem P7.6 Use the pseudoinverse rule to design a network with bias to solved this problem. Treat the bias as another weight, with an input of 1. ii. Wp + b = 0 p1 p2 Ming-Feng Yeh

Solved Problem P7.7 Up to now, we have represented patterns as vectors by using “1” and “–1” to represent dark and light pixels, respectively. What if we were to use “1” and “0” instead? How should the Hebb rule be changed? Bipolar {–1,1} representation:  Binary {0,1} representation:  , where 1 is a vector of ones.  Ming-Feng Yeh

Binary Associative Network + a S1 n 1 b SR R R1 S n = Wp + b a = hardlim(Wp + b) Ming-Feng Yeh