A Review: Architecture Basic architecture : Input layer, output layer Input Unit Output Unit 1 b X1 w1 Y wi Xi wn Xn Single-layer net for pattern classification

Biases & Thresholds  xiwi if net  0 -1 if net < 0 A bias acts exactly as a weight on a connection from a unit whose activation is always 1 Increasing the bias increases the net input to the unit If a bias is included, the activation function is typically taken to be f(net) = if net  0 -1 if net < 0 Where net = b +  xiwi i

Biases & Thresholds  xiwi Some do not use bias weight, but use a fixed threshold  for the activation function if net   -1 if net <  Where net =  xiwi f(net) = i

Biases & Thresholds Input Unit Output Unit 1 b X1 w1 Y w2 X2 Single-layer net for pattern classification

Linear Separable For a particular output unit, the desired response is a ‘yes’(output signal 1) if the input pattern is the member of its class and ‘no’(output signal –1) We want one of two responses Activation function – step function From the net input formula we can draw a line, a plane or hyperplane y_in = b +  xi wi b +  xi wi = 0

Linear Separable b +  xi wi = 0 The problem is said ‘linearly separable’ if there are weights (and a bias) so that all of the training input vectors for which the correct response is +1 lie on one side of the decision boundary and all of the training input vectors for which the correct response is –1 lie on the other side Draw the decision boundary using this equation b +  xi wi = 0

Linear Separability + - x1 - -

Linear Separability + + x1 - +

Linear Separability - + x1 - +

Linear Separability In the case of an elementary perceptron, the n-dimensional space is divided by a hyperplane into two decision regions. The hyperplane is defined by the linearly separable function:

Linear Separability

Linear Separability A perceptron can learn the operations AND and OR, but not Exclusive-OR.

The HEBB NET The earliest and simplest learning rule Proposed that learning occurs by modification of the synapse strengths (weights). If two interconnected neurons are both “on” or “off” at the same time, then the weight between those neurons should be increased. Hebb net also used for training other specific nets

The HEBB NET A single layer nets Interconnected neurons will be between an input unit and one output unit. Suit for bipolar form of data (1, -1) Limitation for binary data (see examples 2.5 & 2.6)

The HEBB NET A single layer nets Input Unit Output Unit 1 b X1 w1 Y w2 Single-layer net for pattern classification Output Unit Input Unit

The HEBB NET bipolar form of data (1, -1) Weight update wi(new) = wi(old) + xi y

The Algorithm Step 0: Initialize all weights: wi = 0 (i= 1 to n) X1 X2 Y b w2 Single-layer net for pattern classification Output Unit Input Unit Step 0: Initialize all weights: wi = 0 (i= 1 to n) Step 1: For each input training vector and target output pair, s:t, do steps 2-4 Step 2. Set activations for input units: xi = si (i = 1 to n). Step 3. Set activation for output unit: y = t (t= target) Step 4. Adjust the weights for wi(new) = wi(old) + wi (i = 1 to n).  wi =xi y Adjust the bias: b(new) = b(old) + y.

The Application Hebb net for And function: binary inputs and binary targets Hebb net for And function: binary inputs and bipolar targets Hebb net for And function: bipolar inputs and bipolar targets

Hebb net for And function: binary inputs and binary targets w1 1 X1 X2 Y b w2 Output Unit Input Unit Input Target   (x1 x2 1) (1 1 1) (1 0 1) (0 1 1) (0 0 1) 1 w1 = x1t w2 = x2 t b = t Weight Changes Weights (1 1 1) (1 0 1) (0 1 1) (0 0 1) ( w1 w2 b) (1 1 1) (0 0 0) (w1 w2 b) (0 0 0)

Hebb net for And function: binary inputs and binary targets Weight Changes Weights (x1 x2 1)   (1 1 1) (1 0 1) (0 1 1) (0 0 1) 1 ( w1 w2 b) (1 1 1) (0 0 0) (w1 w2 b) (0 0 0) The response of the net correct for the first input pattern but not for the 2, 3, 4th pattern because the target values is 0, no learning occurs. Using binary target values prevents the net from learning any pattern for which the target is “off”

How a net learns By adjusting weights If no weight adjusted :: no learning occurs w1 1 X1 X2 Y b w2 Output Unit Input Unit

Hebb net for And function: binary inputs and binary targets Weight Changes Weights (x1 x2 1)   (1 1 1) (1 0 1) (0 1 1) (0 0 1) 1 ( w1 w2 b) (1 1 1) (0 0 0) (w1 w2 b) (0 0 0) No Learning occur

Hebb net for And function: binary inputs and binary targets x2 w1 1 X1 X2 Y b w2 x1 -1 + b +  xi wi = 1 + x1(1)+ x2(1)=0 Separating lines x2= -x1-1 Basic formula to draw separating line b +  xi wi = 0

Hebb net for And function: binary inputs and bipolar targets   (x1 x2 1) (1 1 1) (1 0 1) (0 1 1) (0 0 1) 1 -1 w1 = x1t w2 = x2 t b = t Weight Changes Weights (1 1 1) (1 0 1) (0 1 1) (0 0 1) ( w1 w2 b) (1 1 1) (-1 0 -1) (0 -1 -1) (0 0 -1) (w1 w2 b) (0 0 0) (0 1 0) (0 0 -1) (0 0 -2) w1 1 X1 X2 Y b w2

Hebb net for And function: binary inputs and bipolar targets (x1 x2 1)   (1 1 1) (1 0 1) (0 1 1) (0 0 1) 1 -1 ( w1 w2 b) (1 1 1) (-1 0 -1) (0 -1 -1) (0 0 -1) (w1 w2 b) (0 0 0) (0 1 0) (0 0 -1) (0 0 -2) The response of the net correct for the first input pattern and for the 2, 3, 4th patterns shows that learning continues for each of these since the target value is now -1. However, these weights do not provide the correct response for the first input pattern

Hebb net for And function: binary inputs and bipolar targets x2 w1 1 X1 X2 Y b w2 - + -1 - - x1 -1 b +  xi wi = 1 + x1(1)+ x2(1)=0 Separating lines x2= -x1-1 Basic formula to draw separating line b +  xi wi = 0

Hebb net for And function: bipolar inputs and bipolar targets   (x1 x2 1) (1 1 1) (1 -1 1) (-1 1 1) (-1 -1 1) 1 -1 w1 = x1t w2 = x2 t b = t Weight Changes Weights (1 1 1) (1 -1 1) (-1 1 1) (-1 -1 1) ( w1 w2 b) (1 1 1) (-1 1 -1) (1 -1 -1) (1 1 -1) (w1 w2 b) (0 0 0) (1 1 1) (0 2 0) (1 1 -1) (2 2 -2) w1 1 X1 X2 Y b w2

Hebb net for And function: bipolar inputs and bipolar targets (x1 x2 1)   (1 1 1) (1 -1 1) (-1 1 1) (-1 -1 1) 1 -1 ( w1 w2 b) (1 1 1) (-1 1 -1) (1 -1 -1) (1 1 -1) (w1 w2 b) (0 0 0) (1 1 1) (0 2 0) (1 1 -1) (2 2 -2) The response of the net correct for the first input pattern and the 2, 3, 4th patterns. w1 1 X1 X2 Y b w2

Hebb net for And function: bipolar inputs and bipolar targets x2 - - 1 + -1 1 x1 - -1 - b +  xi wi = -2 + x1(2)+ x2(2)=0 Separating lines x2= -x1+1 Basic formula to draw separating line b +  xi wi = 0

Application: Character Recognition # . . . # . # . # . . . # . . . # . # . . # # # . # . . . # Pattern 1 Pattern 2 1 –1 –1 –1 1, -1 1 –1 1 –1, -1 –1 1 –1 –1, -1 1 –1 1 –1, 1 –1 –1 –1 1 -1 1 1 1 –1, 1 –1 –1 –1 1 1 –1 –1 –1 1, 1 –1 –1 –1 1, 1 –1 –1 –1 1, -1 1 1 1 -1