1. The Perceptron CS/CMPE 333 – Neural Networks

2. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS2 The Perceptron – Basics Simplest and one of the earliest neural network model proposed by Rosenblatt in 1958, 1962 It is based on the McCulloch-Pitts model of a neuron Characteristics  Single-layer feedforward network (A layer of input nodes and one layer of computation/output nodes)  Threshold activation function (or hard limiter function)  Performs classification of linearly separable patterns  Trained using error-correcting learning

3. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS3 The Perceptron Consider a single neuron perceptron

4. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS4 Linear Separability (1) The function of the single neuron perceptron is to classify the input x into one of two classes, C 1 and C 2. In general, a q neuron perceptron can classify the input x into 2 q classes For the two classes case, the decision boundary is defined by the hyperplane Σ i=1 p w i x i – Θ = 0

5. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS5 Linear Separability (2) When p = 2 (i.e. two inputs), the decision boundary is a line

6. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS6 Error-Correction Learning and Pattern Classification

7. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS7 Pattern Classification (1) Let n = number of training samples (set X); X 1 = set of training sample belonging to C 1 ; X 2 = set of training sample belonging to C 2 For a given sample n x(n) = [-1, x 1 (n),…, x p (n)] T = input vector w(n) = [Θ(n), w 1 (n),…, w p (n)] T = weight vector Net activity level v(n) = w T (n)x(n) Output y(n) = +1 if v(n) >= 0 and y(n) = -1 otherwise The decision hyperplane separates classes C 1 and C 2

8. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS8 Pattern Classification (2) If the two classes C 1 and C 2 are linearly separable, then there exists a weight vector w such that w T x ≥ 0 for all x belonging to class C 1 w T x < 0 for all x belonging to class C 2

9. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS9 Error-Correction Learning Update rule: w(n + 1) = w(n) + Δw(n) Learning process  If x(n) is correctly classified by w(n), then w(n + 1) = w(n)  Otherwise, the weight vector is updated as follows w(n + 1) = w(n) – η(n)x(n) if w(n) T x(n) ≥ 0 and x(n) belongs to C 2 and w(n + 1) = w(n) + η(n)x(n) if w(n) T x(n) < 0 and x(n) belongs to C 1

10. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS10 Perceptron Convergence Algorithm (1) Variables and parameters  x(n) = [-1, x 1 (n),…, x p (n)]; w(n) = [Θ(n), w 1 (n),…,w p (n)]  y(n) = actual response (output); d(n) = desired response  η = learning rate, a positive number less than 1 Step 1: Initialization  Set w(0) = 0, then do the following for n = 1, 2, 3, … Step 2: Activation  Activate the perceptron by applying input vector x(n) and desired output d(n) Step 3: Computation of actual response  y(n) = sgn[w T (n)x(n)]  Where sgn(.) is the signum function

11. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS11 Perceptron Convergence Algorithm (2) Step 4: Adaptation of weight vector w(n+1) = w(n) + η[d(n) – y(n)]x(n) Where d(n) = +1 if x(n) belongs to C 1 d(n) = -1 if x(n) belongs to C 2 Step 5  Increment n by 1, and go back to step 2

12. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS12 Performance Measure (1) A learning rule is designed to optimize a performance measure  However, in the development of the perceptron convergence algorithm we did not mention a performance measure Intuitively, what would be an appropriate performance measure for a classification neural network? Define the performance measure J = -E[e(n)v(n)] Or, as an instantaneous estimate J’(n) = -e(n)v(n)  e(n) = error at iteration n = d(n) – y(n); v(n) = linear combiner output at iteration n; E = expectation operator

13. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS13 Performance Measure (2) Can we derive our learning rule by minimizing this performance function… Now v(n) = w T (n)x(n), thus Learning rule

14. CS/CMPE 333 - Neural Networks (Sp 2002/2003) - Asim Karim @ LUMS14 Concluding Remarks A single layer perceptron can perform pattern classification only on linearly separable patterns, regardless of the type of nonlinearity (hard limiter, signoidal) Papert and Minsky in 1969 elucidated limitations of Rosenblatt’s single layer perceptron (e.g. requirement of linear separability, inability to solve XOR problem) and cast doubt on the viability of neural networks However, multilayer perceptron and the back- propagation algorithm overcomes many of the shortcomings of the single layer perceptron