Ch 2. Concept Map ⊂ ⊂ Single Layer Perceptron = McCulloch – Pitts Type Learning starts in Ch 2 Architecture, Learning Adaline : Linear Learning Perceptron : Nonlinear Learning Real valued output Binary output Binary output Objective : W x > 0 , x ω₁ W x < 0 , x ω₂ T ⊂ T T Objective : W x = Any Desired Real Value Can be used for Both Classification / Regression Objective : W x = 1 +- T ⊂ Correction for Correct Sign, too No correction for correct sign Always Finds Perfect hyperplanes for Linearly Separable Cases All Patterns x Move Hyperplanes by an Amount Proportional to x Finds Optimum Hyperplanes for Linearly Nonseparable Cases Error Patterns Push & Pull Hyperplanes

Chapter 2. Single Layer Perceptron Ref. Perceptron, Madaline and Backpropagation, Widrow, Proc.IEEE 90. Rosenblatt - Perceptron Learning 1 2 D Adjusted Training Set Adjustable Weights x (n) 1 ) ( 1 n Adder s(n) 2 x (n) ) ( n y ) ( 2 n å M M M (n)=Bias D+1 x (n) D 1 (n) D + ) ( n e - Error signal ) ( n d Desired response

α = Learning Rate that scales x. Perceptron Learning – Error Correction Learning Initialize to zero or some random vector. 2) Input and Adjust the weights as follows: . α = Learning Rate that scales x. Rosenblatt set α = 1. The choice does not affect stability but it affects convergence time only if w(0) ≠ 0. A. Case 1: d(n) – y(n) = 1 or d(n) = 1 and y(n) =0 ; y(n+1) y(n) B. Case 2: d(n) – y(n) = –1 or d(n) = 0 and y(n) = 1 ; C. Case 3: d(n) = y(n) Do nothing. 3) Repeat 1)2) until No Classification Error for a Linearly Separable Case.

(2) Patterns of Classification Error - push and pull the decision hyperplanes. 1 ( x D ( 1 ) ) 1 ( w ) 2 ( w ) 2 ( D Same as (1) except

(3) Example – OR Learning x(1) pulls down to , x(2) pulls up to . 1 2 2 3 or w (0) X = 0 T 1 : Error at x(1) 1 w (1) X = 0 2 T w (2) X = 0 T 3 Error at x(2) – Overcorrection 3 2 No Error for All Patterns– Terminate Learning

Of AND Function

2. Adaptive Linear Element – ADALINE = Linear Neuron If d(n) = Real, Regression. If d(n) = Integer (0/1 or -1/+1), Classification. Perceptron Adaline s(n) y(n) x(n) - Perceptron error å D + D e(n) d(n)

- LMS Learning Rule – Normalized, controls stability and speed of conv. In general, Data Set is not completely specified in advance. LMS is good for training data stream drawn from stationary distribution at least for x(n) In practice 0.1 < <1 cf. -LMS : Instantaneous Gradient Descent

(2) Minimum Disturbance Principle – Adapt to reduce the Output Error for the Current Training Pattern, with Minimal disturbance to Responses already learned x(n) w(n+1) D w(n) w(n)

(3) Learning Rate Scheduling (Annealing) α, μ = Learning Rate, Step Size μ(n+1) = μ(n) – β or μ(n) = c/n or μ0 / [1 + n/τ] The Perceptron Learning Rule can also be derived from the LMS-Rule :

Perceptron Learning LMS Learning Works with binary (bipolar) outputs Works with both binary and analog Always converges for linearly separable cases - Theorem May not converge even for linearly separable cases May be unstable for linearly nonseparable cases – Use pocket Alg. Also works well for linearly nonseparable cases – finds minimum error solution No correction for correct classification Always corrects Objective : > 0 or < 0 Objective: = 1 or − 1 Nonlinear rule Linear rule

Graphical Representation of Learning For classification with Adaline, one could just stop learning when the signs are all correct – way to speed up Adaline learning. Perceptron ω 1 1 1 ω 1 Adaline After Learning -1 -1 ω 2 ω 2

To Noise Cancellation, Equalization, Echo Cancellation (4) ADALINE To Noise Cancellation, Equalization, Echo Cancellation

Student Questions -2005 Adaline Learning takes longer than Perceptron in general. But, still it may have an edge over Perceptron in some aspect ? Diff. between α–LMS and μ–LMS. Self-normalizing vs. const. Latter always converges in the mean to the minimum MSE solution. Anyway to learn without desired outputs ? What does it mean when Perceptron learning can be derived from LMS using J = … Not clear on Regression. How can we guarantee a whole system convergence ? Which of Perceptron and Adaline performs better, is more efficient ?