10 1 Widrow-Hoff Learning (LMS Algorithm)

10 2 ADALINE Network  w i w i1  w i2  w iR  =

10 3 Two-Input ADALINE

10 4 Mean Square Error Training Set: Input:Target: Notation: Mean Square Error:

10 5 Error Analysis The mean square error for the ADALINE Network is a quadratic function:

10 6 Stationary Point Hessian Matrix: The correlation matrix R must be at least positive semidefinite. If there are any zero eigenvalues, the performance index will either have a weak minumum or else no stationary point, otherwise there will be a unique global minimum x*. If R is positive definite:

10 7 Approximate Steepest Descent Approximate mean square error (one sample): Approximate (stochastic) gradient:

10 8 Approximate Gradient Calculation

10 9 LMS Algorithm

10 Multiple-Neuron Case Matrix Form:

10 11 Analysis of Convergence For stability, the eigenvalues of this matrix must fall inside the unit circle.

10 12 Conditions for Stability Therefore the stability condition simplifies to 12  i –1–  Since,. (where i is an eigenvalue of R)

10 13 Steady State Response If the system is stable, then a steady state condition will be reached. The solution to this equation is This is also the strong minimum of the performance index.

10 14 Example BananaApple

10 15 Iteration One Banana

10 16 Iteration Two Apple

10 17 Iteration Three

10 18 Adaptive Filtering Tapped Delay LineAdaptive Filter

10 19 Example: Noise Cancellation

10 20 Noise Cancellation Adaptive Filter

10 21 Correlation Matrix

10 22 Signals 1.2     cos0.36–==mk   k  –   sin=

10 23 Stationary Point 0 0 h Esk  mk  +  vk  Esk  mk  +  vk1–  =

10 24 Performance Index

10 25 LMS Response

10 26 Echo Cancellation