1. The Widrow-Hoff Algorithm (Primal Form) Repeat: Until convergence criterion satisfied return: Given a training set and learning rate Initial:  Minimize the square loss function using gradient descent  Dual form exists (i.e. ) (Typo on textbook!)

2. Gradient and Hessian  Let be a differentiable function. The gradient of functionat a point is defined as  If is a twice differentiable function. The Hessian matrix ofat a point is defined as

3. Example1:

4. Example 2: The Hessian is positive semi-definite

5. Solution of the Least Squares Problem The Normal Equations Notation: Find such that has the smallest value, i.e. This is a quadratic unconstrained minimization problem is the optimal solution if and only if

6. The Normal Equations of LSQ Letting we have the normal equations of LSQ: If is inversable then Note: The above result is based on the First Order Optimality Conditions (necessary & sufficient for differentiable convex minimization problems) is singular ? What if

7. Ridge Regression (Guarantee Exist) where