WEEK 2 SOFT COMPUTING & MACHINE LEARNING YOSI KRISTIAN Gradient Descent for Linear Regression

Single Variable Linear Regression Gradient Descent

Have some function Want Outline: Start with some Keep changing to reduce until we hopefully end up at a minimum

Ilustration   J(     )

Ilustration   J(     )

The Algorithm Gradient descent algorithm Correct: Simultaneous updateIncorrect:

Algorithm Explained..   = Learning Rate  Following  = are the derivative Gradient descent algorithm

 effects.. If α is too small, gradient descent can be slow. If α is too large, gradient descent can overshoot the minimum. It may fail to converge, or even diverge.

Fixed .. Gradient descent can converge to a local minimum, even with the learning rate α fixed. As we approach a local minimum, gradient descent will automatically take smaller steps. So, no need to decrease α over time.

Applying Gradient Descent for Linear Regresion Gradient descent algorithm Linear Regression Model

Gradient Descent Function..

Algorithm.. Gradient descent algorithm update and simultaneously

Remember Local Minimum Problem.   J(     )

It Wont Happened Here..

“Batch” Gradient Descent “Batch”: Each step of gradient descent uses all the training examples.

Visualization (for fixed, this is a function of x)(function of the parameters )

Contd. (for fixed, this is a function of x)(function of the parameters )

Contd.. (for fixed, this is a function of x)(function of the parameters )

Contd. (for fixed, this is a function of x)(function of the parameters )

Contd… (for fixed, this is a function of x)(function of the parameters )

(for fixed, this is a function of x)(function of the parameters )

(for fixed, this is a function of x)(function of the parameters )

(for fixed, this is a function of x)(function of the parameters )

(for fixed, this is a function of x)(function of the parameters )

Homework  Create a program to demonstrate Gradient Descent usage on One Variable Linear Regression Problem.  Use Diamond Data.  Input : 1 variable  Output : 1 variable.  Visualize your program. (MSE, Line Regression)  Able to manually initialize  0  1

Multiple features Linear Regression with multiple variables

Previously

Multiple Feature

Multiple features (variables). Notation: = number of features = input (features) of training example. = value of feature in training example.

Hypothesis: Previously:

Still Hypothesis… For convenience of notation, define. Multivariate linear regression.

Gradient descent for multiple variables linear regression Linear Regression with multiple variables

Hypothesis: Symplified Cost function: Parameters: (simultaneously update for every ) Repeat Gradient descent:

Gradient Descent (simultaneously update ) Repeat Previously (n=1): New algorithm : Repeat (simultaneously update for )

Gradient descent in practice I: Feature Scaling Linear Regression with multiple variables

Feature Scaling E.g. = size ( feet 2 ) = number of bedrooms (1-5 ) Idea: Make sure features are on a similar scale. size (feet 2 ) number of bedrooms

Feature Scaling Get every feature into approximately a range.

Mean normalization Replace with to make features have approximately zero mean (Do not apply to ). E.g.

Choosing Learning Rate Linear Regression with multiple variables

Making sure gradient descent is working correctly. Example automatic convergence test: Declare convergence if decreases by less than in one iteration. No. of iterations

Making sure gradient descent is working correctly. Gradient descent not working. Use smaller. No. of iterations -For sufficiently small, should decrease on every iteration. -But if is too small, gradient descent can be slow to converge.

Summary: -If is too small: slow convergence. -If is too large: may not decrease on every iteration; may not converge. To choose, try

Homework  Create a program to demonstrate Gradient Descent usage on Multiple Variable Linear Regression Problem.  Use Housing Data.  Input : 2 variable  Output : 1 variable.  Able to manually initialize  0  1   is customizable  Do the “Feature Scalling”

Features and polynomial regression Linear Regression with multiple variables

Housing prices prediction

Polynomial regression Price (y) Size (x)

Finally … Fin…