Regression “A new perspective on freedom” TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAA A A A A AAA A A

Classification

? CatDog

Cleanliness Size

? $$$$$$$$$$

Regression

$ $$ $$$ $$$$ Price Top speed x y

Regression Data Goal: given, predict i.e. find a prediction function

Nearest neighbor

Nearest neighbor To predict x –Find the data point x i closest to x –Choose y = y i + No training – Finding closest point can be expensive – Overfitting

Kernel Regression To predict X –Give data point x i weight –Normalize weights –Let e.g.

Kernel Regression [matlab demo]

Kernel Regression + No training + Smooth prediction – Slower than nearest neighbor – Must choose width of

Linear regression

Temperature [start Matlab demo lecture2.m] Given examples Predict given a new point Temperature

Temperature Linear regression Prediction

Linear Regression Error or “residual” Prediction Observation Sum squared error

Linear Regression n d Solve the system (it’s better not to invert the matrix)

Minimize the sum squared error Sum squared error Linear equation Linear system

LMS Algorithm (Least Mean Squares) where Online algorithm

Beyond lines and planes everything is the same with still linear in

Linear Regression [summary] n d Let For example Let Minimize by solving Given examples Predict

Probabilistic interpretation Likelihood

Overfitting [Matlab demo] Degree 15 polynomial

Ridge Regression (Regularization) Effect of regularization (degree 19) with “small” Minimize Solve Let

Probabilistic interpretation Likelihood Prior Posterior

Locally Linear Regression

[source: Global temperature increase

Locally Linear Regression To predict X –Give data point x i weight –Let e.g.

Locally Linear Regression + Good even at the boundary (more important in high dimension) – Solve linear system for each new prediction – Must choose width of To minimize Solve Predict where

[source: Locally Linear Regression Gaussian kernel 180

[source: Locally Linear Regression Laplacian kernel 180

L1 Regression

Sensitivity to outliers High weight given to outliers Influence function

L 1 Regression Linear program Influence function

Spline Regression Regression on each interval

Spline Regression With equality constraints

Spline Regression With L 1 cost

To learn more The Elements of Statistical Learning, Hastie, Tibshirani, Friedman, Springer