Jeff Howbert Introduction to Machine Learning Winter Regression Linear Regression

Jeff Howbert Introduction to Machine Learning Winter slide thanks to Greg Shakhnarovich (CS195-5, Brown Univ., 2006)

Jeff Howbert Introduction to Machine Learning Winter slide thanks to Greg Shakhnarovich (CS195-5, Brown Univ., 2006)

Jeff Howbert Introduction to Machine Learning Winter slide thanks to Greg Shakhnarovich (CS195-5, Brown Univ., 2006)

Jeff Howbert Introduction to Machine Learning Winter slide thanks to Greg Shakhnarovich (CS195-5, Brown Univ., 2006)

Jeff Howbert Introduction to Machine Learning Winter slide thanks to Greg Shakhnarovich (CS195-5, Brown Univ., 2006)

Jeff Howbert Introduction to Machine Learning Winter l Suppose target labels come from set Y –Binary classification:Y = { 0, 1 } –Regression:Y =  (real numbers) l A loss function maps decisions to costs: – defines the penalty for predicting when the true value is. l Standard choice for classification: 0/1 loss (same as misclassification error) l Standard choice for regression: squared loss Loss function

Jeff Howbert Introduction to Machine Learning Winter l Most popular estimation method is least squares: –Determine linear coefficients ,  that minimize sum of squared loss (SSL). –Use standard (multivariate) differential calculus:  differentiate SSL with respect to ,   find zeros of each partial differential equation  solve for ,  l One dimension: Least squares linear fit to data

Jeff Howbert Introduction to Machine Learning Winter l Multiple dimensions –To simplify notation and derivation, change  to  0, and add a new feature x 0 = 1 to feature vector x: –Calculate SSL and determine  : Least squares linear fit to data

Jeff Howbert Introduction to Machine Learning Winter Least squares linear fit to data

Jeff Howbert Introduction to Machine Learning Winter Least squares linear fit to data

Jeff Howbert Introduction to Machine Learning Winter l The inputs X for linear regression can be: –Original quantitative inputs –Transformation of quantitative inputs, e.g. log, exp, square root, square, etc. –Polynomial transformation  example: y =  0 +  1  x +  2  x 2 +  3  x 3 –Basis expansions –Dummy coding of categorical inputs –Interactions between variables  example: x 3 = x 1  x 2 l This allows use of linear regression techniques to fit much more complicated non-linear datasets. Extending application of linear regression

Jeff Howbert Introduction to Machine Learning Winter Example of fitting polynomial curve with linear model

Jeff Howbert Introduction to Machine Learning Winter l 97 samples, partitioned into: –67 training samples –30 test samples l Eight predictors (features): –6 continuous (4 log transforms) –1 binary –1 ordinal l Continuous outcome variable: – lpsa : log( prostate specific antigen level ) Prostate cancer dataset

Jeff Howbert Introduction to Machine Learning Winter Correlations of predictors in prostate cancer dataset

Jeff Howbert Introduction to Machine Learning Winter Fit of linear model to prostate cancer dataset

Jeff Howbert Introduction to Machine Learning Winter l Complex models (lots of parameters) often prone to overfitting. l Overfitting can be reduced by imposing a constraint on the overall magnitude of the parameters. l Two common types of regularization in linear regression: –L 2 regularization (a.k.a. ridge regression). Find  which minimizes:  is the regularization parameter: bigger imposes more constraint –L 1 regularization (a.k.a. lasso). Find  which minimizes: Regularization

Jeff Howbert Introduction to Machine Learning Winter Example of L 2 regularization L2 regularization shrinks coefficients towards (but not to) zero, and towards each other.

Jeff Howbert Introduction to Machine Learning Winter Example of L 1 regularization L1 regularization shrinks coefficients to zero at different rates; different values of give models with different subsets of features.

Jeff Howbert Introduction to Machine Learning Winter Example of subset selection

Jeff Howbert Introduction to Machine Learning Winter Comparison of various selection and shrinkage methods

Jeff Howbert Introduction to Machine Learning Winter L 1 regularization gives sparse models, L 2 does not

Jeff Howbert Introduction to Machine Learning Winter l In addition to linear regression, there are: –many types of non-linear regression  decision trees  nearest neighbor  neural networks  support vector machines –locally linear regression –etc. Other types of regression

Jeff Howbert Introduction to Machine Learning Winter MATLAB interlude matlab_demo_07.m Part B