Lasso/LARS summary Nasimeh Asgarian

Lasso Summary Least Absolute Shrinkage and Selection operator Given a set of input measurements x1,x2, …,xp and outcome measurement y, the lasso fits a linear model: ŷ = 0+1*x1+2*x2+…+ p*xp By minimizing ((y-ŷ)2) Subject to  | j| <= s

Computation of the lasso solution Start with all j = 0 Find the predictor xj most correlated with y and add it to the model Take residuals r = y – ŷ Continue, at each stage add the predictor most correlated with r, to the model Until all predictors are in the model

Lars Summary Least Angel Regression Lasso is a restricted version of Lars By minimizing L(, ) = ||y -  * X||2 +  ||1 LARS: uses least square directions in the active set of variables. Lasso: uses least square directions; if a variable crosses zero, it is removed from the active set.

Computation of the Lars solution: Start with all j = 0 Find the predictor xj most correlated with y Increase the coefficient j in the direction of the sign of its correlation with y Take residuals r = y – ŷ

Computation of the lasso solution: Lars (Least Angel Regression) Stop when some other predictor xk has as much correlation with r as xj has. Increase (j,k) in their joint least square direction, until some other predictor xm has as much correlation with the residual r. Continue until all predictors are in the model.

Lasso: choice of tuning parameters At each step of LOO CV, Do 10-fold CV, on training set (twice) Find optimal values of  and number of iteration based on 10-fold CV result. i.e. see which  value and how many number of steps gives maximum correlation coefficient. Choose this  and number of iteration to build the model for the test instance.