1. Cross-validation, Revisiting Regression – local models, and non-parametric…
Peter Fox and Greg Hughes
Data Analytics – ITWS-4600/ITWS-6600
Group 4 Module 12, April 3, 2017

2. coleman
> head(coleman) salaryP fatherWc sstatus teacherSc motherLev Y

3. Cross-validation package cvTools
> call <- call("lmrob", formula = Y ~ .) # > # set up folds for cross-validation > folds <- cvFolds(nrow(coleman), K = 5, R = 10) > # perform cross-validation > cvTool(call, data = coleman, y = coleman$Y, cost = rtmspe, + folds = folds, costArgs = list(trim = 0.1)) CV [1,] [2,] [3,] [4,] [5,] [6,] [7,] [8,] [9,] [10,]
Warning messages:
1: In lmrob.S(x, y, control = control) :
S refinements did not converge (to refine.tol=1e-07) in 200
(= k.max) steps
2: In lmrob.S(x, y, control = control) :
3: In lmrob.S(x, y, control = control) :
find_scale() did not converge in 'maxit.scale' (= 200) iterations
4: In lmrob.S(x, y, control = control) :

4. Evaluating?
> cvFits 5-fold CV results: Fit CV 1 LS MM LTS Best model: CV "MM"

5. LS, LTS, MM?
The breakdown value of an estimator is defined as the smallest fraction of contamination that can cause the estimator to take on values arbitrarily far from its value on the uncontaminated data.
The breakdown value of an estimator can be used as a measure of the robustness of the estimator. Rousseeuw and Leroy (1987) and others introduced high breakdown value estimators for linear regression.
LTS – see
MM -

6. 50 and 75% subsets
fitLts50 <- ltsReg(Y ~ ., data = coleman, alpha = 0.5) cvFitLts50 <- cvLts(fitLts50, cost = rtmspe, folds = folds, fit = "both", trim = 0.1) # 75% subsets fitLts75 <- ltsReg(Y ~ ., data = coleman, alpha = 0.75) cvFitLts75 <- cvLts(fitLts75, cost = rtmspe, folds = folds, # combine and plot results cvFitsLts <- cvSelect("0.5" = cvFitLts50, "0.75" = cvFitLts75)

7. cvFitsLts (50/75)
> cvFitsLts 5-fold CV results: Fit reweighted raw Best model: reweighted raw "0.75" "0.75"

8. Tuning
tuning <- list(tuning.psi=c(3.14, 3.44, 3.88, 4.68)) # perform cross-validation cvFitsLmrob <- cvTuning(fitLmrob$call, data = coleman, y = coleman$Y, tuning = tuning, cost = rtmspe, folds = folds, costArgs = list(trim = 0.1))

9. cvFitsLmrob
5-fold CV results: tuning.psi CV Optimal tuning parameter: tuning.psi CV 4.68

10. Lab this week
mammals.glm <- glm(log(brain) ~ log(body), data = mammals) (cv.err <- cv.glm(mammals, mammals.glm)$delta) [1] > (cv.err.6 <- cv.glm(mammals, mammals.glm, K = 6)$delta) [1] # As this is a linear model we could calculate the leave-one-out # cross-validation estimate without any extra model-fitting. muhat <- fitted(mammals.glm) mammals.diag <- glm.diag(mammals.glm) (cv.err <- mean((mammals.glm$y - muhat)^2/(1 - mammals.diag$h)^2)) [1]

11. Cost functions, etc.
# leave-one-out and 11-fold cross-validation prediction error for # the nodal data set. Since the response is a binary variable # an appropriate cost function is > cost <- function(r, pi = 0) mean(abs(r-pi) > 0.5) > nodal.glm <- glm(r ~ stage+xray+acid, binomial, data = nodal) > (cv.err <- cv.glm(nodal, nodal.glm, cost, K = nrow(nodal))$delta) [1] > (cv.11.err <- cv.glm(nodal, nodal.glm, cost, K = 11)$delta) [1]

12. cvTools http://cran.r-project.org/web/packages/cvTools/cvTools.pdf
Very powerful and flexible package for CV (regression) but very much a black box!
If you use it, become very, very familiar with the outputs and be prepared to experiment…

13. Diamonds
require(ggplot2) # or load package first data(diamonds) head(diamonds) # look at the data! # ggplot(diamonds, aes(clarity, fill=cut)) + geom_bar() ggplot(diamonds, aes(clarity)) + geom_bar() + facet_wrap(~ cut) ggplot(diamonds) + geom_histogram(aes(x=price)) + geom_vline(xintercept=12000) ggplot(diamonds, aes(clarity)) + geom_freqpoly(aes(group = cut, colour = cut))

14. ggplot(diamonds, aes(clarity)) + geom_freqpoly(aes(group = cut, colour = cut))

15. bodyfat
## regular linear model using three variables lm1 <- lm(DEXfat ~ hipcirc + kneebreadth + anthro3a, data = bodyfat) ## Estimate same model by glmboost glm1 <- glmboost(DEXfat ~ hipcirc + kneebreadth + anthro3a, data = bodyfat) # We consider all available variables as potential predictors. glm2 <- glmboost(DEXfat ~ ., data = bodyfat) # or one could essentially call: preds <- names(bodyfat[, names(bodyfat) != "DEXfat"]) ## names of predictors fm <- as.formula(paste("DEXfat ~", paste(preds, collapse = "+"))) ## build formula

16. Compare linear models
> coef(lm1) (Intercept) hipcirc kneebreadth anthro3a > coef(glm1, off2int=TRUE) ## off2int adds the offset to the intercept Conclusion?

17. > fm DEXfat ~ age + waistcirc + hipcirc + elbowbreadth + kneebreadth + anthro3a + anthro3b + anthro3c + anthro4 > coef(glm2, which = "") ## select all. (Intercept) age waistcirc hipcirc elbowbreadth kneebreadth anthro3a anthro3b anthro3c anthro attr(,"offset") [1]

18. > gam2 <- gamboost(DEXfat ~
> gam2 <- gamboost(DEXfat ~ ., baselearner = "bbs", data = bodyfat,control = boost_control(trace = TRUE)) [ 1] risk: [ 53] Final risk: > set.seed(123) ## set seed to make results reproducible > cvm <- cvrisk(gam2) ## default method is 25-fold bootstrap cross-validation – what is this call doing????

19. > cvm Cross-validated Squared Error (Regression) gamboost(formula = DEXfat ~ ., data = bodyfat, baselearner = "bbs", control = boost_control(trace = TRUE)) Optimal number of boosting iterations: 33

20. > mstop(cvm) ## extract the optimal mstop [1] 33 > gam2[ mstop(cvm) ] ## set the model automatically to the optimal mstop Model-based Boosting Call: gamboost(formula = DEXfat ~ ., data = bodyfat, baselearner = "bbs", control = boost_control(trace = TRUE)) Squared Error (Regression) Loss function: (y - f)^2 Number of boosting iterations: mstop = 33 Step size: 0.1 Offset: Number of baselearners: 9

21. plot(cvm)

22. > names(coef(gam2)) ## displays the selected base-learners at iteration 30 [1] "bbs(waistcirc, df = dfbase)" "bbs(hipcirc, df = dfbase)" "bbs(kneebreadth, df = dfbase)" [4] "bbs(anthro3a, df = dfbase)" "bbs(anthro3b, df = dfbase)" "bbs(anthro3c, df = dfbase)" [7] "bbs(anthro4, df = dfbase)" > gam2[1000, return = FALSE] # return = FALSE just supresses "print(gam2)" [ 101] risk: [ 153] risk: [ 205] risk: [ 257] risk: [ 309] risk: [ 361] risk: [ 413] risk: [ 465] risk: [ 517] risk: [ 569] risk: [ 621] risk: [ 673] risk: [ 725] risk: [ 777] risk: [ 829] risk: [ 881] risk: [ 933] risk: [ 985] Final risk:

23. > names(coef(gam2)) ## displays the selected base-learners, now at iteration 1000 [1] "bbs(age, df = dfbase)" "bbs(waistcirc, df = dfbase)" "bbs(hipcirc, df = dfbase)" [4] "bbs(elbowbreadth, df = dfbase)" "bbs(kneebreadth, df = dfbase)" "bbs(anthro3a, df = dfbase)" [7] "bbs(anthro3b, df = dfbase)" "bbs(anthro3c, df = dfbase)" "bbs(anthro4, df = dfbase)” > glm3 <- glmboost(DEXfat ~ hipcirc + kneebreadth + anthro3a, data = bodyfat,family = QuantReg(tau = 0.5), control = boost_control(mstop = 500)) > coef(glm3, off2int = TRUE) (Intercept) hipcirc kneebreadth anthro3a

24. More local methods…

25. Why local?

26. Sparse?

27. Remember this one?
How would you apply local methods here?

28. SVM-type
One-class-classification: this model tries to find the support of a distribution and thus allows for outlier/novelty detection;
epsilon-regression: here, the data points lie in between the two borders of the margin which is maximized under suitable conditions to avoid outlier inclusion;
nu-regression: with analogue modifications of the regression model as in the classification case.

29. Reminder SVM and margin

30. Loss functions…
classification
outlier
regression

31. Regression
By using a different loss function called the ε-insensitive loss function ||y−f (x)||ε = max{0, ||y− f(x)|| − ε}, SVMs can also perform regression.
This loss function ignores errors that are smaller than a certain threshold ε > 0 thus creating a tube around the true output.

32. Example lm v. svm

34. Again SVM in R
E the svm() function in e1071 provides a rigid interface to libsvm along with visualization and parameter tuning methods.
kernlab features a variety of kernel-based methods and includes a SVM method based on the optimizers used in libsvm and bsvm
Package klaR includes an interface to SVMlight, a popular SVM implementation that additionally offers classification tools such as Regularized Discriminant Analysis.
Svmpath – you get the idea…

35. Knn is local – right?
nearest neighbors is a simple algorithm that stores all available cases and predict the numerical target based on a similarity measure (e.g., distance functions).
KNN has been used in statistical estimation and pattern recognition already in the beginning of 1970’s as a non-parametric technique. 

36. Distance…
A simple implementation of KNN regression is to calculate the average of the numerical target of the K nearest neighbors. 
Another approach uses an inverse distance weighted average of the K nearest neighbors. Choosing K!
KNN regression uses the same distance functions as KNN classification.
knn.reg and also in kknn

37. Lab… And reminder – Assignment 7 due in 2nd last week – Friday 5pm
Next week – mixed models! i.e. optimizing…
Open lab in 2nd last week (no new work)….