2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 1 Hastie, Tibshirani and Friedman.The Elements of Statistical Learning (2nd edition, 2009). Presentation by: Ayellet Jehassi

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 2 1)Introduction 2)Bias, Variance and Model Complexity 3)AIC and BIC 4)Cross-Validation 5)Bootstrap Methods

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 3  The process of evaluating a model’s performance is known as model assessment, where the process of selecting the proper level of flexibility for a model is known as assessment model selection. For example: - Repeatedly drawing samples from a training set and refitting a model of interest on each sample in order to obtain additional information about the fitted model. - Fitting the same statistical method multiple times using different subsets of the training data.

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 4

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 5  We have many different potential predictors. Why not base the model on all of them?  Two sides of one coin: bias and variance – The model with more predictors can describe the phenomenon better – less bias. – When we estimate more parameters, the variance of estimators grows – we “fit the noise”, i.e. we perform an over fitting!  A clever model selection strategy should resolve the bias–variance trade–off.

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 6

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 7 The best approach for both problems is to randomly divide the dataset into three parts: a training set, a validation set, and a test set.  The training set is used to fit the models.  The validation set is used to estimate prediction error for model selection.  The test set is used for assessment of the generalization error of the final chosen model. The typical proportions are respectively: 50%, 25%, 25% The methods of this chapter approximate the validation step either analytically or by efficient sample re-use.

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 8

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 9

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 10

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 11

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 12

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 13

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 14

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 15

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 16  We randomly divide the available set of samples into two parts: a training set and a validation or hold-out set.  The model is fit on the training set, and the fitted model is used to predict the responses for the observations in the validation set.  The resulting validation-set error provides an estimate of the test error.

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 17  Widely used approach for estimating test error.  Estimates can be used to select best model, and to give an idea of the test error of the final chosen model.  The idea is to randomly divide the data into K equal-sized parts. We leave out part k, fit the model to the other K − 1 parts (combined), and then obtain predictions for the left-out kth part.  This is done in turn for each part k = 1, 2,... K, and then the results are combined.

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 18

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 19

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 20

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 21 Consider a simpler classifier for microarrays: 1)Starting with 5,000 genes, find the top 200 genes having the largest correlation with the class label. 2)Carry about the nearest-centroid classification using top 200 genes. How do we select the tuning parameter in the classifier? Way 1: apply cross-validation to step 2 Way 2: apply cross-validation to steps 1 and 2 Which is right? – Cross validating the whole procedure.

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 22

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 23  The bootstrap is a flexible and powerful statistical tool that can be used to quantify the uncertainty associated with a given estimator or statistical learning method.  For example, it can provide an estimate of the standard error of a coefficient, or a confidence interval for that coefficient.

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 24

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 25  The bootstrap approach allows us to use a computer to mimic the process of obtaining new data sets, so that we can estimate the variability of our estimate without generating additional samples.  Rather than repeatedly obtaining independent data sets from the population, we instead obtain distinct data sets by repeatedly sampling observations from the original data set with replacement.

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 26  Each of these “bootstrap data sets” is created by sampling with replacement, and is the same size as our original dataset.  As a result some observations may appear more than once in a given bootstrap data set and some not at all.

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 27  To estimate prediction error using the bootstrap, we could think about using each bootstrap dataset as our training sample, and the original sample as our validation sample.  But each bootstrap sample has significant overlap with the original data. About two-thirds of the original data points appear in each bootstrap sample.

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 28

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 29

2015 AprilUNIVERSITY OF HAIFA, DEPARTMENT OF STATISTICS, SEMINAR FOR M.A 30