Notes on Bootstrapping 12/9/2018 Jeff Witmer 10 February 2016

New sample  new statistic 12/9/2018 We want to estimate a parameter (e.g., a population mean µ). We have a sample of data and a statistic (e.g., the sample mean). New sample  new statistic Many samples  many statistics  sampling distn But we can only imagine taking repeated samples of size n and observing how the sample mean behaves – e.g., finding out how far it typically is from µ.

12/9/2018 Traditional method: Use theory to study how the statistic should behave. Bootstrap method: Use a computer to simulate how the statistic behaves. Brad Efron, 1980s – “infinitely intelligent, but he doesn’t know any distribution theory” The quote is from David Freedman, who was exaggerating. Efron knows plenty of distribution theory; he developed the bootstrap anyway.

The Bootstrap 12/9/2018 We wonder what would happen (how would a statistic behave) in repeated samples from the population. Basic idea: Simulate the sampling distribution of any statistic (like a sample mean or proportion) by repeatedly sampling from the data. A “pseudo-sample” is to the sample as the sample is to the population. We hope.

The Bootstrap 12/9/2018 Basic idea: Simulate the sampling distribution of any statistic (like a sample mean) by repeatedly sampling from the data. Get a “pseudo-sample”/bootstrap sample and fit a model in order to calculate a statistic (i.e., an estimate of a parameter) from the sample. Do this MANY times, fit the model for each bootstrap sample and collect the estimates.

“Pull yourself up by your bootstraps” Why “bootstrap”? 12/9/2018 “Pull yourself up by your bootstraps” Lift yourself in the air simply by pulling up on the laces of your boots Metaphor for accomplishing an “impossible” task without any outside help Perhaps an unfortunate name…

Sampling Distribution Population BUT, in practice we don’t see the “tree” or all of the “seeds” – we only have ONE seed µ

Bootstrap Distribution 12/9/2018 Bootstrap Distribution What can we do with just one seed? Bootstrap “Population” Grow a new tree! Note: This analogy is to the parametric bootstrap; I’m mostly talking about the non-parametric bootstrap. Hat tip: Robin Lock -   µ

population::statistic The Bootstrap 12/9/2018 We hope that population::statistic is as bootstrap population::bootstrap statistic If so, then by studying the distribution of bootstrap statistics we can understand how our sample statistic arose. E.g., we can get a good idea of what the SE is, and thus create a good confidence interval.

Bootstrap Distribution The Bootstrap Process 12/9/2018 BootstrapSample Bootstrap Statistic BootstrapSample Bootstrap Statistic Original Sample Bootstrap Distribution . . Sample Statistic BootstrapSample Bootstrap Statistic

A small bootstrap example 12/9/2018 Consider these 7 data points on the variable Y = pulse: 48, 50, 60, 66, 68, 72, 90 The (sample) mean is 64.9 (and the SD is 14.3). Let’s take a bootstrap sample, by writing the 7 values on 7 cards, shuffling, and making 7 draws with replacement. Then calculate the mean of those 7 values. (We can call this a bootstrap mean.) From 6 STAT 113 students from California plus 1 from China. Repeat this a few times. This is tedious. Technology helps…StatKey…

Bootstrapping with StatKey 12/9/2018 We can go to StatKey, choose Bootstrap Confidence Interval and choose CI for Single Mean, Median, Std. Dev., choose any of the built-in data sets and click the Edit Data button, and then type in our data (48, 50, 60, 66, 68, 72, 90). Then the Generate 1 Sample button will create a bootstrap sample. And we can get 1000 bootstrap samples easily.

Bootstrapping with StatKey 12/9/2018 From 5000 bootsamples we see that the SE (“st. dev.”) is almost exactly 5. We can make a 95% CI for µ via Estimate ± 2 * SE 64.9 ± 2*(5) 64.9 ± 10 (54.9, 74.9) Good news: We can bootstrap a difference in means or a regression coefficient or… Bad news: Bootstrapping when n is small does not overcome the problem that a small sample might not mimic the shape and spread of the population.

Bootstrap Percentile CIs 12/9/2018 If the bootstrap distribution is not symmetric then “estimate ± 2*SE” won’t be right. And what if we want something other than 95% confidence? We can take the middle 95% of the bootstrap distribution as our interval. Or the middle 90% for a 90% CI, etc.

We can made a 95% CI for µ via Recall the pulse data 12/9/2018 n=7, data are (48, 50, 60, 66, 68, 72, 90). We can made a 95% CI for µ via Estimate ± 2 * SE 64.9 ± 2*(5) 64.9 ± 10 (54.9, 74.9) Click the “Two-Tail” button. The middle 95% of the bootstrap distn is (55.4, 75.1)

Bootstrapping other statistics: Correlation 12/9/2018 We can bootstrap a correlation. Consider the Commute Atlanta (Time as a function of Distance) data. The sample correlation is r=0.81. Generate 5000 bootstrap samples. Each time, get the correlation b/w Distance and Time. Click the “Two-Tail” button. The middle 95% of the bootstrap distn of correlations is (0.72, 0.87) Note that this is not symmetric around 0.81!

Atlanta commute correlation bootstrapped 12/9/2018

Bootstrapping other statistics 12/9/2018 We can use the bootstrap for almost anything – but not everything works... Consider the Ten Counties (Area) data, which are skewed and have a small n of 10:

If we try to bootstrap the median, we don’t get a reasonably smooth and symmetric graph. Instead, we get clumpiness: 12/9/2018

Other Bootstrap CIs 12/9/2018 There are lots of ways to make a bootstrap CI. These include: (a) Percentile method (b) Normal method/ t with bootstrap SE (c) “basic” bootstrap (aka “reverse percentiles”, which is like inverting a randomization test) (d) T method/bootstrap t interval (a) And (b) are commonly used, but may not work well, esp. if n is small. (c) is fairly common but often doesn’t work well. (d) and (e) are better but are harder to do. T method: (theta-hat – q(0.975)*SE, theta-hat – q(0.025)*SE), so (a) we get quantiles from bootstrap t-distribution rather than from a t-table; and (b) we subtract an upper quantile of the bootstrap t-distribution to get the lower endpoint. (e) BCa – bias corrected, accelerated

N.B. We use the bootstrap to get CIs 12/9/2018 The center of the bootstrap distn is not the center of the sampling distn. Bootstrapping informs us about the standard error, it can alert us to bias, etc. An unfortunate name? Taking thousands of bootstrap samples does not increase the original sample size, nor, e.g., push the sample mean closer to µ. We don’t really “get something for nothing.” But we do learn about uncertainty in the sample statistic.

Bootstrap and Regression/Correlation 12/9/2018 Suppose we have a set of (x,y) pairs in a regression setting. If we don’t trust a t-test or t-based CI, how might we bootstrap? (1) Randomly resample (x,y) pairs and fit a regression model to the new “boot-sample.” (2) Randomly resample the residuals from the fitted model and attach them to the (fixed) (x,y) pairs. Then fit a regression model to this new “boot-sample.” ^ If we think of the x’s as fixed (e.g., a designed experiment), then (2) looks good. If not, then (1).

Implementing the bootstrap 12/9/2018 How does one actually do this? If it is 1991, then maybe write your own code. Today, use software such as StatKey for a small problem in a standard setting (1 mean, 2 means, simple regression, etc.) In general, use R and either (a) write a function for the statistic you want to bootstrap and use the boot() command or (b) use the do()* command in the mosaic package.

Example 12/9/2018 A consulting client recently needed a CI for a ratio estimator, so I wrote a short script in R. Here is part of it: ratioBoot <- function(mydata, indices) { d <- mydata[indices,] # allows boot to select sample ratio <- sum(d$Y)/sum(d$X) return(ratio) } mydata=ratio.data #this is the client’s data file library(boot) #make the boot package active bootratios5000 = boot(mydata, ratioBoot, R=5000) boot.ci(bootratios5000, type="norm") #gives 0.1236 to 0.2486 boot.ci(bootratios5000, type="perc") #gives 0.1221 to 0.2479 hist(bootratios5000$t) #a histogram of the 5000 values mean(bootratios5000$t) #gives 0.1847 sd(bootratios5000$t) #gives 0.0316 Note: Lots of digits reported here because the client is dealing with millions of dollars (and courts/judges tend to like things reported to the nearest dollar, if not the nearest penny).

Using the do()* command 12/9/2018 The mosaic package in R has a command that will do something (anything) many times. library(mosaic) pulse=c(48, 50, 60, 66, 68, 72, 90) mean(pulse) mean(resample(pulse)) do(5)*resample(pulse) do(5)*mean(resample(pulse)) mymeans = do(5)*mean(resample(pulse)) mymeans mymeans = do(10000)*mean(resample(pulse)) hist(mymeans$mean, breaks=100) mean(mymeans$mean) sd(mymeans$mean) quantile(mymeans$mean, probs=c(0.025,0.975)) #gives 55.7 to 75.1

There is more… 12/9/2018 There are yet other variations on the theme. E.g., the "wild bootstrap" in which you keep each residual with its observation, but randomly multiply the residual by -1 with probability 1/2. [Bootstrap tilting: Reweight the data to satisfy a null hypothesis, sample from the weighted distribution, then invert a hyp test to get a CI.] Note: I have been talking about the nonparametric bootstrap. There is also the smoothed bootstrap (take samples from a smooth (kernel density estimated) population) and the parametric bootstrap (take samples from a parametric distribution after estimating the parameters) [the “grow a new tree analogy’].

References 12/9/2018 Tim Hesterberg’s 2015 paper “What Teachers Should Know About the Bootstrap: Resampling in the Undergraduate Statistics Curriculum” in The American Statistician. See http://amstat.tandfonline.com/doi/full/10.1080/00031305.2 015.1089789 This is a summary of a longer (2014) paper with the same title, available at http://arxiv.org/abs/1411.5279. Efron and Tibshirani (1993) An Introduction to the Bootstrap, Springer.