Review bootstrap and permutation

Main points Definition of confidence intervals Definition of p-value Why use bootstrap, why use permutation tests? What are the differences between bootstrap and permutation tests? How to use bootstrap? How to use permutation tests?

Definition of confidence intervals A confidence interval represents the precision of the estimation of a test statistic If the same experiment was replicated a hundred times, the 95% CI would, on average, contain the estimated TS in 95 of these samples.

Definition of p-value A p-value represents the probability of observing a TS this extreme or more extreme if the null hypothesis is true

Why use bootstrap, why use permutation tests? Test statistic Theoretical hypothesis Operational hypothesis Null hypothesis P-value OR CI Underlying distribution If one of those is unusual or unknown, bootstrap or permutation is useful 5

Test statistics Is not implemented in standard software Is estimated by a single value in the whole sample Is a relationship between two TS

Null hypothesis Is different from zero in a way that is hard to quantify Example: If you are expecting a certain variable to explain more than 50 percent of a different variable, such that the r squared is greater than 50. We would want to bootstrap the r-statistics and see whether the confidence interval includes 50, or values below 50. If the confidence interval does not, then we can say that our hypothesis is supported. If it does, then we cannot reject the null.

Underlying distribution Is unknown because TS was unknown Is unknown because conditions of applications for parametric tests do not seem to be met Is known for the null hypothesis but seems likely to be different for the alternative hypothesis p-value should be correct but CI will be incorrect.

Differences between bootstrap and permutation tests estimates confidence interval, bias and standard error Simulates data under the alternative hypothesis Sampling is done with replacement of subjects Many bootstrap samples because of replacement Permutation tests estimates p-value and distribution under the null. Simulates data under the null hypothesis Sampling is done without replacement of subjects Finite number of potential permutation samples

Main points: how to How to bootstrap a test statistics Determine the test statistic of interest (must be a single value) What is randomly sampled? How many subjects in the bootstrap samples (with or without replacement)? How many bootstrap samples Examine histogram of TS* with TS (the observed TS), average of TS*, and boundaries of percentile and bca CI Interpret results

Main points: how to How to use a permutation test Determine the test statistic of interest (must be a single value) What must be shuffled in order to simulate what happens under the null hypothesis? How many subjects in the permutation samples (with or without replacement)? How many permutation samples? Examine histogram of TS* with TS (the observed TS) Compute 2 p-values if possible: one-tailed, two-tailed Interpret results

Extra assignment Read the article by Zentner et al. (2007). If an hypothesis is tested by either bootstrap or permutation test, describe in details: the hypothesis, How the hypothesis was operationalized The procedure, The results The interpretation of the results.