1. Review bootstrap and permutation

2. Main points Definition of confidence intervals Definition of p-value
Why use bootstrap, why use permutation tests?
What are the differences between bootstrap and permutation tests?
How to use bootstrap?
How to use permutation tests?

3. Definition of confidence intervals
A confidence interval represents the precision of the estimation of a test statistic
If the same experiment was replicated a hundred times, the 95% CI would, on average, contain the estimated TS in 95 of these samples.

4. Definition of p-value
A p-value represents the probability of observing a TS this extreme or more extreme if the null hypothesis is true

5. Why use bootstrap, why use permutation tests?
Test statistic
Theoretical hypothesis
Operational hypothesis
Null hypothesis
P-value OR CI
Underlying distribution
If one of those is unusual or unknown, bootstrap or permutation is useful 5

6. Test statistics Is not implemented in standard software
Is estimated by a single value in the whole sample
Is a relationship between two TS

7. Null hypothesis
Is different from zero in a way that is hard to quantify
Example: If you are expecting a certain variable to explain more than 50 percent of a different variable, such that the r squared is greater than 50. We would want to bootstrap the r-statistics and see whether the confidence interval includes 50, or values below 50. If the confidence interval does not, then we can say that our hypothesis is supported. If it does, then we cannot reject the null.

8. Underlying distribution
Is unknown because TS was unknown
Is unknown because conditions of applications for parametric tests do not seem to be met
Is known for the null hypothesis but seems likely to be different for the alternative hypothesis p-value should be correct but CI will be incorrect.

9. Differences between bootstrap and permutation tests
estimates confidence interval, bias and standard error
Simulates data under the alternative hypothesis
Sampling is done with replacement of subjects
Many bootstrap samples because of replacement
Permutation tests
estimates p-value and distribution under the null.
Simulates data under the null hypothesis
Sampling is done without replacement of subjects
Finite number of potential permutation samples

10. Main points: how to How to bootstrap a test statistics
Determine the test statistic of interest (must be a single value)
What is randomly sampled? How many subjects in the bootstrap samples (with or without replacement)? How many bootstrap samples
Examine histogram of TS* with TS (the observed TS), average of TS*, and boundaries of percentile and bca CI
Interpret results

11. Main points: how to How to use a permutation test
Determine the test statistic of interest (must be a single value)
What must be shuffled in order to simulate what happens under the null hypothesis? How many subjects in the permutation samples (with or without replacement)? How many permutation samples?
Examine histogram of TS* with TS (the observed TS)
Compute 2 p-values if possible: one-tailed, two-tailed
Interpret results

12. Extra assignment Read the article by Zentner et al. (2007).
If an hypothesis is tested by either bootstrap or permutation test, describe in details:
the hypothesis,
How the hypothesis was operationalized
The procedure,
The results
The interpretation of the results.