Bootstrapping and Randomization Techniques Q560: Experimental Methods in Cognitive Science Lecture 15

Parametric and Nonparametric Methods Most of the tests we have examined so far have been parametric tests. Parametric tests must make assumptions about the distribution of the population, and estimate parameters of a theoretical probability distribution from statistics Nonparametric tests (distribution free) make no parameter assumptions; parameters are built from data, not the model

Parametric and Nonparametric Methods I lied about a couple of things related to parameters… “Enumerate” An example with Fisher’s Lady Tasting Tea…

The Binomial Distribution An example graph:

Probability and Frequency Graphs Example: For the population of scores shown below, what is the probability in a random draw of obtaining a score greater than 4? p(X>4) =

The Normal Distribution Diagram:

Sample Means Let’s do an example from a very small population of 4 scores: X: 2, 4, 6, 8

Sample Means We construct a distribution of sample means for n=2. 1. Step: Write down all 16 possible samples.

Sample Means 2. Step: Draw the distribution of sample means.

General Approach Original t-test: how abnormal is our mean difference relative to all possible mean differences? We built a chance model based on CLT, assumption of normality let’s us estimate SEM This is a general technique if the population is normal, and allows us to look up critical values from a table But we can compute a custom chance model for our particular observed data. If the distributions are not normal, we have to do this.

We might want a nonparametric test if: We don’t meet the assumptions for a parametric test We want to do tests on medians They aren’t just tools to break out if assumptions fail.... Generally, npar tests: Do not make assumptions about population parameters We just want to know the likelihood of the sample data falling as they did if the treatments are equally effective (random assignment to conditions is important factor)

Resampling Techniques: Pros: More liberal (and realistic) in their assumptions Distribution free…validity of test is unaffected by whether or not population distribution is known Cons: Tend to be less powerful…require more observations than parametric test to detect a true effect Note that this is still being debated (see papers on website) Computationally intensive

Resampling Techniques: Rather than assuming imaginary populations, they focus on the data at hand Divide into estimation methods, or custom chance models (desc/inferential) 1.Bootstrapping and Jackknifing 2.Randomization (Permutation) Tests 3.Sign Tests 4.Monte Carlo Techniques

Bootstrapping Parameters Make sure you have a random sample Rather than assuming our population is normal, we assume it is distributed exactly as our sample Sample w/ replacement from your “pseudo” population Create N random samples from your sample, and determine sampling distribution Example: Dependent-t, independent-t

Randomization Tests Different philosophy: primary hypothesis is “exchangeability”...if null is true, the arrangement of Ss to conditions is random Focus is on random assignment Note that this is philosophically different from NSHT We don’t talk about parameters or populations, but the manipulation in the experiment Rather than “different populations had the same mean” we use “different treatments had the same effect” Compare the observed data to all permutations

Randomization + Independent Samples If the null is true, the groups differ only by random assignment Compute observed mean difference t* Randomly assign scores to groups again from the full sample and compute mean difference; do this a large number of times, e.g. 10,000 How unlikely is t* under this chance distribution of random assignments? (i.e., how many are more extreme?)

Randomization and Dependent Samples Make sure you have a random sample If the null is true, the difference scores should be zero; +/- signs that are not zero should be chance Under null, each score has a 50/50 chance of being +/- Compute actual mean of difference scores, D* Select scores from the sample and randomly assign them a +/- sign…do this a large number of times, and this is your chance model. How unlikely is D* under this chance model?

References (on our website): Mewhort, D. J. K. (2005). A comparison of the randomization test with the F test when error is skewed. Behavioral Research Methods, 37, · Moore, D. S., G. McCabe, W. Duckworth, and S. Sclove (2003): Bootstrap Methods and Permutation Tests · Hesterberg, T. C., D. S. Moore, S. Monaghan, A. Clipson, and R. Epstein (2005): Bootstrap Methods and Permutation Tests, software.

Examples: S and R both have nice built-in routines for computing resampling statistics; SPSS does not. Let’s do some examples using the program Resampling.exe (available on our website) Paired samples comparison Oneway ANOVA with bootstrapping Bootstrapping correlation sig test (Data files are on our Website under Lecture 15)