1. Review of Stats Fundamentals
Emily Falk, Ph.D.
University of Michigan
(with slides from Thad Polk, Sky Chafin, Guillaume Flandin, Jeff Grynaviski, Luiz Pessoa, and Tor Wager)

2. Lecture Outline Hypothesis Testing Tomorrow: 2. General Linear Model
Null hypothesis vs. alternative hypothesis; Testing hypotheses about population based on a sample; Sampling distributions & Central Limit Theorem; t-statistic, t-distribution, t-tests, p-values; Interpreting results, Type I error, Type II error; One-tailed vs. two-tailed tests; Multiple comparisons
Tomorrow:
2. General Linear Model
Regression, multiple regression, model fitting, assumptions, matrix notation, design matrix, example, issues
Overview of fMRI data analysis
Build design matrix, fit model to get betas, contrasts, statistical parametric maps, threshold for significance (correcting for multiple comparisons)

3. What is true about the world?

4. Does fMRI competence make people hotter?

5. Are fMRI researchers rated as more attractive than your average academic?

6. Hypothesis Testing 1. Define the Research Hypothesis.
A Research or Alternative Hypothesis is a statement derived from theory about what the researcher expects to find in the data.
2. Define the Null Hypothesis.
The Null Hypothesis is a statement of what you would not expect to find if your research or alternative hypothesis was consistent with reality.
3. Conduct an analysis of the data to determine whether or not you can reject the null hypothesis with some pre-determined probability.
If you can reject the null hypothesis with some probability, then the data is consistent with the research hypothesis.

7. Population vs. Sample Hypotheses are about population parameters:
E.g., The true population mean (not the sample mean) for fMRI researchers is greater than zero on the attractiveness index
E.g., the true population mean (not the sample mean) is 1000ms

8. Population vs. Sample
But data is just a sample, not the entire population
The population is described by the population mean and the population variance, but these are unknown.
The sample is described by the sample mean and the sample variance. These are known.
Recall: Mean = Average of observations
Recall: Variance = how variable the observations are
Standard deviation = sqrt of the variance;
standard error = standard deviation of the sampling distribution of means

9. Population vs. Sample
The sample mean and variance provide estimates of the true mean and variance of the entire population
But in general they won’t be right

10. Population vs. Sample
To make statistical inferences, we need to know how likely the observed sample stats are, assuming the null hypothesis is true
If observed values are sufficiently unlikely, we’ll reject the null hypothesis in favor of the alternative hypothesis

11. How do we judge the probability of the observed results?
Assume that our null hypothesis is that the population mean is zero
Observed sample mean equals two.
Want to know:
How likely is it to draw a sample with mean of two, if population mean is actually zero?
If this is very unlikely, then we would reject the null hypothesis and claim that the population mean isn’t zero

12. The Central Limit Theorem
As the sample size increases, the distribution of sample means approaches a normal (bell-shaped) distribution.
Important because we know a lot about normal distributions
The larger the sample size, the tighter the shape of the bell.
Mean  population mean
Variance  population variance/sample size
Surprisingly, this is true regardless of the shape of the population distribution

13. Central Limit Effect Histograms of sample means

14. Central Limit Effect Histograms of sample means

15. Central Limit Effect Histograms of sample means

16. Central Limit Effect Histograms of sample means

17. When I say…
You say…
Central
When n goes up…
Big n
Limit
We get a bell
Known shape
The bell gets tight
Tight bell

18. CLT Magic: we can take something unruly and turn into something with rules.
The sampling distribution of means takes on a known shape, regardless of the shape of the underlying distribution of individual observations

19. Why do we care?
The central limit theorem provides a way of summarizing our uncertainty about the sample mean. It therefore allows us to make probabilistic statements about the null hypothesis. The key is that we have estimates of the population mean and the population variance based on the sample. The mean and variance define the shape of the bell, and once we know those things, we can translate to something we know a lot about.

20. Related to this, we can prove that the following statistic t = Sample Mean – Population Mean assuming Null Hypothesis Sample Standard Deviation / n follows the t distribution with n-1 degrees of freedom.
Standard Error
Note, The larger the sample size, the tighter the shape of the bell. The variability of the sampling distribution of means gets smaller as we have more observations.

21. The t-distribution df = inf df = 3 df = 1
The t-distribution is a symmetric, bell-shaped curve much like the normal distribution.
df = inf
df = 1
df = 3

22. Interpreting the t-statistic
The x-axis of the t-distribution helps us interpret our t-stats. In effect, the t statistic measures how far the sample mean is from the null hypothesis in standardized units.
How big of an effect do you see in your sample relative to the natural variability in the world?

23. Interpreting the t-statistic cont.
As hypothesis testers, we only want to reject the null hypothesis if we are very confident that the null hypothesis is mistaken.
The standard is that we reject the null if only 5% of scores would be more extreme.
Note: when we refer to statistical significance, we say that a finding is statistically significant if we can reject the null hypothesis at the 95% level.

24. Interpreting the t-statistic cont.
For a given number of degrees of freedom, the t-distribution tells us how large the t-statistic must be in order to reject the null.
We call that number the “critical value” of the t-statistic.
If the value of the t-statistic calculated from the data is greater than this critical value, then we “reject the null hypothesis.”

25. Example =5% tcrit = 1.98 tobs = 6.6
Suppose our null hypothesis is that the true population mean X is 0 (no change in attractiveness). The sample mean is 2; The sample standard deviation is 3; There are 100 observations. Step 1. We need to establish our “critical value.” We wish to reject the null hypothesis if we are 95% certain that it is false. For 100 observations, it turns out that t-values with an absolute value greater than 1.98 occur less than 5% of the time by chance (assuming the null hypothesis is true) Step 2. The t-statistic = ( 2 – 0 ) / ( 3/ 100 ) = 2 / .3 = 6.6 Step 3. Compare the t-statistic with the critical value. If the t-statistic is greater than the critical value, then you can reject the null hypothesis. In this case, 6.6 is greater than 1.98, so we can reject the null hypothesis that X is zero.
=5%
tcrit = 1.98
tobs = 6.6

26. P-values
P-Values: Rather than using a critical value of the t-statistic, it is possible to determine a p-value based on the number of degrees of freedom and the t-statistic derived from the data The p-value is the probability of rejecting the null hypothesis even though it is true (a false alarm). If the p-value is less than .05, or whatever we define to be our pre-determined cut-off, we say the result is statistically significant.
How much area is more extreme than this point?

27. If we retain H0, what can we conclude?
The decision to retain H0 does not mean that it is likely that H0 is true.
This is the fallacy of accepting the null hypothesis
We simply do not have sufficient evidence to reject the null hypothesis.
Retaining the null hypothesis is a weak decision.
If we reject H0, what can we conclude?
Rejecting the null hypothesis means that H0 is probably false, a strong decision.

28. When I say…
You say… T
When t’s extreme…
Big t
When t is small
Small t
test
Reject the null
Improbable null
Who knows
Low info

29. Two types of error in hypothesis testing
False positive rate
False negative rate  
Do not
reject H0  

30. Two-tailed Test
The alternative hypothesis states that the population parameter may be either less than or greater than the value stated in H0.
The critical region is divided between both tails of the sampling distribution

31. Two-tailed Test
This type of test is used when you predict some effect, but not the direction of the effect
For example, in most cases in which the performance of a group is compared to a known standard, it would be of interest to discover that the group is superior or inferior.

32. One-tailed Test
The alternative hypothesis states that the population parameter differs from the value stated in H0 in one particular direction.
The critical region is located only in one tail of the sampling distribution.

33. One-tailed Test
Upper-tail Critical
Lower-tail Critical

34. One-tailed Test
The advantage of a one-tailed test is that it is more sensitive to rejecting the null hypothesis in the predicted direction than a two-tailed test.
The major disadvantage of a one-tailed test is that it precludes any chance of discovering that reality is just the opposite of what the alternative hypothesis says.

35. Multiple Comparisons
Suppose you perform multiple statistical tests simultaneously
If you perform enough tests, one is likely to appear significant (p<0.05) by chance alone, even if it isn’t
Testing 100 different coins for fairness by flipping each 10 times
Or looking for neural effect at 20,000 different voxels in the brain
To deal with this multiple comparisons problem, need to adopt a stricter threshold
Instead of p<0.05, use p<0.05/#tests (Bonferroni)
Perform fewer tests (e.g., in a small region of interest)
Gaussian random field theory (we’ll get back to this next week)

36. When I say…
You say…
False
Type
With lots of tests…
When I say
With strict thresholds
Trade
More
Positive
One
False positives
You say
Negatives
False negatives
Offs
Tuesday

37. Lecture Outline Hypothesis Testing QUESTIONS?
Null hypothesis vs. alternative hypothesis; Testing hypotheses about population based on a sample; Sampling distributions & Central Limit Theorem; t-statistic, t-distribution, t-tests, p-values; Interpreting results, Type I error, Type II error; One-tailed vs. two-tailed tests; Multiple comparisons
QUESTIONS?