The t distribution and the independent sample t-test

How do we test to see if the means between two sample populations are, in fact, different? Note that we could also ask if the sample means are different for the same sample populations at different times…but more on that later.

Comparing mean difference while accounting for variability in samples Image: http://www.socialresearchmethods.net/kb/stat_t.php

t-test Null and Alternative Hypotheses Null Hypothesis: H0: μ1 = μc Alternative Hypotheses: H1: μ1 < μc H2: μ1 > μc H3: μ1 ≠ μc

One and Two-Tailed Tests: Defining Critical Regions Step 1: form your hypothesis Step 2: calculate your t-statistic Step 3: plot your t-value on the appropriate curve to get the p-value.

Why use the t distribution? Useful for testing mean differences when the N is not very large In very large samples, a t-test is the same as a z-score test.

Z-distribution versus t-distribution

t distribution As the degrees of freedom increase (towards infinity), the t distribution approaches the z distribution (i.e., a normal distribution) Because N plays such a prominent role in the calculation of the t-statistic, note that for very large N’s, the sample standard deviation (s) begins to closely approximate the population standard deviation (σ) http://www.econtools.com/jevons/java/Graphics2D/tDist.html

The independent sample t-test Where: M = mean SDM = Standard error of the difference between means N = number of subjects in group s = Standard Deviation of group df = degrees of freedom Essentially, the t-value is the difference between group means (mean difference), divided by the variability of the two groups (standard error of difference).

Degrees of freedom Example: Choosing 10 numbers that add up to 100. d.f. = the number of independent pieces of information from the data collected in a study. For a one-sample t-test, the degrees of freedom = number of measurements – 1. Example: Choosing 10 numbers that add up to 100. This kind of restriction is the same idea: we had 10 choices but the restriction reduced our independent selections to N-1. In statistics, further restrictions reduce the degrees of freedom. In the independent sample t-test, we deal with two means; our degrees of freedom are reduced by two (df = n1 + n2 – 2)

Assumptions Underlying the Independent Sample t-test Assumption of Normality Variables are normally distributed within each group Assumption of Homogeneity of Variance Variation of scores in the two groups should not be very different. Normality: we could transform the variables like we saw a few weeks ago Homogeneity of Variance: we can test to see if we meet this assumption

What if the Variances are not equal? We can test to see if the variances are significantly different between groups (equality of variance test) If they *are* significantly different, we can use a modified t-test that corrects for this problem (Welch’s t-test) Essentially, the Welch’s t-test uses a different estimation of the degrees of freedom and also avoids using the pooled variance. Uses Welch-Satterthwaite equation. The key is that the denominator uses a different estimate of the variance (since we cannot just assume that both distributions are equal now), and the degrees of freedom are reduced– making this test highly conservative.

Conducting an independent sample t-test 1) State your hypothesis (1 or 2-tailed?) 2)Check the assumption of equal variances 3) if equal, use a standard t-test If unequal, use a modified (welch’s) t-test

Statistical Significance vs. Practical Significance

Calculating Effect Size Note that the SD in the effect size calculation is the pooled (combined) standard deviation for both groups Effect size allows you to compare relative effect of one test to another (using same dependent variable) Conceptually, think of effect size as increasing the probability that you could guess which group a given person came from. Values of .8 or greater are usually large effects, .5 medium, .2 small. Effect size is a measure of practical significance, not statistical significance. Cohen’s d is a common effect size calculation.

Effect Size Example Sample of 10,000 participants. Average “skill with computers” Men = 6.5 (S.D. = 3.3) on a 10-point scale. Women = 5.9 (S.D. =3) The t-test is highly sig (p < .001) Effect size = .19 What if we have a fairly large effect size, but its just not statistically significant? Perhaps we just need more data– but how much??

How Big of A Sample? We can use a power analysis to determine how big our sample needs to be in order to reject the null that there is no difference. Requires an idea of the two means and their S.D.’s (usually through pre-testing or literature) In STATA, use “sampsi” command. A power analysis is based on our best guesses about our sample(s). It is not a guarantee of significance.

t-tests using STATA t-test.do auto.dta dataset grades.dta dataset