Experimental Design & Analysis Hypothesis Testing and Analysis of Variance January 30, 2007 DOCTORAL SEMINAR, SPRING SEMESTER 2007

Outline Statistical inferences Null hypothesis testing Sources of variance Treatment variance vs. error variance Sums of squares

Statistical Inference Means The mean is a measure of central tendency μ or x Sum of squares The sum of squares refers to the sum of the squared deviations of a set of scores or values from their mean (1) Subtract the mean from each score (2) Square each resulting difference (3) Add up the squared differences Mean square, or variance Mean square of a sample is a measure of variability Divide sum of squares by the degrees of freedom σ2 or s2 Standard deviation σ or s

Statistical Inference Student's t-test is used to compare two groups when sample sizes are small (N < 30), but with larger samples the normal curve z test is used Let E be the experimental condition and let C be the control condition. Let m be the means, s the standard deviations, and n be the sample size t = (mE - mC) / √[(s2E + s2C) / n ] The critical value is the value found in t-tables

Statistical Inference Hypothesis testing Inferences about the population based on parameters estimated from a sample μ1 = μ2 = μ3 = etc. suggests that there are no treatment effects Observing an effect requires falsifying the null hypothesis H0 How different are two mean scores?

Statistical Inference Differences due to treatment Systematic source of difference by virtue of experimental condition Differences due to error Random source of difference because participants are randomly assigned to groups

Statistical Inferences Evaluation of null hypothesis differences between experimental groups of subjects differences among subjects within same groups experimental error = 1 treatment effects + experimental error experimental error = 1 ?

Null Hypothesis Significance Testing 5 steps of NHST State H0, H1, and  level Determine rejection region and state rejection rule Compute test statistic Make a decision (reject H0 or fail to reject H0) Conclude in terms of the problem scenario

Statistical Inference When H0 is true, F value = 1, although random variation can result in it being > 1 or < 1 When H0 is false, F expected >1 Since random variation can account for why F > 1, the problem we face is determining how much >1 it must be in order for us to conclude, on the basis of improbability, that its size indicates a real difference in the treatments The F distribution enables us to determine how improbable an experimental outcome is under the assumption our null hypothesis is true. If the value for F is so large that it is improbable (p < .05 or p < .01), given that our null hypothesis is true, we will conclude our null hypothesis is false and that the hypothetical population means are not equal

Sources of Variance Partitioning variance In order to distinguish systematic variability (treatment effects) from random error, we must partition variance by calculating component deviation scores Total deviation Between-groups deviation Within-groups deviation

Component Deviations Total deviation Within-group deviation 2 4 6 8 10 Y2 Y5,2 YT - Y2 - YT Total deviation Within-group deviation Between-group deviation From Keppel & Wickens, p. 23

Sources of Variance What is the grand mean, YT? What are the group means? Treatment A1 Treatment A2 Treatment A3 Participant 1 16 Participant 2 18 Participant 3 10 Participant 4 12 Participant 5 19 Participant 1 4 Participant 2 7 Participant 3 8 Participant 4 10 Participant 5 1 Participant 1 2 Participant 2 10 Participant 3 9 Participant 4 13 Participant 5 11 A1 = ? A2 = ? A3 = ? T = ? Y2 = ? Y3 = ? YT = ? Y1 = ? From Keppel & Wickens, p. 24

Calculating Sums of Squares Identify bracket terms* Using sums [Y] = ΣYij2 = 162 + 182 + … + 132 + 112 = 1,890 [A] = ΣAj2 /n = (752 + 302 + 452)/5 = 1,710 [T] = ΣT2/an = 1502/15 = 22,500/15 = 1,500 Using means [A] = nΣYj2 = (5)(152 + 62 + 92) = 1,710 [T] = anΣYT2 = (3)(5)102 = 15 (100) = 1,500 See authors’ note on bracket terms, Keppel and Wickens, p. 31.

Sums of SquaresTotal = Sums of SquaresBetween + Sums of SquaresWithin Sources of Variance Calculate sums of squares Total sums of squares = Σ(Yij – YT)2 Between sums of squares = Σ(Yj – YT)2 Within sums of squares = Σ(Yij – Yj)2 Sums of SquaresTotal = Sums of SquaresBetween + Sums of SquaresWithin

Calculating Sums of Squares Write sums of squares using bracket terms SST = [Y] – [T] = 1,890 – 1,500 = 390 SSA = [A] – [T] = 1,710 – 1,500 = 210 SSS/A = [Y] – [A] = 1,890 – 1,710 = 180

Analysis of Variance Sums of squares provide the building blocks for analysis of variance and significance testing Analysis of variance involves 2 steps Calculate variance estimates, known as mean squares, by dividing component sums of squares by degrees of freedom Ratio of between-group mean square and within-group mean square provides F statistic

Analysis of Variance Source df Mean Square F A a-1 SSA/dfA MSA/ MSS/A S/A a (n-1) SSS/A/dfS/A Total an-1 a = # of levels of factor A n = sample size of a group df = degrees of freedom Check F value in table of critical values organized by degrees of freedom to determine significance level Numerator degrees of freedom = a-1 Denominator degrees of freedom = a(n-1)