1. Oneway/Randomized Block Designs Q560: Experimental Methods in Cognitive Science Lecture 8

2. Loftus and Palmer (1974) “How fast were the cars going when they ____ each other?” (hit, bumped, smashed) Reconstructive Memory:

3. The problem with t-tests… We could compare three groups with multiple t- tests: M1 vs. M2, M1 vs. M3, M2 vs. M3 But this causes our chance of a type I error (alpha) to compound with each test we do Testwise Error: Probability of a type I error on any one statistical test Experimentwise Error: Probability of a type I error over all statistical tests in an experiment ANOVA keeps our experimentwise error = alpha

4. What is ANOVA? In ANOVA an independent or quasi-independent variable is called a factor. Factor = independent (or quasi-independent) variable. Levels = number of values used for the independent variable. One factor  “single-factor design” More than one factor  “factorial design”

5. A example of a two-factor design: An example of a single-factor design:

6. What are we interested in? Two interpretations: 1)Differences are due to chance. 2)Differences are real.

7. ANOVA Test Stastic Remember the t statistic: t = actual difference between sample means difference expected by chance ANOVA test statistic (F-ratio) is similar: F = actual variance between sample means variance expected by chance Variance can be calculated for more than two sample means … An example:

8. The Logic of ANOVA Hypothetical data from an experiment examining learning performance under three temperature conditions: Treatment 1 50° (sample 1) Treatment 2 70° (sample 2) Treatment 3 90° (sample 3)

9. The Logic of ANOVA Looking at the data, there are two kinds of variability (variance): -Between treatments -Within treatments Variance between treatments can have two interpretations: -Variance is due to differences between treatments. -Variance is due to chance alone. This may be due to individual differences or experimental error.

10. The Logic of ANOVA

11. F = variance between treatments variance within treatments Another way of expressing it: F = treatment effect + chance chance F-ratio compares between and within variance as follows:

12. The Logic of ANOVA F = treatment effect + chance chance If there is no effect due to treatment: F  1.00. If there is a significant effect due to treatment: F > 1.00. The denominator of the F-ratio is also called the error term (it measures only unsystematic variance).

13. ANOVA Notation

14. What do all the letters mean? k = number of levels of the factor (i.e. number of treatments) n = number of scores in each treatment N = total number of scores in the entire study T = X for each treatment condition G = “grand total” of all the scores We also need SS, M, and X 2.

15. ANOVA Notation What are the calculations we need to do? 1) 2) 3) 4)

16. 1) Analysis of Sum of Squares: Total: Within: Between: SS total = X 2 - G2NG2N SS within = SS inside each treatment SS between =  - T2nT2n G2NG2N Note: SS total = SS within + SS between.

17. 1) Analysis of Sum of Squares: Total: Within: Between: SS total = X 2 - G2NG2N SS within = SS inside each treatment SS between =  - T2nT2n G2NG2N Note: SS total = SS within + SS between. Just Remember: SS within = SS total - SS between.

18. 2) Analysis of Degrees of Freedom: Total: Within: Between: Note: df total = df within + df between. df total = N-1 df between = k-1 df within = df in each treatment = N - k

19. 3) Calculation of Variances (MS) and of F-ratio: Note: In ANOVA, variance = mean square (MS) MS between = SS between df between MS within = SS within df within F = MS between MS within

20. Summary of ANOVA data:

21. F distribution In our example, the value for the F-ratio is high (11.28). Is this value really significant? Need to compare this value to the overall F distribution. Note: 1)F-ratios must be positive. 2)If H 0 is true, F is around 1.00. 3)Exact shape of F distribution will depend on the values for df.

22. F distribution Shape of the F distribution for df = 2, 12:

23. F distribution Let’s take a look at an F distribution table: degrees of freedom: denominator degrees of freedom: numerator 1 2 3 4 5 6

24. Hypothesis Testing with ANOVA Step 1: Hypotheses H 0 : all equal; H 1 : at least one is different Step 2: Determine critical value F ratios are all positive (only one tail) Need: df B and df W Step 3: Calculations SS B and SS W MS B and MS W F Step 4: Decision and conclusions And maybe a source table

25. Hypothesis Testing with ANOVA Data for three drugs designed to act as pain relievers: Placebo Drug A Drug B Drug C

26. Step 1: State hypotheses H 0 :  1 =  2 =  3 =  4. H 1 : At least one  is different. Step 2: Determine the critical region Set =.05 Determine df. df total = N-1 = 20-1= 19 df within = N-k = 20-4 = 16 df between = k-1 = 4-1 = 3

27. For the data given in the example: df = 3,8

28. Step 3: Calculate the F-ratio for the data 1)Obtain SS between and SS within. 2)Use SS and df values to calculate the two variances MS between and MS within. 3)Finally, use the two MS values to compute F- ratio.

29. 1) Sum of Squares Between: Total: Within:

30. 2) Mean Squares Between: Within: 3) F-Ratio

31. Step 4: Decision and Conclusion F obt exceeds F crit Reject H 0 We must reject the null hypothesis that all of the drugs are the same, F(3,16) = 8.33, p<.05 Summary Table: SourceSSdfMSF Between Within Total 50 32 82 3 16 19 16.67 2.00 8.33*

32. SourceSSdfMSF Between Within Total k-1 N-k N-1 Source Table for Independent-Measures ANOVA

33. Let’s visualize the concepts of between- treatment and within-treatment variability.

35. What are the corresponding F-ratios? F = MS between MS within Experiment A: F = 56/0.667 = 83.96 Experiment B: F = 56/40.33 = 1.39

36. Randomized Block Designs

37. The Logic of ANOVA F = treatment effect + chance chance If there is no effect due to treatment: F  1.00. If there is a significant effect due to treatment: F > 1.00. The denominator of the F-ratio is also called the error term (it measures only unsystematic variance).

38. Two Types of ANOVA Independent measures design: Groups are samples of independent measurements (different people) Dependent measures design: Groups are samples of dependent measurements (usually same people at different times; also matched samples) “Repeated measures” With t-tests, we used different formulae depending on the design…this is also true of ANOVA

39. The Logic of ANOVA

40. Independent Measures Differences between groups could be due to Treatment effect Individual differences Error or chance (tired, hungry, etc) Differences within groups could be due to Individual differences Error or chance A repeated-measures design removes variability due to individual differences, and gives us a more powerful test

41. Repeated Measures In a repeated-measures design, the same people are tested in each treatment, so differences between treatment groups cannot be due to individual differences So, we need to estimate differences between individuals to remove it from the denominator Then we will have a purer measure of the actual treatment effect (if it exists)

42. Partitioning of Variance/df Total Variance Between-treatments variance: 1)Treatment effect 2)Error or chance (excluding indiv differences) Within-treatments variance: 1)Individual diffs 2)Error or chance Between-subjects variance: 1)Individual diffs Error variance: 1)Error or chance (excluding indiv differences)

43. PersonBaselineTime 1Time 2Time 3 Person Totals ABCDEABCDE 3020030200 4311143111 6343463434 7654376543 P = 20 P = 12 P = 8 n = 5 k = 4 N = 20 G = 60 T = 5 SS=8 T = 10 SS=8 T = 20 SS=6 T = 25 SS=10 Example: Number of errors on a typing task while coffee is consumed We also compute person totals to get an estimate of individual differences

44. Sum of Squares: Stage 1 First step is identical to independent-measures ANOVA Total: Between: SS total = X 2 - G2NG2N SS between =  - T2nT2n G2NG2N Within: SS within = SS total - SS between df total = N-1 df between = k-1 df within = N-k

45. Sum of Squares: Stage 2 In the second stage, we simply remove the individual differences from the denominator of the F-ratio

46. Mean Squares and F-ratio Now we just substitute MS error into the denominator of F:

47. SourceSSdfMSF Between Within b/w subjects Error Total k-1 N-k n-1 (N-k)-(n-1) N-1 Source Table for Repeated-Measures ANOVA

48. PersonBaselineTime 1Time 2Time 3 Person Totals ABCDEABCDE 3020030200 4311143111 6343463434 7654376543 P = 20 P = 12 P = 8 n = 5 K = 4 N = 20 G = 60 T = 5 SS=8 T = 10 SS=8 T = 20 SS=6 T = 25 SS=10 Example: Number of errors on a typing task while coffee is consumed

49. Step 1: State hypotheses H 0 :  1 =  2 =  3 =  4. H 1 : At least one  is different. Step 2: Determine the critical region Set =.05 Determine df. df total = N-1 = 20-1= 19 df within = N-k = 20-4 = 16 df between = k-1 = 4-1 = 3 df b/s = n-1 = 5-1 = 4 df error = df within - df b/s =16-4 = 12 F crit (3,12)=3.49

50. Step 3: Calculate the F-ratio for the data 1)Obtain SS between and SS error. 2)Use SS and df values to calculate the two variances MS between and MS error. 3)Finally, use the two MS values to compute F- ratio.

51. 1) Sum of Squares, Stage 1: Between: Total: Within:

52. 1) Sum of Squares, Stage 2: Between subjects: Error:

53. 2) Mean Squares Between: Error: 3) F-Ratio Note: These are the same data we used in the independent- measures ANOVA on Thurs, by changing from an independent to repeated measures design, weve gone from F=8.33 to F=24.88

54. Step 4: Decision and Conclusion F obt exceeds F crit Reject H 0 We must reject the null hypothesis that coffee has no effect on errors, F(3,12) = 24.88, p<.05 Summary Table: SourceSSdfMSF Between Within b/w Ss Error Total 50 32 24 8 82 3 16 4 12 19 16.67 0.67 24.88*

55. Advantages of Repeated-Measures Remember: variance (=“noise”) in the samples increases the estimated standard error and makes it harder for a treatment-related effect to be detected. (Remember how we added up two sources of variance in the independent-measures design.) Repeated-measures design reduces or limits the variance, by eliminating the individual differences between samples.

56. Problems with Repeated Measures Carryover effect (specifically associated with repeated-measures design): subject’s score in second measurement is altered by a lingering aftereffect from the first measurement. Examples: testing of two drugs in succession, motivation effects, etc. Important: Aftereffect from first treatment

57. Problems with Repeated Measures Progressive error: Subject’s score changes over time due to a consistent (systematic) effect. Examples: fatigue, practice Important: effect of time alone History: changes outside the individual that may be confounded w/ the treatment Maturation: changes within the individual that may be confounded w/ the treatment