1. PS215: Methods in Psychology II W eek 8

2. 2 Next Friday (Week 9) Evaluating research, class test First ten minutes of lecture (2.05-2.15) Please come a little early Please sit one seat space apart if possible Please do not talk once seated, until the test finishes There will be a lecture after the test

3. 3 Learning objectives specific contrasts are sometimes more useful than ANOVA main effects linear contrasts and pairwise comparison are important examples of contrasts effect size can be more relevant than significance multiplicity affects the interpretation of results distinction between planned and unplanned comparisons affects interpretation of p-value

4. 4 Study: Development of motor skill 50 children at five ages (11, 12, 13, 14, 15) record how well they play a new video game Is Age a good predictor of Game Score? Age1112131415 Score2530405055 Example from Rosenthal, Rosnow, and Rubin (2000)

5. 5

6. 6 ANOVA SourceSSdfMSFp ------------------------------------------------------------------- Age levels6,50041,6251.03.40 Within error70,875451,575 not significant! Should we conclude that age is not a useful predictor?

7. 7 ANOVA main effect did not use information about the order of the ages  ANOVA tests an unfocused question "any differences among the five age levels“ A more focused question  a more powerful test

8. 8 A specific contrast Choose –a weight for each level –weights reflect the contrast you want to test –weights add up to zero Age 1112131415 Contrast-2-1012 The contrast weight represents a specific model  the form you expect the relationship to take

9. 9 ANOVA- scores are different at different ages Linear contrast - scores go up in a straight line as age increases In this example, the linear contrast is statistically significant: t(45) = 2.02, p =.025

10. 10 Pairwise comparisons Overall main effect often not an especially interesting hypothesis Week 5 ANOVA tested whether the average comfort score was different for different drugs (main effect of 'Drug') Effect significant, but what can you conclude? "The drugs did not all have the same effect"

11. 11 Pairwise comparisons A more interesting question would be: 'Is aspirin more effective than tylenol?‘ When two groups are compared, it's called a pairwise comparison You can express a pairwise comparison as a contrast too: Drug Asprin Tylenol Nuprin Bufferin +1 -1 0 0

12. 12 Effect size If asprin is significantly better than tylenol, should we stop ordering tylenol for the pharmacy? Significance level (p-value) & sample size a very large sample can detect tiny effects too small a sample can miss even a large effect A very small p (eg. p =.001) does not in itself mean a strong effect Significance and effect size are different things

13. 13 To measure effect size d= M1 – M2   Where: M1 and M2 are the respective group means  is an estimate of population s.d. 0.2 is "small";0.8 is a "large" effect (Cohen, 1977)

14. 14 Multiplicity Take 15 measures of individual differences Correlate each with all the others There will be 105 different correlations So we expect 5 to reach the 5% p-value (.05) even if there are no real relationships

15. 15 Not appropriate to claim statistical significance for results in such circumstances Choice use a stricter, more conservative, criterion attempt to replicate your result

16. 16 More conservative criterion Bonferroni adjustment For 105 comparisons set required p-value to 0.05 / 105 Simple approach, wide applicability

17. 17 Replication Does the result continue to appear? If it is real, it should appear again in another study Meta-analysis takes this method further by aggregating results from several studies

18. 18 Planned and unplanned comparisons Planned (“a priori”) contrast envisaged at the outset follows from the logic of the study design Treat significance values straightforwardly

19. 19 Unplanned comparisons Unplanned (“post hoc” tests) chosen on the basis of looking at the data often – is an unexpected difference or pattern statistically reliable? Multiplicity issue -- even if you actually do just one, effectively you looked at them all

20. 20 Unplanned comparisons Choice use a stricter, more conservative, criterion Bonferroni adjusted tests Special purpose tests eg. Tukey HSD attempt to replicate your result

21. 21 Learning objectives specific contrasts are sometimes more useful than ANOVA main effects linear contrasts and pairwise comparison are important examples of contrasts effect size can be more relevant than significance multiplicity affects the interpretation of results distinction between planned and unplanned comparisons affects interpretation of p-value

22. 22 Getting a contrast in SPSS Syntax window (start setting up ANOVA, then choose paste) For a two way ANOVA IVs a(2) x b(4) DV y To contrast the four means within b Note F-ratio for doing this is bigger than if collapse b as a one –way, cos including the extra predictor a will reduce error variance glm y by a b /contrast(b)= special (0 0 1 -1). Actually, the following is what you get from GLM if you set up a two-way ANOVA, and the same contrast can be added. UNIANOVA y BY a b /METHOD = SSTYPE(3) /INTERCEPT = INCLUDE /CRITERIA = ALPHA(.05) /DESIGN = a b a*b /contrast(b) = special (0 0 1 -1).