1. KNR 295 Research Design Slide 1 Design Ch. 6 – Internal Validity

2. KNR 295 Research Design Slide 2 Internal Validity  Cause and effect  Cause… “Function: transitive verb Inflected Form(s): caused; caus·ing Date: 14th century 1 : to serve as a cause or occasion of : MAKE 2 : to effect by command, authority, or force - caus·er noun” *from Merriam-Webster online

3. KNR 295 Research Design Slide 3 Internal Validity  Cause and effect  Effect… “Function: noun Etymology: Middle English, from Middle French & Latin; Middle French, from Latin effectus, from efficere to bring about, from ex- + facere to make, do -- more at DO Date: 14th century 1 something that inevitably follows an antecedent (as a cause or agent) ” *from Merriam-Webster online

4. KNR 295 Research Design Slide 4 Internal Validity  Cause and effect Internal validity is a “zero-generalizability concern”

5. KNR 295 Research Design Slide 5 Internal Validity  Cause and effect  So…did the treatment you used cause the measurement you took to be what it was?  Or…to what extent did the treatment you used influence the measurement you took?

6. KNR 295 Research Design Slide 6 Internal Validity  Establishing cause and effect  Example: Does beer make you happy?  Temporal precedence beer happiness

7. KNR 295 Research Design Slide 7 Internal Validity  Establishing cause and effect  Example: Does beer make you happy?  Covariation of cause and effect If beer then happy If not beer then not happy If more beer then more happy If less beer then less happy

8. KNR 295 Research Design Slide 8 Internal Validity  Establishing cause and effect  Example: Does beer make you happy?  No plausible alternative explanations “Never drink alone”…social causes? To drink, you need to be flush…it’s an economic difference? Drinkers experience smoke too…it’s all down to ciggies? When you drink you go to the loo more often…something to do with bladder swelling/emptying? Rank – most plausible? - least plausible?

9. KNR 295 Research Design Slide 9 Internal Validity  Single group threats  Sticking with the beer idea…  Imagine you take a group of students, and give them a dose of beer. You might measure their happiness just once at the end, or you might measure them both before and after the beer dose…

10. KNR 295 Research Design Slide 10 Internal Validity  Single group threats  Sticking with the beer idea… Give beer Measure happiness Give beer

11. KNR 295 Research Design Slide 11 Internal Validity  Single group threats  Sticking with the beer idea…  History Threat  It’s not the beer. It’s some other major historical event that influenced your group’s happiness…  Perhaps on the day of the test, Saddam decided to pack it in, and everyone simultaneously found out that the recession was a big mass-induced dream state, awaking to find that the world was infinitely happier than they had first thought? Plausible?

12. KNR 295 Research Design Slide 12 Internal Validity  Single group threats  Sticking with the beer idea…  Maturation Threat  As people get older, they get happier. You took your first measurement during your group’s teenage angst years, then waited until the wisdom, wonder and general delight of 40ish life settled in before taking the second measurement  [can be anything related to general “internal” change in people – any “big event” external to the person causing the change is generally regarded as a history threat] Plausible?

13. KNR 295 Research Design Slide 13 Internal Validity  Single group threats  Sticking with the beer idea…  Testing Threat  (Pre-post only)  Measuring happiness made participants focus on their degree of happiness – if they were unhappy, they resolved to buckle up, and cheer up. If they were happy they were made aware of this and were therefore delighted to be so well-adjusted Plausible?

14. KNR 295 Research Design Slide 14 Internal Validity  Single group threats  Sticking with the beer idea…  Instrumentation Threat (pre-post again)  A couple of possibilities…  The group are scored with different instruments on each occasion. The second is more sensitive than the first. The group’s happiness doesn’t change, but the scores arising from the instruments differ anyway  The person giving out the happiness instrument is such a happy-go-lucky lass that she infects everyone else with her sense of “joi de vivre” (experimenter as instrument)- the longer she’s around the happier you feel Plausible?

15. KNR 295 Research Design Slide 15 Internal Validity  Single group threats  Sticking with the beer idea…  Mortality Threat  As the study wears on and the beer starts to have its inevitable effect, many people are past the stage of being happy and are approaching that other unfortunate stage of influence…but they aren’t around for the happiness measurement at the end of the study Plausible?

16. KNR 295 Research Design Slide 16 Internal Validity  Single group threats  Sticking with the beer idea…  Regression Threat  [The Fugawe Tribe]  When you look at the scores of your group on the pre- test, you notice that they are unusually low…  This situation, where for any reason scores are just by chance suppressed on the first measurement, will probably result in artificially higher scores (in comparison to the first) on the second reading  It’s not that the treatment is having an effect, it’s that the scores are just going back to normal  Normally occurs when selecting a group by pre-test scores… Plausible?

17. KNR 295 Research Design Slide 17 Internal Validity  Single group threats  Sticking with the beer idea…  Regression Threat  You want to find out if the effect even works on really unhappy people. So you get 200 grad students to fill out a happiness questionnaire, and select only the lowest 25% for your study.  Lo and behold, after a good dose of beer, they all score higher on the hap-o-meter!  Why? Plausible?

18. KNR 295 Research Design Slide 18 Internal Validity  Solving single group threats  Add a control group  The only difference between the control group and the treatment group should be the presence or absence of the treatment  Sometimes this means you need multiple control groups  Often, it still fails…

19. KNR 295 Research Design Slide 19 Internal Validity  Multiple group threats  All are selection bias threats  The critical question is: “were the groups equal (on the measure of choice) at the start of the study”  The general case is simply a selection bias. Other specific differences follow

20. KNR 295 Research Design Slide 20 Internal Validity  Multiple group threats Single-group threats – taken care of now Multiple-group threats – still a problem

21. KNR 295 Research Design Slide 21 Internal Validity  Multiple group threats  More examples with the hoppy one…  Selection-history threat  Say you got your 2 groups from grad classes  On the day of the test, one of the groups’ classes has a nasty test while the other has a normal class  What is causing the difference in happiness now?

22. KNR 295 Research Design Slide 22 Internal Validity  Multiple group threats  More examples with the hoppy one…  Selection-maturation threat  Say you’re looking at the long-term effects of beer on happiness (not temporary as now)  Say also that one group (the drinkers) is selected from among the student population, while the other is selected from among the faculty  It might be easy to imagine that the older group is somewhat more stable with respect to the happiness measure than the younger

23. KNR 295 Research Design Slide 23 Internal Validity  Multiple group threats  More examples with the hoppy one…  Selection-testing threat  Generally, that one group is for some reason more sensitive to be influenced by the first test than the second  Say you get the beer group to volunteer from a pub, while the non-beer group volunteers from a teetotal social club  The pub group also (unknown to you) score higher on a neuroticism scale. This leads them to be more easily influenced by knowledge of their score on a happiness scale…and thus alters their responses second time around

24. KNR 295 Research Design Slide 24 Internal Validity  Multiple group threats  More examples with the hoppy one…  Selection-instrumentation threat  Say we have our happy-go-lucky experimenter from the single group threats operating here too, but she can’t do both groups (schedule can’t allow), so she gets her partner (who is rather gloomy…funny how that seems to happen) to monitor the other group

25. KNR 295 Research Design Slide 25 Internal Validity  Multiple group threats  More examples with the hoppy one…  Selection-mortality threat  Even if you add another (non-beer) group, it is quite probable that your non-beer group will be intact at the end of the study, while your beer group might suffer substantial drop-out

26. KNR 295 Research Design Slide 26 Internal Validity  Multiple group threats  More examples with the hoppy one…  Selection-regression threat  Say you select your volunteers from a pub on a weekday afternoon. The others are selected at random from grad research classes  Trouble is, folk who drink at a pub on a weekday afternoon tend to be a depressed lot…and thus are more likely to improve their scores on a subsequent post-test  [this is fiction, folks!]

27. KNR 295 Research Design Slide 27 Internal Validity  Solving Multiple group threats  Randomly assign to groups  Random assignment ensures no systematic difference between groups  We’re still not done yet though…

28. KNR 295 Research Design Slide 28 Internal Validity  Social interaction threats Single-group threats – taken care of by adding control group Multiple-group threats – taken care of by random assignment Social interaction threats

29. KNR 295 Research Design Slide 29 Internal Validity  Social interaction threats  One more time with the amber nectar (as Ben Franklin once said, “beer is proof that God loves us and wants us to be happy”)  Diffusion or imitation of treatment  Imagine you pluck all your participants from a college hall of residence. They are aware of each other’s participation, and the other group’s treatment. On seeing the beer group go off for their treatment, the non beer group think “that’s a good idea” and troop off to the liquor store…

30. KNR 295 Research Design Slide 30 Internal Validity  Social interaction threats  One more time…  Compensatory rivalry  In some way the groups become aware of each other’s treatments and of the measures, and something about them sets up a different behavior that influences the final measurement  For example, the non-beer set may for some reason not want to be judged as less happy” than the beer group second time around.

31. KNR 295 Research Design Slide 31 Internal Validity  Social interaction threats  One more time…  Resentful demoralization  Say the non-beer group are aware the beer group is getting the beer  Say also the non beer group enjoy a beer  They’re jealous  This unhappiness finds its way into the post-test

32. KNR 295 Research Design Slide 32 Internal Validity  Social interaction threats  One more time…  Compensatory equalization  Say the non-beer group are aware the beer group is getting the beer  Say also the non beer group enjoy a beer  Say the experimenters are aware of this  They treat the other group with extra kindness and comfort to offset the perceived slight at the lack of beer

33. KNR 295 Research Design Slide 33 Internal Validity  Social interaction threats  And one extra I’d add here (repeated from construct validity)  Experimenter bias  The belief in the treatment in some way causes the experimenter to behave differently around the two groups, thus turning the findings into a self-fulfilling prophecy

34. KNR 295 Research Design Slide 34 Introduction to design

35. KNR 295 Research Design Slide 35 Types of designs

36. KNR 295 Research Design Slide 36 Types of designs