1. Internal Validity Questions
Did the treatment cause a change in behavior?
Change in Behavior
Treatment
Causes

2. 8 Threats to Internal Validity (Cambell & Stanley, 1963)
History
Maturation
Testing
Instrumentation
Regression
Selection
Selection by maturation interactions
Mortality

3. Identifying the Spurious 8
Individuals changing without treatment
Maturation
History
Testing
Measurement error disguised as treatment effect
Instrumentation
Regression

4. Identifying the Spurious 8 (cont)
Non-treatment differences between groups
Mortality
Selection
Selection by maturation interactions

5. Ruling out the Spurious 8
Find out what threats are automatically ruled out by your design
Use logic to try to rule out the remaining threats

6. External Validity Questions: Can the Results be Generalized?
Can the results be generalized to other participants?
Can the results be generalized to other settings?
Results from the research study
(a “Mini-world”)
Larger
World
Generalize

7. Threats to Internal Validity History
A threat when an observed effect might be due to an event which takes place between the pre-test and post-test; when the event is not of research interest at that time
In the lab we control for this by “insulating” the P
Rarely able to fully control for this outside lab

8. Maturation
A threat when an observed effect might be due to the respondent’s changing with the passage of time per se.
Not specific to events
Examples
Aging (wiser, stronger)
More experience on DV
Rested, tired
Time of the month
Etc. etc.

9. Testing
A threat when observed effect may be due to the number of times particular responses are measured.
Frequently familiarity with a test or scale could affect performance or results because past responses were remembered
Error responses might be remembered (very likely)

10. Instrumentation
A threat when an effect might be due to a change in the measuring instrument between pre-test and post-test and not to the IV(s) at each time interval

11. Statistical Regression
A threat when an effect might be due to respondent’s being classified into experimental groups, highs, or lows, based on pretest position in the distribution
High on pretest, more like to score lower on posttest
Low on posttest, more likely to score higher on posttest

12. Selection
A threat when an effect may be due to the difference between kinds of people in one experimental group as opposed to another group
Pervasive problem in quasi-experimental designs, P’s are not randomly assigned to groups
Serious potential challenge if not dealt with

13. Interactions with selections
Many threats to internal validity can interact with selection to produce forces that might spuriously appear as treatment effects
Selection-maturation
Selection-history
Selection-instrumentation

14. The King of Internal Validity: Randomization
If you can randomize, then there are no threats due to:
Selection
Maturation
Selection-maturation interaction
If P’s all experience same testing conditions, then there is also no threat from:
Testing
Instrumentation

15. Threats to External Validity
Reactive Testing
Pretest may cause increase or decrease in P’s sensitivity to the IV
Interaction of Setting and Treatment
Results would only generalize to other Ps in very similar settings
For example: other colleges, hospitals, mental health centers, etc.

16. Coding
X=IV
O=Measurement of DV
R=Randomization

17. Pre-Experimental Designs One Shot Case Study
----X—O
Nothing is controlled
Why do it?

18. One-Group Pretest-Posttest Study:
O---X---O
Static-Group Comparison
X---O
-----O
What is Controlled?
Selection
Mortality
Reactive arrangement???
Nothing else is controlled
What is Controlled?
History
Maturation
Testing
Instrumentation
Nothing else is controlled

19. True Experimental Design Pre-test—Post-test Control Group Design
R O X O
R O --- O
What is Controlled?
All threats to internal validity
Does not control threats to external validity

20. Solomon Four-Group Design
R O X O
R O --- O
R --- X O
R O
What is Controlled?
All threats to internal validity
Threat of interaction of testing and X
Might control for interactions of selection and X
Might control for Multiple interactions

21. Post-test Only Control Group Design
R X O
R O
What is Controlled?
All threats to internal validity
Interaction of testing and X
Most common design in experimental design

22. Quasi-Experimental Designs Time Series Design
OOOOOXOOOOO
What is Controlled?
Maturation
Testing
Regression
Selection
Mortality
Interaction with selection, maturation, etc.

23. What is not covered? History Questionable on Instrumentation
Interaction of testing and treatment
Questionable on selection X interaction and multiple interactions

24. Non-Equivalent Control Group Design
O X O
O O
What is Covered?
History maturation
Testing
Instrumentation
Selection
Mortality
What is Not Covered?
Interaction of Testing and treatment
Interaction of Selection, maturation, etc.
Questionable for regression, reactive arrangements