Internal Validity Questions Did the treatment cause a change in behavior? Change in Behavior Treatment Causes

8 Threats to Internal Validity (Cambell & Stanley, 1963) History Maturation Testing Instrumentation Regression Selection Selection by maturation interactions Mortality

Identifying the Spurious 8 Individuals changing without treatment Maturation History Testing Measurement error disguised as treatment effect Instrumentation Regression

Identifying the Spurious 8 (cont) Non-treatment differences between groups Mortality Selection Selection by maturation interactions

Ruling out the Spurious 8 Find out what threats are automatically ruled out by your design Use logic to try to rule out the remaining threats

External Validity Questions: Can the Results be Generalized? Can the results be generalized to other participants? Can the results be generalized to other settings? Results from the research study (a “Mini-world”) Larger World Generalize

Threats to Internal Validity History A threat when an observed effect might be due to an event which takes place between the pre-test and post-test; when the event is not of research interest at that time In the lab we control for this by “insulating” the P Rarely able to fully control for this outside lab

Maturation A threat when an observed effect might be due to the respondent’s changing with the passage of time per se. Not specific to events Examples Aging (wiser, stronger) More experience on DV Rested, tired Time of the month Etc. etc.

Testing A threat when observed effect may be due to the number of times particular responses are measured. Frequently familiarity with a test or scale could affect performance or results because past responses were remembered Error responses might be remembered (very likely)

Instrumentation A threat when an effect might be due to a change in the measuring instrument between pre-test and post-test and not to the IV(s) at each time interval

Statistical Regression A threat when an effect might be due to respondent’s being classified into experimental groups, highs, or lows, based on pretest position in the distribution High on pretest, more like to score lower on posttest Low on posttest, more likely to score higher on posttest

Selection A threat when an effect may be due to the difference between kinds of people in one experimental group as opposed to another group Pervasive problem in quasi-experimental designs, P’s are not randomly assigned to groups Serious potential challenge if not dealt with

Interactions with selections Many threats to internal validity can interact with selection to produce forces that might spuriously appear as treatment effects Selection-maturation Selection-history Selection-instrumentation

The King of Internal Validity: Randomization If you can randomize, then there are no threats due to: Selection Maturation Selection-maturation interaction If P’s all experience same testing conditions, then there is also no threat from: Testing Instrumentation

Threats to External Validity Reactive Testing Pretest may cause increase or decrease in P’s sensitivity to the IV Interaction of Setting and Treatment Results would only generalize to other Ps in very similar settings For example: other colleges, hospitals, mental health centers, etc.

Coding X=IV O=Measurement of DV R=Randomization

Pre-Experimental Designs One Shot Case Study ----X—O Nothing is controlled Why do it?

One-Group Pretest-Posttest Study: O---X---O Static-Group Comparison X---O -----O What is Controlled? Selection Mortality Reactive arrangement??? Nothing else is controlled What is Controlled? History Maturation Testing Instrumentation Nothing else is controlled

True Experimental Design Pre-test—Post-test Control Group Design R O X O R O --- O What is Controlled? All threats to internal validity Does not control threats to external validity

Solomon Four-Group Design R O X O R O --- O R --- X O R --- --- O What is Controlled? All threats to internal validity Threat of interaction of testing and X Might control for interactions of selection and X Might control for Multiple interactions

Post-test Only Control Group Design R --- X O R --- --- O What is Controlled? All threats to internal validity Interaction of testing and X Most common design in experimental design

Quasi-Experimental Designs Time Series Design OOOOOXOOOOO What is Controlled? Maturation Testing Regression Selection Mortality Interaction with selection, maturation, etc.

What is not covered? History Questionable on Instrumentation Interaction of testing and treatment Questionable on selection X interaction and multiple interactions

Non-Equivalent Control Group Design O X O O --- O What is Covered? History maturation Testing Instrumentation Selection Mortality What is Not Covered? Interaction of Testing and treatment Interaction of Selection, maturation, etc. Questionable for regression, reactive arrangements