Threats to validity in intervention studies Potential problems Issues to consider in planning

Threats to Internal validity Threats to External validity Problems we might encounter in this particular study. Problems that might limit the generalizability of the results An important distinction Credited to Campbell & Stanley (1963)

Threats to Internal validity Outside events (History) Attrition Maturation Selection Practice (testing) Instrumentation Regression

Outside events (History) Problems with issues that happen out in the real world News events The “Sesame Street” example Control groups can help deal with this

Attrition Drop outs A big issue in longitudinal studies College studies are a classic example Differential attrition –Can make new programs look good –Can make new programs look bad Not always a problem in short-term

Maturation Normal development may occur while the new treatment is being studied An important issue with young children Comparison groups can help

Selection effects In comparative studies How are people “selected” for the treatment and non- treatment conditions? –Volunteers? –Intact groups? What “biases” might be present?

Practice effects Can occur with tests Also a potential problem with interviews Observations Other situations? Can be tricky to control

Instrumentation Changes in the “instruments” Different tests Different questions (in interviews or questionnaires) Different interviewers Different observers Different scoring systems Different raters

Regression to the mean Regression artifacts High scores tend to be “not quite as high” Low scores tend to be “not quite so low” Only an issue in special circumstances Can have important effects –The “Head Start” example

Threats to External validity Limited samples Special arrangements Expectancy effects Sensitization Multiple treatments