1. Overview: Threats to Validity
1) People are Different
2) People Change
3) Process of Studying People Changes Them
4) Variables with Treatment Change People

2. 1) People are Different: Individual Differences
Can’t determine if outcome is due to IV or to individual differences
Pitfall of one group experiments
Threat to internal validity

3. 1) People are Different: Selection Bias/ Nonresponse Bias
Choosing participants from a nonrepresentative sample
Threat to external validity
President Alf Landon?
Nonresponse bias
When a substantial proportion of those invited to participate refuse to do so

4. 2) People Change History
Events occurring between first and second measurements but not part of the manipulation.
Subjects
Smoking
Measure
Training
Program
DV: Pretest
DV: Posttest
Independent
Variable
Pitfall of one group
pretest-posttest design
Threat to internal validity

5. 2) People Change Maturation
People change over time
Grow older wiser, stronger , healthier, more tired, more bored…
Third
graders
Math Exam
Training
Program
Pitfall of one group pretest-posttest design
Threat to internal validity

6. 3) People Change Regression Toward The Mean
AKA statistical regression
Extreme scores (high or low) change over time (regress to the mean)
High scores become lower
Low scores become higher
Hex of the Sports Illustrated cover
Pitfall of one group
pretest-posttest design

7. Does Studying Something Change it?
"The more precisely the POSITION is determined, the less precisely the MOMENTUM is known"
The Hawthorne effect
An increase in worker productivity produced by the obtrusive observation of that process

8. 3) Studying People Changes People Testing Effects
Simply taking the pretest changes the participant!
May cue people in on what is being studied
Make them more adept at a skill
Students
Science
Exam
Program
Pitfall of one group pretest-posttest design
Threat to internal validity

9. 3) Studying People Changes People Mortality (Attrition)
No, this isn’t that people in your study are dying
Participants drop out of the study before completing it
Couples
Relationship
Status
Marriage
Counseling
Homogeneous attrition
Attrition rates equal across experimental conditions
Threat to external validity
Heterogeneous attrition
Attrition rates different across experimental conditions
Threat to internal validity

10. 3) Studying People Changes People Participant Reaction Bias
Research participants realize they are being studied and behave in a way they normally would not
Good subject role
Participant reactance
Evaluation apprehension
Threats to internal validity

11. Applicant Hiring Experiment
INSTRUCTIONS
This study assesses job applicants capacity to handle stress
You will be interviewing an unemployed person
Your task is to read questions to the job applicant and then give negative responses
After the first eight questions you should read the 'Stress Remarks'
SCENARIO
The applicant appears well-qualified and experienced
The answers to the first eight questions are intelligent, lucid and to the point.
Stress Remark:
"This job is much too difficult for you according to the test".

12. Jane
Jane remembered the mix of male and female participants in the waiting area and noted the fact that the researcher was male. She decided that the experimenter had predicted that female subjects will be less likely to make the stress remarks and she so simply ignored them and said "well done you are doing well, aren't you!" in order to help the researcher's project.

13. Good-Subject Role
Participants consciously/unconsciously try to behave consistent with experimenters hypothesis
Demand characteristics
Feature of an experiment that may inform participants of the purpose of the study

14. Su
Su remembered the mix of male and female participants in the waiting area and noted the fact that the researcher was male. She also decided that the experimenter has predicted that female subjects will be less likely to make the stress remarks and made a point of not only making the stress remarks, but saying it in an aggressive manner - in order to disabuse the researcher of female stereotypes

15. Participant Reactance
Participants try to disconfirm perceived hypotheses

16. John
John looked at the highly professional instruction cards and the expensive video recording equipment and decided this was a test of management skills. So, he attempted to stay calm and read out the questions in a controlled, professional manner in order to show the experimenter that he was made of stern stuff and competent at handling difficult interpersonal situations. However, he was so nervous about doing the task well that he mixed up the order of questions and read them out too quickly.

17. Evaluation Apprehension
Participants try to behave in a way that will portray them most favorably.

18. Controlling for Demand Characteristics
Keep participants in the dark
Cover stories
Schacter and Singer’s emotion study
Unobtrusive observations
One-way mirror
Hot sauce
Radio stations
Ensure anonymity

19. 4) Treatment Changes People Experimenter Bias
AKA expectancy effects
Experimenter expectations biasing results
Two sources of experimenter bias
Unintentionally treat participants differently
Experimenters record the behaviors differently
Threat to internal validity
The story of Clever Hans

20. Expectancy Effects Rosenthal and Jacobson (1966) Students IQ tested
Told teachers some students were going to be “intellectual bloomers”
These students actually chosen at random
At end of school year their IQ tested again
Results
“Ready to bloomers” saw marked improvements
Self-fulfilling prophecy
Teacher expectations are communicated to students who in turn respond to those cues by adjusting their behavior.
The result: the original expectation becomes true.

21. Solution to Expectancy Effects
1) Practice makes perfect (or nearly so)
2) Run all participants in one fell swoop
3) Go high tech!
4) Ignorance can be bliss
Unaware of hypotheses
Unaware of condition
Single blind
Double blind

22. 4) Treatment Changes People Confounds
Some additional variable varies systematically with the independent variable
Really, all threats we have discussed the problem was that something else was accompanying the independent variable.