1. Agenda Group Hypotheses Validity of Inferences from Research Inferences and Errors Types of Validity Threats to Validity

2. Validity Inferences and Errors Types of Validity Threats to Validity

3. Inferential hazards Empirical tests are inherently ambiguous Results cannot confirm or prove theory Possible rival explanations persist Conclusions about causal effects are clouded by a number of design factors Lack of control over extraneous influences Processes of measurement

4. Inferential hazards Two fundamental mistakes possible: We conclude our theory is TRUE when in truth it is FALSE (Type One Error) We conclude our theory is FALSE when in truth it is TRUE (Type Two Error)

5. Theory Is Actually: True False Conclude Theory Is True Conclude Theory Is False Correct Type I Error Type II Error

6. Validity Has to do with the degree of doubt surrounding our inferences Doubts about whether we are measuring what we think we are Doubts about whether we observe a relationship Doubts about whether the observed relationship is causal Doubts about whether we can generalize from the relationship

7. X Exposure to violent TV Y Aggressive Behavior X Viewing either a boxing film or sports film Y Administration of shocks in a learning study External Validity Construct (Measurement) Validity Internal Validity Generalization to Population in the Real World Statistical Conclusion Validity ?

8. Construct validity All measures imply a theory All measures contain some error Random error (noise) Systematic error (bias) How valid is our interpretation of a measure?

9. Statistical conclusion validity Observed relationships depend upon: Random processes Number of observations The noisiness (unreliability) of our measures How certain are we that we have observed a relationship?

10. Randomness raises chance of Type II Error Hypothesis Is Actually: True False Accept Hypothesis Reject Hypothesis Correct Type I Error Type II Error Systematic mistakes raise chances of either I or II, depending

11. Internal validity Concluding that a relationship is causal requires: Covariation Temporal ordering of observed cause and effect Non-spuriousness i.e., elimination of rival explanations How certain are we of a cause-effect relationship?

12. External validity Sound generalizations depend upon: Representativeness of the sample The observational setting Representativeness of the processes observed How certain are we that generalizations are warranted?

13. Threats to validity Threats are rival explanations For our measures For why we did (or didnt) observe a relationship For what caused the relationship For the meaning of the relationship in everyday life

14. Ambiguous Causal Direction Lack of established temporal order may render reverse causation plausible Common in correlational studies 1

15. Selection Systematic differences may be present in groups selected for comparison Groups must be made comparable By random assignment (best) By matching (not as good) 2

16. History Changes in the environment may occur between measurements Example: Studying a community intervention 3 (A) X (health campaign) (B) No X ( no campaign) Y (diet) Other events could account for changes in Y

17. Maturation People change naturally over time The longer the time between measurements, the greater the possibility of maturation effects 4

18. Regression artifacts Extreme scores tend to regress toward the mean on a second test A problem if tests are used to select groups and then repeated 5

19. First Time Scores 400 510 560 600...... Second Time Scores 240 280 310 350...... 690 730 780 800 280 350 380 500 520 490 640 700 760 790 240

20. Attrition People may drop out of studies May produce non-random group comparisons 6

21. Testing Repeated measures can affect each other 7 (A) Y (B) Y Y Y Y (diet) X (campaign) X No X

22. Testing Measures can also interact with the manipulation to produce effects (AKA interaction of testing and treatment) 7 (A) Y (C) Y Y X X (B) Y Y No X (D) Y No X

23. Instrumentation Changes in measurement processes can confound interpretation Instruments should be identical over time and across groups Ceiling and floor effects may cloud interpretation of measurements 8

24. Interactions with selection Different groups maturing at different rates Different groups experience local history Different groups experience unique ceiling or floor effects on measures 9

25. Reactivity to experimental setting People may respond, not to the independent variable per se, but to the situation Experimenter demand 1

26. Compensatory equalization Administrative equity can spoil comparisons Relevant when interventions provide desirable public goods 2

27. Compensatory rivalry Recognition of treatment can cause control groups to compensate 3

28. Resentful demoralization Recognition of treatment can cause control groups to become despondent, act up, etc. 4

29. Diffusion of treatments Contact between treatment and control groups can spoil comparisons Possible in experimental interventions May be present in some quasi-experimental comparisons 5

30. Interaction of causal relationship with units The effects might only apply to the groups manipulated or observed Volunteers College students 1

31. Interaction of causal relationship with outcomes The effects might only apply to particular, observed facets of complex phenomena Fuller picture of effects might lead to different conclusions 2

32. Interaction of causal relationship with setting Experimental setting may be complex or artificial Effects may be limited to a particular time or place 3

33. For Tuesday Manipulation, observation, and control of Variables Experiments Schutt, Ch. 7 on experimental design Shadish, Cook & Campbell, Ch. 8 on problems and solutions in conducting experiments