1. Establishing a Cause-Effect Relationship

2. Internal Validity The “treatment” and the “outcomes”The “treatment” and the “outcomes” The independent and dependent variables.The independent and dependent variables. Observation TreatmentOutcomes What you do What you see Is the relationship causal between... Alternativecause Alternativecause Alternativecause Alternativecause In this study

3. Establishing Cause and Effect Temporal precedence

4. Establishing Cause and Effect Temporal precedence CauseEffect then Time It can get complicated through: -sloppiness (campaign contributions - Chicken and egg cyclical functions (democracy and GDP)

5. Establishing Cause and Effect Temporal precedence Covariation of cause and effect CauseEffectthen Time

6. Establishing Cause and Effect Temporal precedence Covariation of cause and effect CauseEffectthen Time if X, then Y if not X, then not Y if treatment given, then outcome observed (usually) if program not given, then outcome not observed

7. Establishing Cause and Effect Temporal precedence Covariation of cause and effect CauseEffectthen Time if X, then Y if not X, then not Y if program given, then outcome observed if program not given, then outcome not observed Dosage effects or comparative statics: If more of treatment, then more of outcome observed if less of treatment given, then less of outcome observed

8. Establishing Cause and Effect Temporal precedence Covariation of cause and effect No alternative explanations CauseEffectthen Time if X, then Y if not X, then not Y TreatmentOutcome Micromediation

9. Establishing Cause and Effect Temporal precedence Covariation of cause and effect No alternative explanations CauseEffectthen Time if X, then Y if not X, then not Y TreatmentOutcome Micromediation Alternativecause Alternative cause (nuisance) Alternativecause (substantive)

10. In Lab or Field Experiments… l Is taken care of because you intervene before you measure outcome l Is measured by comparing treated and untreated groups l Is the central issue of internal validity -- usually taken care of through random assignment Temporal precedence Covariation of cause and effect No alternative explanations

11. Single-Group Threats to Internal Validity

12. The Single Group Case Two designs:

13. The Single Group Case AdministerprogramMeasureoutcomesXO Two designs: “Post-test only single-group design” - X is the treatment - O is the observation

14. The Single Group Case Two designs: AdministerprogramMeasureoutcomesXO Measurebaseline O “pre-test, post-test single-group design” or “interrupted time-series”

15. The Single Group Case AdministerprogramMeasureoutcomesXO Two designs: AdministerprogramMeasureoutcomesXO Measurebaseline O Alternativeexplanations Alternativeexplanations Alternativeexplanations

16. ExampleExample l After the 2003 recall election, did Democrats in the California Assembly move to the center? l California ran a full legislative “season” before the October, 2003 election, then ran another “season” afterward. l We can look at roll call vote behavior

17. Example: What Kind of Design?

18. History Threat l Any other event that occurs between pretest and posttest l Perhaps the nation was just shifting to the center at this time. l How might we rule it out? ProgramPosttest XO Pretest O

19. Maturation Threat l Normal growth between pretest and posttest. l Coming into an election year, state legislators always shift to the center. ProgramPosttest XO Pretest O

20. Ruling Out a Maturation Threat

21. Testing Threat l The effect on the posttest of taking the pretest l Legislators may have learned that the state was watching them. When real tests are given, this is a big problem. ProgramPosttest XO Pretest O

22. Instrumentation Threat l Any change in the test from pretest and posttest l A different test may have been used if a different roll call estimation technology used. ProgramPosttest XO Pretest O

23. Mortality Threat l Nonrandom dropout between pretest and posttest l If some legislators had been recalled along with Gray Davis, this would be a problem. ProgramPosttest XO Pretest O

24. Regression Threat l Group is a nonrandom subgroup of population. l The 2003 session was particularly extreme, any other session would look more centrist. ProgramPosttest XO Pretest O

25. Multiple-Group Threats to Internal Validity

26. The Central Issue l When you move from single to multiple group research the big concern is whether the groups are comparable. l Usually this has to do with how you assign units (for example, persons) to the groups (or select them into groups). l If you are not careful, may mistake a selection effect for a treatment effect.

27. The Multiple Group Case AdministertreatmentMeasureoutcomesMeasurebaseline Alternativeexplanations AlternativeexplanationsXOOOO Do not administer treatment MeasureoutcomesMeasurebaseline

28. ExampleExample l Suppose USAID looked before and after at countries where it did and didn’t run governance programs in the last decade l Pre-post program-comparison group design l Measures (O) are all of the things Clark hates, but let’s set that aside for now.

29. Selection Threats l Any factor other than the program that leads to posttest differences between groups. l USAID did not randomly select the countries in which it ran programs, and sent aid to those with the lowest-rated governments XOOOO

30. Selection-History Threat l Any other event that occurs between pretest and posttest that the groups experience differently. l For example, countries that begin with more stable democracies faced fewer challenges in the past decade. XOOOO

31. Selection-Maturation Threat l Differential rates of normal growth between pretest and posttest for the groups. l It is easier to move from a semi- democracy to a full democracy than it is to move from a non-democracy to a semi-democracy XOOOO

32. Selection-Testing Threat l Differential effect on the posttest of taking the pretest. l At least these measures are “unobtrusive,” so this probably is not a grave threat XOOOO

33. Selection-Instrumentation Threat l Any differential change in the test used for each group from pretest and posttest l For example, the Polity measures may give some countries credit for having a USAID program XOOOO

34. Selection-Mortality Threat l Differential nonrandom dropout between pretest and posttest. l Perhaps the countries with weak governments are more likely to cease being a country over the past decade. XOOOO

35. Selection-Regression Threat l Different rates of regression to the mean because groups differ in extremity. l For example, the countries that USAID chooses may have nowhere to go but up. XOOOO

36. “Social Interaction” Threats to Internal Validity

37. What Are “Social” Threats? All are related to social pressures in the research context, which can lead to posttest differences that are not directly caused by the treatment itself. Most of these can be minimized by isolating the two groups from each other, but this leads to other problems (for example, hard to randomly assign and then isolate, or may reduce generalizability).

38. Types of Designs

39. Random assignment?

40. Types of Designs Random assignment? Yes

41. Types of Designs Random assignment? Yes Randomized or true experiment?

42. Types of Designs Random assignment? YesNo Randomized or true experiment?

43. Types of Designs Random assignment? Control group or multiple measures? YesNo Randomized or true experiment?

44. Types of Designs Random assignment? Control group or multiple measures? YesNo Yes Randomized or true experiment?

45. Types of Designs Random assignment? Control group or multiple measures? YesNo Yes Randomized or true experiment? Quasi-experiment

46. Types of Designs Random assignment? Control group or multiple measures? YesNo YesNo Randomized or true experiment? Quasi-experiment

47. Types of Designs Random assignment? Control group or multiple measures? YesNo YesNo Randomized or true experiment? Quasi-experimentNonexperiment

48. Design Notation Example ROXOROOROXOROO Os indicate different waves of measurement.

49. Elements of a Design l Observations and measures l Treatments l Groups l Assignment to group l Time

50. Design Notation Example ROXOROXOROOROOROXOROXOROOROO Vertical alignment of Os shows that pretest and posttest are measured at same time.

51. Design Notation Example ROXOROOROXOROO X is the treatment.

52. Design Notation Example ROXOROOROXOROO There are two lines, one for each group.

53. Design Notation Example ROXOROOROXOROO R indicates the groups arerandomlyassigned.

54. Design Notation Example RO 1 XO 1, 2 RO 1 O 1, 2 Subscriptsindicate subsets of measures.

55. Design Notation Example Pretest-posttest (before-after) Treatment versus comparison group Randomized experimental design ROXOROOROXOROO

56. Design Example Posttest Only Randomized Experiment

57. Design Example Posttest Only Randomized Experiment RXORORXORO

58. Design Example Pretest-Posttest Nonequivalent Groups Quasi-Experiment

59. Design Example Pretest-Posttest Nonequivalent Groups Quasi-Experiment (note multiple groups or multiple observations are REQUIRED to have a quasi-experiment) NOXONOONOXONOO

60. Design Example Posttest Only Nonexperiment

61. Design Example Posttest Only Nonexperiment XOXO