Establishing a Cause-Effect Relationship

Internal Validity The “treatment” and the “outcomes”The “treatment” and the “outcomes” The independent and dependent variables.The independent and dependent variables. Observation TreatmentOutcomes What you do What you see Is the relationship causal between... Alternativecause Alternativecause Alternativecause Alternativecause In this study

Establishing Cause and Effect Temporal precedence

Establishing Cause and Effect Temporal precedence CauseEffect then Time It can get complicated through: -sloppiness (campaign contributions - Chicken and egg cyclical functions (democracy and GDP)

Establishing Cause and Effect Temporal precedence Covariation of cause and effect CauseEffectthen Time

Establishing Cause and Effect Temporal precedence Covariation of cause and effect CauseEffectthen Time if X, then Y if not X, then not Y if treatment given, then outcome observed (usually) if program not given, then outcome not observed

Establishing Cause and Effect Temporal precedence Covariation of cause and effect CauseEffectthen Time if X, then Y if not X, then not Y if program given, then outcome observed if program not given, then outcome not observed Dosage effects or comparative statics: If more of treatment, then more of outcome observed if less of treatment given, then less of outcome observed

Establishing Cause and Effect Temporal precedence Covariation of cause and effect No alternative explanations CauseEffectthen Time if X, then Y if not X, then not Y TreatmentOutcome Micromediation

Establishing Cause and Effect Temporal precedence Covariation of cause and effect No alternative explanations CauseEffectthen Time if X, then Y if not X, then not Y TreatmentOutcome Micromediation Alternativecause Alternative cause (nuisance) Alternativecause (substantive)

In Lab or Field Experiments… l Is taken care of because you intervene before you measure outcome l Is measured by comparing treated and untreated groups l Is the central issue of internal validity -- usually taken care of through random assignment Temporal precedence Covariation of cause and effect No alternative explanations

Single-Group Threats to Internal Validity

The Single Group Case Two designs:

The Single Group Case AdministerprogramMeasureoutcomesXO Two designs: “Post-test only single-group design” - X is the treatment - O is the observation

The Single Group Case Two designs: AdministerprogramMeasureoutcomesXO Measurebaseline O “pre-test, post-test single-group design” or “interrupted time-series”

The Single Group Case AdministerprogramMeasureoutcomesXO Two designs: AdministerprogramMeasureoutcomesXO Measurebaseline O Alternativeexplanations Alternativeexplanations Alternativeexplanations

ExampleExample l After the 2003 recall election, did Democrats in the California Assembly move to the center? l California ran a full legislative “season” before the October, 2003 election, then ran another “season” afterward. l We can look at roll call vote behavior

Example: What Kind of Design?

History Threat l Any other event that occurs between pretest and posttest l Perhaps the nation was just shifting to the center at this time. l How might we rule it out? ProgramPosttest XO Pretest O

Maturation Threat l Normal growth between pretest and posttest. l Coming into an election year, state legislators always shift to the center. ProgramPosttest XO Pretest O

Ruling Out a Maturation Threat

Testing Threat l The effect on the posttest of taking the pretest l Legislators may have learned that the state was watching them. When real tests are given, this is a big problem. ProgramPosttest XO Pretest O

Instrumentation Threat l Any change in the test from pretest and posttest l A different test may have been used if a different roll call estimation technology used. ProgramPosttest XO Pretest O

Mortality Threat l Nonrandom dropout between pretest and posttest l If some legislators had been recalled along with Gray Davis, this would be a problem. ProgramPosttest XO Pretest O

Regression Threat l Group is a nonrandom subgroup of population. l The 2003 session was particularly extreme, any other session would look more centrist. ProgramPosttest XO Pretest O

Multiple-Group Threats to Internal Validity

The Central Issue l When you move from single to multiple group research the big concern is whether the groups are comparable. l Usually this has to do with how you assign units (for example, persons) to the groups (or select them into groups). l If you are not careful, may mistake a selection effect for a treatment effect.

The Multiple Group Case AdministertreatmentMeasureoutcomesMeasurebaseline Alternativeexplanations AlternativeexplanationsXOOOO Do not administer treatment MeasureoutcomesMeasurebaseline

ExampleExample l Suppose USAID looked before and after at countries where it did and didn’t run governance programs in the last decade l Pre-post program-comparison group design l Measures (O) are all of the things Clark hates, but let’s set that aside for now.

Selection Threats l Any factor other than the program that leads to posttest differences between groups. l USAID did not randomly select the countries in which it ran programs, and sent aid to those with the lowest-rated governments XOOOO

Selection-History Threat l Any other event that occurs between pretest and posttest that the groups experience differently. l For example, countries that begin with more stable democracies faced fewer challenges in the past decade. XOOOO

Selection-Maturation Threat l Differential rates of normal growth between pretest and posttest for the groups. l It is easier to move from a semi- democracy to a full democracy than it is to move from a non-democracy to a semi-democracy XOOOO

Selection-Testing Threat l Differential effect on the posttest of taking the pretest. l At least these measures are “unobtrusive,” so this probably is not a grave threat XOOOO

Selection-Instrumentation Threat l Any differential change in the test used for each group from pretest and posttest l For example, the Polity measures may give some countries credit for having a USAID program XOOOO

Selection-Mortality Threat l Differential nonrandom dropout between pretest and posttest. l Perhaps the countries with weak governments are more likely to cease being a country over the past decade. XOOOO

Selection-Regression Threat l Different rates of regression to the mean because groups differ in extremity. l For example, the countries that USAID chooses may have nowhere to go but up. XOOOO

“Social Interaction” Threats to Internal Validity

What Are “Social” Threats? All are related to social pressures in the research context, which can lead to posttest differences that are not directly caused by the treatment itself. Most of these can be minimized by isolating the two groups from each other, but this leads to other problems (for example, hard to randomly assign and then isolate, or may reduce generalizability).

Types of Designs

Random assignment?

Types of Designs Random assignment? Yes

Types of Designs Random assignment? Yes Randomized or true experiment?

Types of Designs Random assignment? YesNo Randomized or true experiment?

Types of Designs Random assignment? Control group or multiple measures? YesNo Randomized or true experiment?

Types of Designs Random assignment? Control group or multiple measures? YesNo Yes Randomized or true experiment?

Types of Designs Random assignment? Control group or multiple measures? YesNo Yes Randomized or true experiment? Quasi-experiment

Types of Designs Random assignment? Control group or multiple measures? YesNo YesNo Randomized or true experiment? Quasi-experiment

Types of Designs Random assignment? Control group or multiple measures? YesNo YesNo Randomized or true experiment? Quasi-experimentNonexperiment

Design Notation Example ROXOROOROXOROO Os indicate different waves of measurement.

Elements of a Design l Observations and measures l Treatments l Groups l Assignment to group l Time

Design Notation Example ROXOROXOROOROOROXOROXOROOROO Vertical alignment of Os shows that pretest and posttest are measured at same time.

Design Notation Example ROXOROOROXOROO X is the treatment.

Design Notation Example ROXOROOROXOROO There are two lines, one for each group.

Design Notation Example ROXOROOROXOROO R indicates the groups arerandomlyassigned.

Design Notation Example RO 1 XO 1, 2 RO 1 O 1, 2 Subscriptsindicate subsets of measures.

Design Notation Example Pretest-posttest (before-after) Treatment versus comparison group Randomized experimental design ROXOROOROXOROO

Design Example Posttest Only Randomized Experiment

Design Example Posttest Only Randomized Experiment RXORORXORO

Design Example Pretest-Posttest Nonequivalent Groups Quasi-Experiment

Design Example Pretest-Posttest Nonequivalent Groups Quasi-Experiment (note multiple groups or multiple observations are REQUIRED to have a quasi-experiment) NOXONOONOXONOO

Design Example Posttest Only Nonexperiment

Design Example Posttest Only Nonexperiment XOXO