Political Science 30 Political Inquiry Experiments

The Beauty of Random Assignment How Experiments Work Strengths and Weakness

The Beauty of Random Assignment Problem: In non-experimental studies, what determines the values that an independent variable takes on? Often, a confounding variable determines these values, and affects the DV. For instance, the confound of “Are you a serious student” may determine where you will sit in a class.

An Example of a Confound from Recent Research Getting Out the Vote!, a recent book by Don Green and Alan Gerber, asks: Does contacting registered voters through phone calls or visits make them more likely to turn out on election day? Potential Confound: Previous participation records. Campaigns with limited resources concentrate their mobilization efforts on voters who have turned out in the past.

The Beauty of Random Assignment Solution: Interrupt the causal path that leads from the confound to the independent variable by “randomly assigning” the values that the IV takes on in each case. Randomly assign seats so that there are just as many serious students and slackers in each part of the lecture hall. Green and Gerber randomly assigned some voters to be contacted, in order to measure the actual effects of mobilization (which are quite weak).

The Beauty of Random Assignment Treatment Group All of the cases in this group have been assigned one value of the IV (sit in front, take medicine, etc.) Control Group All of the cases have been assigned a different value of the IV (in most cases, left alone or given placebo) In all other respects (including the values of confounding variables that they take on), these groups are similar.

How Experiments Work Step #1: Random Assignment Begin by splitting your cases into two or more groups of 30 or more through a process that is truly random. Using something like a random number generator is key, because many seemingly neutral assignment processes may be correlated with a confounding variable. Examples: arrival times, last names, section times, bleeding hearts.

How Experiments Work Step #2: (Optional) Pre-Test To check how the random assignment process worked, measure the value that the DV takes on for each case before any treatment has been applied. Each group should average about the same values on the dependent variable. Even if something went wrong, we can still learn from the “time-series” comparison. Often it is hard to pre-test.

How Experiments Work Step #3: Apply the Treatment Change the value of the independent variable that cases in at least the treatment group take on. Administer the medicine or the placebo, put students in their seats, request that subjects administer an electric shock. This is where ethical issues arise.

How Experiments Work Step #4: Post-Test Measure the value that the DV takes on for each case after the treatment has been applied. Comparing values of the DV in treatment group vs. control group tells us the effect of the treatment, if random assignment worked. Comparing shifts from pre-test to post-test is helpful when random assignment failed.

Schematic of an Experiment Treatment Group (pre-test) Treatment(post-test) RandomCompare Assignment Control Group (pre-test)(post-test)

Strength of Experiments: High Internal Validity Internal validity judges how well a research design has tested a causal relationship, in the cases examined. “Among the cases in our study, do we have reason to believe that IV #1 causes DV? Because random assignment takes away our fear of confounds, experiments have high internal validity.

Weakness of Experiments: Low External Validity External validity judges how confident we can be that a causal relationship identified in our cases can be generalized to the outside world. Our cases may be different than the general population, or our cases may react differently to treatments, or our treatments may be very artificial. “College sophomore problem” makes external validity the flaw in experiments. You can’t assign every treatment: gender, race.