Approaches to social research Lerum Experiments

Here are some new terms: Internal validity External validity True experiments, quasi-experiments, pre-experiments Laboratory experiments, field experiments, natural experiments Reactivity Random assignment Test of statistical significance Matching Experimental and control groups

Experiments, overview The main reason for conducting an experiment is to test a hypothesis that one variable causes a change in another variable. This allows one to strongly infer (but not prove) cause-effect. Research designs that do this are called “true experiments” as opposed to “pre-experimental” or “quasi-experimental”

Features of a true (classical) experiment Random and matched assignment of subject into experimental and control groups (both can be used, but one should never use matching alone) Manipulation of an independent variable Followed by measurement of the dependent variable Tests for statistical significance

Example: Experiment of impact of teacher comments on student performance (Page, 1958). Students randomly put into two groups (w/o their knowledge) Teachers wrote comments for one group, but not for the other. Students’ subsequent test scores were compared, and the students w/ teacher comments scored better overall.

What is the point of a true experiment? To see if there is a causal relationship between X and Y. in order to determine this you must find: 1) An association between the two variables 2) Direction of influence (which variable influences which?) 3) Elimination of plausible rival explanations

Association In the teacher comment study, it was found the the IV (teacher comments) was associated with the DV (test scores).

Direction of Influence Evidence that the IV influenced the DV and not the other way around is based on time order in experiments. Y (e.g. test scores) cannot be the cause of X (teacher comments) if it occurred after X.

Elimination of rival explanations There may be lots of reasons for students doing better or worse on later exams (e.g. not just teacher comments). How do we know it’s the comments, and not something else that improved test scores? As long as you randomly assign people into groups, each participant has an equal chance of being in either group. In an airtight experiment, there is only one rival explanation: the results occurred by chance.

Tests for statistical significance Testing for statistical significance allow us to assess the likelihood that the results of an experiment cold have occurred by chance. When we read that the results of an experiment were significant at the .05 level, this means that only about 5% of the time, or 5 times in 100, would differences this large between groups occur simply out of chance.

Internal validity True experiments can provide relatively sound evidence of a causal relationship, and hence they generally have high internal validity. An experiment is internally valid to the extent that it rules out the possibility that extraneous variables, rather than the manipulated IV, or responsible for the observed outcome.

External validity External validity refers to the question of generalizability. In other words, what do the experimental results mean outside of the particular context of the experiment. Would the experiment produce the same results on a different sample? This can be ensured by replicating the study in a number of different settings, with different kinds of people, etc.

Weaknesses of “true” experiments Typically low in external validity. E.g., most experiments conducted in university settings, on white college sophomores taking psychology. Cannot generalize to larger populations from a small, uniquely situated sample. Reactivity and social desirability effects Ethical issues (abuse of power more likely due to extremely controlled conditions; valid results harder to achieve if people have full consent)

Strengths of “true” experiments Generally high in internal validity Can establish strong inference for a causal relationship (non-spurious)

In addition to laboratory experiments, two other types exist: 1)Field experiments, and 2) Natural experiments Field experiments refer to studies that meet all the requirements of a true experiment but are conducted in a “natural” setting Both field and lab experiments can be “true” experiments As in a lab experiment, there is a staging of events. But this staging occurs in an environment that is familiar to the subject.

Example of a field experiment Common in public health – otherwise known as randomized, controlled Trials E.g. randomized controlled male circumcision trials http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0020298

Example of a natural experiment (from Babbie, p. 245): After WWII, researchers studied impact of mass bombing on civilian morale, comparing residents of heavily bombed cities to those in cities w/o bombing. Natural experiments occur “after the fact,” and hence can not be randomized or subjected to pre and post analysis. Thus natural experiments are always “quasi” experiments.