Experimental Design

Experimental designs Experiment is probably the strongest design with respect to internal validity Recall that internal validity is at the center of all causal or cause-effect inferences When you want to determine whether some program or treatment causes some outcome to occur, then you are interested in having internal validity

Experimental design To really show that there is a causal relationship, you have to simultaneously address the two propositions: If X, then Y and If not X, then not Y

In other words… If the program is given, then the outcome occurs and If the program is not given, then the outcome does not occur If you are able to provide evidence for both of these propositions, then you've isolated the program from all of the other potential causes of the outcome. That points to the causal effectiveness of the program.

Logic of experimental design In the simplest type of experiment, we create two groups that are "equivalent" to each other One group (treatment group) gets the program and the other group (control group) does not The people are assigned randomly to each group Now, if we observe differences in outcomes between these two groups, then the differences must be due to the only thing that differs between them -- that one got the program and the other didn't.

Components of Experiments Independent and dependent variables Pre-testing and post-testing Experimental and control groups Babbie 2005

Independent and dependent variables Treatment or Independent Variable the stimulus, manipulation, or intervention that the researcher creates or delivers to one set of participants or clients Dependent Variable is the outcome or condition that may change as a result of being subjected to or exposed to the treatment or an independent variable

Pre-testing and post-testing Pre-test is the measurement of the dependent variable prior to treatment or intervention Post-test is measurement of the dependent variable after the treatment/ independent variable has been applied

Experimental and Control groups Must be as similar as possible. An Experimental Group are those who receive the treatment or are exposed to the independent variable under study. The Control Group are those who do not receive the treatment or independent variable under study. Copyright @ Allyn & Bacon 2003

NOTATIONS Observations or Measures Treatments or Programs Groups Assignment to Group Time

Observations or Measures These are symbolized by an 'O' in design notation An O can refer to a single measure (e.g., a measure of body weight), a single instrument with multiple items (e.g., a 10-item self-esteem scale), a complex multi-part instrument (e.g., a survey), or a whole battery of tests or measures given out on one occasion If you need to distinguish among specific measures, you can use subscripts with the O, as in O1, O2, and so on.

Because all of the Os have a subscript of 1, there is some measure that is collected for both groups on both occasions But the design also has two Os with a subscript of 2, both taken at the posttest. This means that there was some measure that were collected only at the posttest.

Treatments or Programs These are symbolized with an 'X' in design notations The X can refer to a simple intervention (e.g., a one-time surgical technique) or to a complex program (e.g., an employment training program) Usually, a no-treatment control or comparison group has no symbol for the treatment (some researchers use X+ and X- to indicate the treatment and control respectively)

Groups Each group in a design is given its own line in the design structure If the design notation has three lines, there are three groups in the design

Assignment to Group Assignment to group is designated by a letter at the beginning of each line (i.e., group) that describes how the group was assigned. The major types of assignment are: R = random assignment N = nonequivalent groups C = assignment by cutoff

Time Time moves from left to right. Elements that are listed on the left occur before elements that are listed on the right.

Diagram of Classical Experimental Design Babbie 2005

Types of Designs

Types of Designs

The Nonequivalent Groups Design The term means that assignment to group was not random In other words, the researcher did not control the assignment to groups through the mechanism of random assignment. As a result, the groups may be different prior to the study

The Nonequivalent Groups Design We most often use intact groups that we think are similar as the treatment and control groups In education - we might pick two comparable classrooms or schools In community-based research, we might use two similar communities We try to select groups that are as similar as possible, but we can never be sure the groups are comparable Because it's often likely that the groups are not equivalent, this design was named the nonequivalent groups design to remind us.

Posttest only nonexperimental design You might use this design if you want to study the effects of a natural disaster like a flood or tornado and you want to do so by interviewing survivors Notice that in this design, you don't have a comparison group (e.g., interview in a town down the road that didn't have the tornado to see what differences the tornado caused) and you don't have multiple waves of measurement (e.g., a pre-tornado level of how people in the ravaged town were doing before the disaster)

Example Effect of the decision to conduct crackdown on drinking and driving by a local police force (planned intervention)

Posttest only nonexperimental design Does it make sense to do the non-experimental study? Of course! You could gain lots of valuable information by well-conducted post-disaster interviews But you may have a hard time establishing which of the things you observed are due to the disaster rather than to other factors like the peculiarities of the town or pre-disaster characteristics.

Types of Designs A randomized experiment generally is the strongest of the three designs when your interest is in establishing a cause-effect relationship A non-experiment is generally the weakest in this respect The simplest form of non-experiment is a one-shot survey design that consists of nothing but a single observation O This is probably one of the most common forms of research and, for some research questions -- especially descriptive ones -- is clearly a strong design

Expanding Across Time We can add to the basic design by including additional observations either before or after the program The addition of such pretests provides a "baseline" which, for instance, helps to assess the potential of a maturation or testing threat. If a change occurs between the first and second pre-program measures, it is reasonable to expect that similar change might be seen between the second pretest and the posttest even in the absence of the program.

Expanding across time Similarly, additional postprogram measures could be added This would be useful for determining whether an immediate program effect decays over time, or whether there is a lag in time between the initiation of the program and the occurrence of an effect

We might also add and remove the program over time This design is frequently used in clinical psychology and psychiatry. The design is particularly strong against a history threat. When the program is repeated it is less likely that unique historical events can be responsible for replicated outcome patterns.

Expanding Across Groups Often, it will be to our advantage to add additional groups to a design in order to rule out specific threats to validity If this design were implemented within a single institution where members of the two groups were in contact with each other one might expect that intergroup communication, group rivalry, or demoralization of a group which gets denied a desirable treatment

Expanding Across Groups In such a case, one might add an additional nonequivalent group from a similar institution which consists of persons unaware of the original two groups:

The Solomon Four-Group Design Note that two of the groups receive the treatment and two do not. Further, two of the groups receive a pretest and two do not. Within each treatment condition we have a group that is pretested and one that is not. By explicitly including testing as a factor in the design, we are able to assess experimentally whether a testing threat took place.

Switching Replications Design This is a two group design with three waves of measurement. The implementation of the treatment is repeated or replicated. And in the repetition of the treatment, the two groups switch roles -- the original control group becomes the treatment group in phase 2 while the original treatment acts as the control. By the end of the study all participants have received the treatment.

Switching Replications Design This design works especially well in schools that are on a semester system. All students are pretested at the beginning of the school year. During the first semester, Group 1 receives the treatment and during the second semester Group 2 gets it. The design also enhances organizational efficiency in resource allocation Schools only need to allocate enough resources to give the program to half of the students at a time.

Strengths Weaknesses In real life, only rarely one variable actually a cause of another one Difficult to test very complex hypotheses (difficult to manipulate and control more than one or two variables) Ethical issues The only method that allows us to test the causal relationships between variables Random assignment of subjects to experimental and control groups allows us to test our hypotheses