Lecture 6-II Experimental Design Research Methods and Statistics

Threats to External Validity External validity refers to the appropriateness of generalizing research results to other settings. Interaction of selection and experimental treatment: the treated population may have a unique trait that other treatment groups do not have. Reactive effect of experimental arrangements: the conditions that make the experimental setting seem atypical of other settings in which the treatment might be used.

Threats to External Validity Multiple-treatment interference (effects of history): may occur where the subject is receiving more than one type of treatment at the same time. Irrelevant replicability of treatments: replications of the treatment by different experimenters may fail to include those components which actually caused the change in the dependent variable.

Notation for Experimental Designs Assignment to Group: describes how the group was assigned. The 2 most common types of assignment are: R = random assignment N = nonequivalent groups Observations/Measures (DV): an 'O' in design notations. Treatments/Programs (IV): an 'X' in design notations. Time: Time moves from left to right. R O X O R O O

Types of Experimental Designs Weak Experimental Designs Quasi- Experimental Designs Strong Experimental Designs

Examples of Weak Experimental Designs Experimental designs are considered “weak” when they are very vulnerable to the threats of internal validity This is problematic in terms of internal validity: maturation, there’s no control, a limited ability to say that there is a cause and effect N means that it is not randomly assigned. Therefore limited ability to state cause and effect X O One Group Posttest Only O X O One Group Pretest Posttest N X O ------------- N O Posttest Only with Nonequivalent Groups

Example of Quasi-Experimental Designs Quasi-experimental designs are used when random assignment is not possible Time series designs contain a series of observations or measurements on a group or individual and the presentation of a treatment manipulation into this time series. N O X O ----------------- N O O Pretest Posttest Nonequivalent Comparison Group Interrupted Time Series Design O1 O2 O3 O4 X O5 O6 O7 O8

Example of Quasi-Experimental Designs Advantages : Greater external validity (more like real world conditions) Much more feasible given time and logistical constraints Disadvantages: Not as many variables controlled (less causal claims)

Examples of Strong-Experimental Designs Experimental designs are considered “strong” when they control for the influence of confounding extraneous variables R O X O R O O Pretest Posttest Control Group R X O R O Posttest Only Control Group Advantages Greater internal validity Causal claims can be investigated Disadvantages Less external validity (not like real world conditions) Often not practical or feasible to conduct

Class Exercise 2 Read the brief evaluation statements 1) Determine which of the designs we discussed best fits the statements. Justify your answer 2) Explain the primary threats to internal and external validity in the description.

1) In an attempt to clean up the city, a new mayor has the police department arrest prostitutes on sight. After a three-week campaign, the number of arrests for prostitution drops off to zero. The mayor holds a press conference to announce that prostitution has been driven out of the city. 2) The Army is concerned about low rates of reenlistments. A survey indicates part of the problem is that the men do not like living in the barracks with its row and rows of bunk beds. The Army randomly assigns men to one of two conditions after noting the men’s reenlistment rate. The first group is assigned to live in the barracks just as before. The second group is assigned to individual private rooms. At the end of a six-month period, the reenlistment rates for the two groups are compared.   3) A school is interested in reducing the amount of vandalism of school property. It randomly assigns students to one of two conditions. In the first, the students hear a speaker talk about the momentary and social costs involved in vandalism. After the speech, the students’ attitudes about vandalism are assessed. The second group does not hear a speaker. They are tested at the same time as the first group. 4) State legislators, concerned over the yearly increase in traffic deaths during the previous five years, passed a 55mph speed limit. During the four years after the law was passed, the yearly increases were reduced. The legislators conclude that the 55mph speed limit was responsible for the change in traffic fatalities. 5) A university library is concerned about the number of books which have been stolen in each of the past four semesters. The library personnel develop new checking-out procedures. The number of stolen books is recorded each semester for the next three semesters. At the end of the third semester, the number of stolen books has been reduced. The library administration concludes that the program was a success.