HDFS 361—Research Methods Experimental and Quasi- Experimental Design

Two or more groups or treatment conditions Random assignment of participants to each of these groups Exposure of each group to a different level of an independent variable or to different levels of an intervention What is an Experiment? Experiments, be they qualitative or quantitative, involve three essential features:

Randomization Randomization is a process whereby members of a sample (i.e., individual participants) are randomly distributed throughout experimental conditions or groups. Randomization assures that qualities of individual participants that might somehow influence the outcome of an experiment are not systematically concentrated in any particular experimental condition.

Groups and Manipulation of Exposure Control groups are experimental conditions where participants are not exposed to independent variables of interest or are exposed at very different levels than are subjects in treatment groups. Treatment groups are experimental conditions where participants are exposed to independent variables of interest. The exposure to the independent variable is manipulated.

Randomization of Maria’s Participants to Experimental Conditions Maria begins by listing her 40 students in alphabetical order and assigning each of them a number from 1 to 40. If she had 50 students, she would use numbers 1 to 50. She lists students’ names in alphabetical order, but any order is acceptable because random assignment to groups is made on the basis of random numbers. She then lists the numbers she assigned to individuals in order (without the names) as shown in the left two columns of Figure 1.1.

Randomization of Maria’s Participants to Experimental Conditions Next, she begins the process of random assignment to groups by randomly selecting 20 students to be in the treatment group. She selects numbers at random beginning at any point in Appendix A’s table of random numbers. The first number between 1 and 40 is the number 12 so she records a 12 in the box labeled Treatment Group. This means the 12 th person is going to be assigned to the treatment group and we put a check by participant number 12. Continuing down the column, the next number in the range of 1 to 40 is the number 29, so she checks this number and writes it under the control group.

| id | |-----| 1. | 118 | 2. | 156 | 3. | 182 | 4. | 180 | 5. | 57 | |-----| 6. | 12 | 7. | 73 | 8. | 174 | 9. | 29 | 10. | 8 | |-----| 11. | 86 | 12. | 161 | 13. | 71 | 14. | 51 | 15. | 129 | First 15 numbers from Appendix A

Randomization Treatment Group 12,29,8,38,6,22,1,33,11,19,36,23,26,15,9,18,39,35,4,7 Control Group 2,3,4,5,7,10,13,14,16,17,20,21,22,24,25,27,28,30,31, 32,34,37,40 We put the first 20 numbers between 1 and 20 in the treatment group and the remaining 20 in the control group

Internal Validity A experiment’s internal validity refers to its ability to rule out the influence of extraneous variables on the dependent variable of interest. A study with high internal validity eliminates threats from extraneous variables such that any influence on the dependent variable can be assumed to proceed from the independent variable. For example, randomization of participants to groups strengthens internal validity by insuring that groups are not systematically different on extraneous variables.

Internal Validity—Selection Bias Selection bias refers to how participants are selected into treatment and control groups in an experiment. Whenever any source of bias operates in the process of composing treatment and control groups in a study, that bias threatens the internal validity of the study.

Internal Validity—History The threat of history is the possibility that an external event occurring during the course of an experiment may influence how participants score on dependent measures. For example, a natural disaster could happen during the course of the study.

Internal Validity—Maturation The threat of maturity regards the possibility that an internal change participants may go through during the course of an experiment may somehow influence how they score on dependent measures. Adolescence is a period where this can be a serious problem.

Internal Validity—Testing The threat of testing regards the possibility that a participant’s experience taking pretest measures influence how they score on posttest measures. A pretest measuring knowledge of the adverse effects of smoking may sensitize people to these risks and be the real reason they quit.

Internal Validity—Mortality/Attrition The threat of mortality or attrition regards the possibility participants may drop out of a study for some systematic reason, leaving only participants with certain qualities to take dependent measures. Program to reduce delinquency may have the more delinquent participants quit and those who are left make the program look better than it is.

Internal Validity—Statistical Regression The threat of statistical regression toward the mean is the possibility that a subject who scores extremely high or low on a pretest measurement of a dependent variable may naturally score closer to the mean for their group at the posttest measurement of the same variable. The difference between pretest and posttest scores for such a subject may not reflect the influence of an independent variable on the dependent variable so much as it may reflect the natural tendency for people who achieve extreme scores at one point in time to achieve less than extreme scores at a later time.

External Validity An experiment’s external validity refers to the ability to generalize beyond the study to a larger population, realistic circumstances, and other periods of time. A study with high external validity will guarantee that what it found to be true for its participants should be true for the population of people those participants were meant to represent. For example, a random sample of participants to include in the study strengthens external validity by insuring that the participants are not systematically different from the population.

External Validity—Sampling Bias Sampling bias refers to the possibility that the sample of participants in an experiment is somehow biased. An unbiased sample would accurately mirror the population it is meant to represent. Ideally the pool of participants in a study are a probability sample of a larger population.

External Validity—Environmental Validity An experiment has a high level of environmental validity when the conditions and circumstances of the experiment accurately mirror the reality to which the study is generalized Many studies are done in artificial situations and these do not generalize to the real world

External Validity—Demand Characteristics Demand characteristics of a study threaten external validity by influencing responses of participants. These characteristics essentially compromise the ability of a sample to accurately represent the population from which it was drawn A double blind procedure, where staff who interact with study participants are not told the research hypothesis or which group participants are in, can mitigate demand characteristics

Experimental Designs Posttest-Only Treatment and Control Group GroupRandomi- zation Pretest Measurement of Dependent Variable Exposure to the Independent Variable Posttest Measure- ment of Dependent Variable Treatment Group YesNot applicableYes Control Group YesNot applicableNoYes

Experimental Designs Pretest-Posttest Treatment and Control Group GroupRandomiz- ation Pretest Measurement of Dependent Variable Exposure to the Independent Variable Posttest Measurement of Dependent Variable Treatment Group Yes Control Group Yes NoYes

Experiment--Solomon Four-Group Design GroupRandomizationPretest Measurement of Dependent Variable Exposure to the Independent Variable Posttest Measurement of Dependent Variable Treatment Group I Yes Control Group I Yes NoYes Treatment Group II YesNot applicableYes Control Group II YesNot applicableNoYes

Pseudo Experiment--After Only Case Study GroupRandomi- zation Pre-test Measureme nt of Dependent Variable Exposure to the Independent Variable Posttest Measure- ment of Dependent Variable Treatment Group NoNot applicableYes No Control Group Not applicable

Pseudo Experiment—Before-After Case Study GroupRandomi- zation Pretest Measurement of Dependent Variable Exposure to the Independent Variable Posttest Measurement of Dependent Variable Treatment Group NoYes No Control Group Not applicable

Pseudo Experiment--Posttest-Only Nonequivalent Group Design GroupRandomi- zation or Matching Pretest Measurement of Dependent Variable Exposure to the Independent Variable Posttest Measurement of Dependent Variable Treatment GroupNo Yes Nonequivalent Group No Yes

Quasi Experiment—Pretest Posttest