Quasi-Experimental Design 10 Quasi-Experimental Design

10.1 Foundations of Quasi-Experimental Design “Quasi” means “sort of” Quasi-experiments have: A control group A treatment (or program) group Variables Quasi-experiments do not have: Random assignment to groups

10-2 The Nonequivalent Groups Design One of the most frequently used quasi-experimental designs Looks just like a pretest-posttest design Lacks random assignment to groups As a result, the treatment and control groups may be different at the study’s start Raises a selection threat to internal validity Figure 10.2 Notation for the Nonequivalent-Groups Design (NEGD).

10.2a Reaching Cause-and-Effect Conclusions with the NEGD Figure 10.3 This is a flowchart of a quasi-experimental study to examine the effects of providing job resources and recovery opportunities on the health, well-being, and performance of nursing home staff (Spoor, de Jonge, & Hamers, 2010). Flow charts like this are very helpful in communicating study design.

10.2a Plot of Pretest and Posttest Means for Possible Outcome 1 Figure 10.6 Plot of pretest and posttest means for possible outcome. Here, there are selection threats. Selection threats are any factor other than the program that leads to posttest differences between groups. The comparison group started with a lower score on the pretest than the program group, so the groups were not equivalent from the start. The comparison group did not change from pretest to posttest, while the program group did—they improved. This could be due to selection-history effects, selection-maturation effects, or selection-regression effects.

10.2a Plot of Pretest and Posttest Means for Possible Outcome 2 Figure 10.7 Plot of pretest and posttest means for possible outcome 2. Here, both the program and the comparison group improve from pretest to posttest, but there are still differences between the groups from the start of the study. Selection-testing, selection-instrumentation, and selection-mortality are all plausible explanations for this outcome.

10.2a Plot of Pretest and Posttest Means for Possible Outcome 3 Figure 10.8 Plot of pretest and posttest means for possible outcome 3 Here, a selection-regression effect is the most likely explanation for the results. A selection-regression threat is a threat to internal validity that occurs when there are different rates of regression to the mean in the two groups. The comparison group starts lower than the program group, and does not change throughout the study. The program group starts out very high on the pretest, and there is a sharp drop in the group’s performance, which is likely due to regression toward the mean.

10.2a Plot of Pretest and Posttest Means for Possible Outcome 4 Figure 10.9 Plot of pretest and posttest means for possible outcome 4. Here, a selection-regression threat is also a likely explanation. The program group starts out considerably lower than the comparison group, and shows a sharp increase in performance. The comparison group does not change. This is likely because the program group is regressing up toward the mean.

10.2a Plot of Pretest and Posttest Means for Possible Outcome 5 Figure 10.10 Plot of pretest and posttest means for possible outcome 5. Here, most threats to internal validity are unlikely, and the treatment is likely causing the result.

10.3 The Regression-Discontinuity Design A pretest- posttest program comparison- group quasi-experimental design in which a cutoff criterion on the preprogram measure is the method of assignment to a group

10.3a The Basic RD Design Notation C indicates that groups are assigned by means of a cutoff score on the premeasure An O stands for the administration of a measure to a group. An X depicts the implementation of a program Each group is described on a single line Figure 10.11 Notation for the Regression-Discontinuity (RD) design.

10.3a Regression Line A line that describes the relationship between two or more variables Figure 10.12 Pre-post distribution for an RD design with no treatment effect. Figure 10.13 The RD design with ten-point treatment effect.

10.3b The Role of the Comparison Group in RD Designs Figure 10.14 Regression lines for the data shown in Figure 10.13. Figure 10.15 How the RD design got its name

10.3c The Internal Validity of the RD Design In principle, then, the RD design is as strong in internal validity as its randomized experimental alternatives In practice, however, the validity of the RD design depends directly on how well you can model the true pre-post relationship, certainly a serious statistical challenge

10.3d Statistical Power and the RD Design To achieve the same level of statistical accuracy, an RD design needs as much as 2.75 times the participants as a randomized experiment Example: if a randomized experiment needs 100 participants to achieve a certain level of power, the RD design might need as many as 275

10.3e Ethics and the RD Design RD designs tend to be more ethical, because those who need a program or treatment the most can receive it

10.4a The Proxy Pretest Design A post-only design in which, after the fact, a pretest measure is constructed from preexisting data Usually done to make up for the fact that the research did not include a true pretest Figure 10.16 The Proxy- Pretest design.

10.4b The Separate Pre-Post Samples Design A design in which the people who receive the pretest are not the same as the people who take the posttest Figure 10.17 The Separate Pre-Post Samples design.

10.4c The Double-Pretest Design A design that includes two waves of measurement prior to the program Addresses selection-maturation threats Figure 10.19 The Double-Pretest design.

10.4d The Switching-Replications Design A two-group design in two phases defined by three waves of measurement In the repetition of the treatment, the two groups switch roles: The original control group in phase 1 becomes the treatment group in phase 2, whereas the original treatment group acts as the control Figure 10.20 The Switching-Replications design.

10.4e The Nonequivalent Dependent Variables (NEDV) Design At first, looks like a weak design But pattern matching gives researchers a powerful tool for assessing causality The degree of correspondence between two data items Figure 10.21 The NEDV design. Figure 10.23 Example of a Pattern-Matching variation of the NEDV design.

10.4f The Regression Point Displacement (RPD) Design A pre-post quasi-experimental research design where the treatment is given to only one unit in the sample, with all remaining units acting as controls This design is particularly useful to study the effects of community level interventions Figure 10.24 The Regression Point Displacement (RPD) design

10.4f The Regression Point Displacement (RPD) Design Figure 10.25 An example of the RPD design Figure 10.26 How the RPD design got its name ANCOVA: An analysis that estimates the difference between the groups on the posttest after adjusting for differences on the pretest Analyze with ANCOVA

Discuss and Debate Why can quasi-experiments be more ethical than randomized experiments? What are the strengths and the weaknesses of quasi-experimental designs? In a quasi-experiment, those participants who may need a treatment or program the most can be included in the experimental group, whereas in a randomized experiment, they cannot.   Quasi-experimental designs can be preferable to randomized experiments for ethical reasons and also for logistical reasons (sometimes, randomization is not possible due to the research topic or accessible population). However, quasi-experiments have internal validity problems that must be addressed in both the design and the analysis of the study. Sometimes, these designs require more participants than a randomized study, in order to increase statistical power.