URBDP 591 A Lecture 8: Experimental and Quasi-Experimental Design Objectives Basic Design Elements Experimental Designs Comparing Experimental Design Example Quasi-Experimental Designs The Nature of Good Design

Research Approaches

Basic Design Elements Time: A causal relationship implies that some time has elapsed between the occurrence of the cause and the consequent effect. Program(s) or Treatment(s). The presumed cause may be treatment under the explicit control of the researcher or the occurrence of some natural or human induced event not explicitly controlled. In design notation we depict it with the symbol "X". When multiple programs or treatments are being studied, we can use subscripts such as "X1" or "X2". Observation(s) or Measure(s). Measurements are typically depicted in design notation with the symbol "O” if the same measurement or observation is taken at every point in time in a design. If different measures are given at different times, we can use subscripts such as ”O1" or ”O2". Groups or Individuals. Typically, there will be one or more program and comparison groups. In design notation, each group is indicated on a separate line. Assignment to the group is indicated by "R" to indicate the group was randomly assigned, "N" was nonrandomly assigned and a "C" that a cutoff score or a measurement was used.

Types of Designs

A. Post-only design B. Pre and post design C. Multiple Levels of single IV D. Multiple Experimental and Control Groups E. Multiple IVs (Factorial design) Experimental Designs

Subjects randomly assigned to experimental\control groups Introduction of IV in experimental condition Measurement of DV (single or multiple instances) Post-Only Design

Group 1RXO Group 2RO RandomizedIVDV Post-Only Design

Subjects randomly assigned to experimental/ control groups Preliminary measurement of DV before before treatment (check of random assgn) Introduction of IV in experimental condition Measurement of DV (single or multiple instances) Pre and Post Design

Group 1 R O1XO2 Group 2 R O1O2 Randomized IVDV Pre- and Post- Design DV

Subjects randomly assigned to experimental\control groups Introduction of multiple levels of IV in experimental condition Measurement of DV across different conditions Multiple Levels of Single IV

Group 1RX1O Group 2RX2O Group 3RX3O Group 4RO RandomizedIVDV Multiple Levels Design

Subjects randomly assigned to experimental\control groups Preliminary measurement of DV in one exp\control pair Introduction of IV in both experimental conditions Measurement of DV (assess effects of pretest) Multiple Experimental and Control Groups (Solomon Four-Group design)

Group 1 R O1XO2 Group 2 R O1O2 Group 3 RXO2 Group 4 RO2 RandomizedIVDV Multiple Levels Design DV

Subjects randomly assigned to experimental\control groups Introduction of multiple levels of IVs in experimental condition Measurement of DV across different conditions (cells) Multiple IVs (Factorial Design)

Group 1RX1Y1O Group 2RX2Y2O Group 3RX1Y2O Group 4RX2Y1O Randomized IVDV Multiple IVs Design IV

Requirements for randomized experimental design has at least two groups has two distributions (measures), each with an average and variation assess treatment effect = statistical difference between the groups Different statistical tests

t-test and one-way Analysis of Variance (ANOVA)

The t-test

Two group posttest only randomized experimental design : T-test or One way ANOVA

Pretest-posttest randomized experimental design : ANCOVA

Pretest-posttest randomized experimental design with interaction: ANCOVA

Quasi-Experimental Design Matching instead of randomization is used. For example, someone studying the effects of growth management on residential density will try to find a similar metropolitan areas similar to the experimental metropolitan area. That other town is not technically a control group, but a comparison group, and this matching strategy is sometimes called nonequivalent group design. Time series analysis is involved. A time series is perhaps the most common type of longitudinal (over time) research found in public policy. A time series can be interrupted or noninterrupted. Both types examine changes in the dependent variable over time, with only an interrupted time series involving before and after measurement.

Non-equivalent Comparison Group Post Only, Pre-Post, Multiple Treatments No random assignment into exp\con “Create” comparison groups –Selection criteria or eligibility protocol –Partial out confounding variance – (statistical control)

Time Series Multiple observations before and after treatment or intervention is introduced Examine changes in data trends (slope and intercept) Investigate effects of both onset and offset of interventions

Time Series Group 1 N R O1 O2 XO3 Group 1 C O1 O2 O3

Regression Discontinuity Separate sample based on some criterion (pre-test) One group administered treatment, other is control group Examine trends in both groups; hypothesize equivalent

Regression Discontinuity Group 1 C O1 O2 XO3 Group 1 C O1 O2 O3

Advantages of Experiments 1. Isolation of the experimental variable 2. Allows for (relatively) easy replication 3. Establishing causality 4. Control - A true experiment offers the ultimate in control 5. Longitudinal analysis - The experiment offers the opportunity to study change over time.

Disadvantages of Experiments 1. Artificial environment 2. Experimenter effect - The experimenter's expectations can affect the results of the experiment. 3. Lack of control - Placing subjects in a laboratory may alter the very behavior the experimenter is trying to study. 4. Sample size - The larger the group, the more difficult it is to control extraneous variables.

Summary: Experiment Aim to measure the effect of IV on DV Involve experimenter’s intervention Should control for experimental error (confounding variables) Comparing the DV produced by at least two levels (conditions) of IV

The Nature of Good Design 1.Theory-Grounded. Good research strategies reflect the theories which are being investigated. Where specific theoretical expectations can be hypothesized these are incorporated into the design to improve discriminant validity and demonstrates the predictive power of the theory. 2.Situational. Good research designs reflect the settings of the investigation. This was illustrated above where a particular need of teachers and administrators was explicitly addressed in the design strategy. 3.Feasible. Good designs can be implemented. The sequence and timing of events are carefully thought out. Potential problems in measurement, adherence to assignment, database construction and the like, are anticipated. 4.Redundant. Good research designs have some flexibility built into them. Often, this flexibility results from duplication of essential design features. For example, multiple replications of a treatment help to insure that failure to implement the treatment in one setting will not invalidate the entire study. 5.Efficient. Good designs strike a balance between redundancy and the tendency to overdesign. Where it is reasonable, other, less costly, strategies for ruling out potential threats to validity are utilized.