1. S-005 Intervention research: True experiments and quasi- experiments

2. Intervention research Goal is to assess the effects of some intervention or new idea or approach Examples: –Children’s vocabulary growth under a new curriculum –Effects of new technologies on student achievement –New approaches to professional development for teachers –New drug treatments or other interventions –Other examples? Key distinction between –True experiments We compare the new program (or treatment) to a traditional treatment (or alternative treatment or control group) Random assignment is the key ingredient for a study to qualify as a true experiment –Quasi-experiments Comparison of new program with an alternative (“new” vs. “traditional”, for example) But random assignment not used Comparisons may be biased by other factors Need for great care and additional information for these studies to be convincing Quasi-experimental designs are common in education, but they may have some important limitations. (Some describe them as “mere observations” and not a strong basis for drawing conclusions. Others are not quite so harsh.)

3. True Experiments (for evaluating an intervention) New treatment Control (or traditional or alternative) New Comparison

4. True Experiments (for evaluating an intervention) New treatment Control (or traditional or alternative) New Comparison Key issue: How do people get assigned to (or chosen for) the two options? Start with a group of eligible people Assign them randomly

5. True Experiments (for evaluating an intervention) New treatment Control (or traditional or alternative) New Comparison Key issue: How do people get assigned to (or chosen for) the two options? Start with a group of eligible people Assign them randomly Random assignment is the key ingredient required for a true experiment.

6. True experiments (for assessing an intervention) New Comparison The process of random assignment Makes it very likely that the groups will be similar. The only difference will be which treatment they receive. The groups are very likely to be similar in age, gender ratio, prior experience, etc. Makes it very likely that the groups will be similar. The only difference will be which treatment they receive. The groups are very likely to be similar in age, gender ratio, prior experience, etc. Some cautions: Compliance? Implementation? Sometimes the special conditions under which we carry out a true experiment make it hard to generalize. Others? Some cautions: Compliance? Implementation? Sometimes the special conditions under which we carry out a true experiment make it hard to generalize. Others?

7. Quasi-experiments Groups are compared, but no random assignment New Comparison Do we really have a good comparison? Can we really be sure about the new program? Often we need lots of additional information to help us draw stronger conclusions?

8. Quasi-experiments New programTraditional program Post test average8580 What might we conclude?

9. Quasi-experiments Two scenarios New programTraditional program Post test average8580 Pre-test6560 New programTraditional program Post test average8580 Pre-test60

10. Quasi-experiments An even better scenario New programTraditional program Post test average8580 Pre-test6560 % girls50%52% Parent education (HS graduates) 70%74% ELL20%18%

11. Adjustment strategies in quasi-experiments New programTraditional program Post test average8580 Pre-test6560 % girls50%52% Parent education (HS graduates) 70%74% ELL20%18% Regression adjustments Multiple regression to adjust for other variables Propensity score matching See where the two groups differ and where they overlap Then find good matches for those in the new program (Private schools in Mexico example) Regression adjustments Multiple regression to adjust for other variables Propensity score matching See where the two groups differ and where they overlap Then find good matches for those in the new program (Private schools in Mexico example) What to do when there is non-random assignment?