1. Randomized Control Trials: What, Why, How, When, and Where Mark L. Davison MESI Conference March 11, 2016 Department of Educational Psychology

2. What Is A Randomized Control Trial (RCT)? A randomized control trial is an experiment in which the people being studied are randomly assigned to one of several treatment conditions. –People can be students, teachers, administrators, etc. –Can also randomly assign schools, districts –Treatment conditions, e.g. No treatment vs. treatment New treatment vs. business-as- usual Department of Educational Psychology

3. Why Do an RCT? RCT designs support causal inferences; that is if there is a difference in outcomes for treatment and control, RCT supports conclusion that treatment caused the difference As an educator, you need to know what actions you can take that will CAUSE an improvement in outcomes –We need to understand the causal processes affecting outcomes if we are to do education well Not the only design that can support causal inferences

4. How to Do An RCT? Logic Model (or Theory of Change) Sampling Plan Measurements –Independent Variables –Dependent Variables –Covariates –Mediators –Moderaters Treatments, fidelity measures, and fidelity observations

5. Logic Model Theory of how intervention has its effects Include the various variables that you will include in study as measured variables. Text to accompany figure that explains how the variables are related to the change process

6. How to Do an RCT: Planning Process? Logic Model which includes –Intermediates steps between the intervention and the outcome (mediators) –Factors that may weaken or strengthen the effect of the intervention (moderators) –Alternative variables that may explain why the treatment group may have better outcomes than the control group (covariates)

7. RCT Variables Independent variable = treatment condition Dependent variable = outcome variable Covariates = variables to be statistically controlled in the analysis –Must be measured in data collection –E.g. prior achievement in studies of achievement Mediator variables = processes through which independent variables has its effects Moderator variables = variables that influence the size of the treatment effect Department of Educational Psychology

8. Logic Model Department of Educational Psychology

9. Random Sampling and the Independent Variable Random Assignment options (influence analysis strategy) –Randomly assign student When people receive the treatment independently Students in same classroom can receive different treatments Randomly assign by teacher or classroom When treatment is administered by classroom (all students in same classroom receive same treatment) When treatment is by teacher (all students with same teacher receive same treatment) Studies of classroom or teacher, policy, or interventions Randomly assign by school if treatment administered at school level (all students in same school receive same treatment) Studies of school policy Avoids cross-over effects Generally not feasible for district research (not enough schools)

10. Stratified Random Sampling Simple random sampling controls confounding variables only within limits of random sampling Stratified random sampling identifies variables to be controlled, called strata, and samples in way that controls –Sex as stratifying variable in reading Half males are randomly assigned to treatment and control; half females are randomly assigned to treatment and control

11. Stratified Random Sampling Sex and ESL as stratifying variables –Four groups: boys/ESL, boys/non-ESL, girls/ESL, girls/non-ESL –Half of each group is randomly assigned to treatment and controls Stratified random sampling actually controls stratifying variables, not just within limits of sampling

12. Sample Size Select sample size so power of detecting effect is large (e.g. >.8) Power is the probability of detecting an improvement resulting from the treatment if there is one Consult applied statistician and websites –http://sitemaker.umich.edu/group- based/optimal_design_softwarehttp://sitemaker.umich.edu/group- based/optimal_design_software

13. Dependent Variable If students are not the target of the intervention, do you want a student outcome variable? For example, –Teacher intervention designed to improve reading achievement Teacher measures e.g. pedagogical knowledge, fidelity of implementation in classroom –Student outcome Reading achievement measure, number of books read independently

14. Covariates Alternative explanations for an effect (if it occurs) and that were not controlled by stratification Used to determine if randomization well-controlled the variable Used to statistically control for alternative explanation –Test hypothesis of whether there is still a treatment effect controlling for differences in gender, prior achievement, etc. In studies of achievement, use prior achievement as covariate; increases power and controls for alternative explanation

15. Mediators Variables that represent intermediate steps in the processes leading from the intervention to the outcome School processes, teacher processes, student processes Mediator variables are used to assess whether the intervention led to the intermediate processes and whether intermediate process are related to outcome

16. Moderators Used to test hypotheses about whether the intervention is equally effective for all E.g. wonder whether intervention will be equally effective for boys and girls? Include sex as a moderator? Used to test hypotheses about whether intervention will help close gaps? E. g. Wonder if intervention will close gaps between blacks and whites? Include race as a moderator

17. Treatments Describe each treatment Treatment fidelity –Develop or select a measure of how well treatment is administered in treatment and control conditions Quantitative treatment fidelity index can be used as a mediator and to see if outcome increases as function of how well it is implemented Observe classrooms (treatment and control) to see what goes on in each treatment and how different they really are Qualitative observations of treatment and control –RCTs tend to be mixed quantitative and qualitative studies

18. When RCT? RCT canNOT be used if –Independent variable is not malleable (it is not under the control of educators) E.g. sex of students, race of students –Manipulation of independent variable by researchers would be ethically unacceptable or practically not feasible E.g. randomly assigning students to a suspension and no suspension condition E.g. randomly assigning people to high income and low income conditions

19. When RCT? Alternatives to RCT Regression discontinuity designs Assignment to treatment is based on a strict cut-off on a continuous variable Single-subject designs, such as ABA designs in which intervention is turned off and on at designated points to see if outcome changes as the intervention is turned off and on –Recognized by special education more than regular education (NCSER, but not NCER) –Must distinguish between single subject designs that show effects in one-on-one settings vs those that show effectiveness in class settings –Used when treatments have strong, immediate effects that are reversible

20. When RCT? As Research Capstone My opinion. RCT’s are best used –After “bugs” have been worked out of the intervention –http://www.whale.to/vaccine/bayly.html#THE%20SALK% 20VACCINE%20DISASTER –After research methods have been piloted in less rigorous quasi-experimental studies –After pilot studies (quasi-experimental studies or small RCTs) have provided some evidence for the intervention –In short, after a lengthy program of intervention development and pilot studies

21. Summary What? –RCT is an experiment in which people are randomly assigned to treatment groups Why? –RCT permits causal inference, and educators need to know what actions they can take to cause an improvement in student outcomes?

22. Summary How –Based on plan including logic model, sampling plan, variables (independent, dependent, mediators, moderators, covariates), treatment description, and fidelity measures and observations When and where –If feasible and ethically acceptable –If independent variable is malleable educational factor –If regression discontinuity or single subject design is less appropriate –If research design and treatment implementation have been well worked out

23. References Booth, J. L., Oyer, M. H., Pare-Blagoev, E. J., Elliott, A. J., Barbieri, C., Augustine, A., & Koedinger, K. R. (2015). Learning algebra by example in real-world classrooms. Journal of Research on Educational Effectiveness, 8(4), 530 – 551. Borman, G. D. & Dowling, N. M. (2006). Longitudinal achievement: Effects of multi-year summer school: Evidence from the Baltimore randomized field trial. Educational Evaluation and Policy Analysis, 28, 25 – 48.

24. References Fehr, C. N., Davison, M. L., Graves, M. G., Sales, G. C. & Seipel, B. E. (2012). The effect of individualized, online vocabulary instruction on picture vocabulary scores: An efficacy study. Computer Assisted Language Learning, 25(1), 87- 102. Food and Nutrition Service, U. S. Department of Agriculture (February, 2004). Evaluation of the school breakfast program pilot project. Washington, D. C.: Author.

25. What Works Clearinghouse ies.ed.gov/ncee/wwc Contains evaluations of research evidence on various interventions –Evidence for a causal effect of the intervention

26. Group Work: Research Plan Statement of research question Logic Model Research Plan –Sampling Plan –Independent Variable –Dependent Variable(s) –Covariates –Mediator variables –Moderator variables –Treatment conditions –Fidelity measure