Randomized Control Trials: What, Why, How, When, and Where Mark L. Davison MESI Conference March 11, 2016 Department of Educational Psychology.

Slides:



Advertisements
Similar presentations
Experimental Research Designs
Advertisements

1 Language of Research Adapted from The Research Methods Knowledge Base, William Trochim (2006). & Methods for Social Researchers in Developing Counries,
Research in Psychology Chapter Two
Who are the participants? Creating a Quality Sample 47:269: Research Methods I Dr. Leonard March 22, 2010.
Understanding the Research Process
Evaluating Hypotheses Chapter 9. Descriptive vs. Inferential Statistics n Descriptive l quantitative descriptions of characteristics.
Chapter One The Nature of Probability and Statistics.
Impact Evaluation: The case of Bogotá’s concession schools Felipe Barrera-Osorio World Bank 1 October 2010.
Aaker, Kumar, Day Seventh Edition Instructor’s Presentation Slides
S-005 Types of research in education. Types of research A wide variety of approaches: –Theoretical studies –Summaries of studies Reviews of the literature.
Association vs. Causation
Experimental Design The Gold Standard?.
Fig Theory construction. A good theory will generate a host of testable hypotheses. In a typical study, only one or a few of these hypotheses can.
Testing Hypotheses.
Chapter 2: The Research Enterprise in Psychology
Research Methods Unit 2 (Chapter 2).
Dr. Bonnie J. Faddis & Dr. Margaret Beam RMC Research Fidelity of Implementation and Program Impact.
Moving from Development to Efficacy & Intervention Fidelity Topics National Center for Special Education Research Grantee Meeting: June 28, 2010.
Methodology Describe Context & setting Design Participants Sampling Power Analysi s Interventions Outcome (study variables) Data Collection Procedures.
Research Methodology For IB Psychology Students. Empirical Investigation The collecting of objective information firsthand, by making careful measurements.
Final Study Guide Research Design. Experimental Research.
The Research Enterprise in Psychology. The Scientific Method: Terminology Operational definitions are used to clarify precisely what is meant by each.
S-005 Intervention research: True experiments and quasi- experiments.
Evaluating a Research Report
Overview of Evaluation Designs. Learning objectives By the end of this presentation, you will be able to: Explain evaluation design Describe the differences.
Chapter 2 The Research Enterprise in Psychology. Table of Contents The Scientific Approach: A Search for Laws Basic assumption: events are governed by.
The Research Enterprise in Psychology
Seeking Funding from the Institute of Education Sciences A Guide for WCER Researchers.
Quantitative and Qualitative Approaches
An Introduction to Statistics and Research Design
Unit 1 – Intro to Statistics Terminology Sampling and Bias Experimental versus Observational Studies Experimental Design.
OSEP Project Directors’ Conference Washington, DC July 21, 2008 Tools for Bridging the Research to Practice Gap Mary Wagner, Ph.D. SRI International.
Research Methods Objectives –Understanding sampling –Understanding different research designs –Understanding strengths and weaknesses of different designs.
A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality Matthew Linick & Diane Fuselier-Thompson.
© 2011 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license.
Chapter 2 The Research Enterprise in Psychology. Table of Contents The Scientific Approach: A Search for Laws Basic assumption: events are governed by.
Experimentation in Computer Science (Part 2). Experimentation in Software Engineering --- Outline  Empirical Strategies  Measurement  Experiment Process.
S-005 Types of research in education. Types of research A wide variety of approaches: –Theoretical studies –Summaries of studies Reviews of the literature.
Aim: What factors must we consider to make an experimental design?
Understanding Quantitative Research Design
HMS 320 Quincy University Dr. David Shinn Spring 2009.
Developing an evaluation of professional development Webinar #2: Going deeper into planning the design 1.
EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013.
QUANTITATIVE METHODS I203 Social and Organizational Issues of Information For Fun and Profit.
Research in Psychology Chapter Two 8-10% of Exam AP Psychology.
Producing Data 1.
Evidence-Based Mental Health PSYC 377. Structure of the Presentation 1. Describe EBP issues 2. Categorize EBP issues 3. Assess the quality of ‘evidence’
Chapter 11 Experimental Designs PowerPoint presentation developed by: Sarah E. Bledsoe & E. Roberto Orellana.
Experimental Design Ragu, Nickola, Marina, & Shannon.
Chapter 6 Selecting a Design. Research Design The overall approach to the study that details all the major components describing how the research will.
Unit 1 Section 1.3.
Issues in Evaluating Educational Research
CHAPTER 4 Designing Studies
EXPERIMENTAL RESEARCH
CHAPTER 4 Designing Studies
Teaching and Educational Psychology
CHAPTER 4 Designing Studies
CHAPTER 4 Designing Studies
Explanation of slide: Logos, to show while the audience arrive.
CHAPTER 4 Designing Studies
CHAPTER 4 Designing Studies
CHAPTER 4 Designing Studies
CHAPTER 4 Designing Studies
Psychological Research Methods and Statistics
CHAPTER 4 Designing Studies
Reminder for next week CUELT Conference.
Enhancing Causal Inference in Observational Studies
CHAPTER 4 Designing Studies
Misc Internal Validity Scenarios External Validity Construct Validity
Enhancing Causal Inference in Observational Studies
Presentation transcript:

Randomized Control Trials: What, Why, How, When, and Where Mark L. Davison MESI Conference March 11, 2016 Department of Educational Psychology

What Is A Randomized Control Trial (RCT)? A randomized control trial is an experiment in which the people being studied are randomly assigned to one of several treatment conditions. –People can be students, teachers, administrators, etc. –Can also randomly assign schools, districts –Treatment conditions, e.g. No treatment vs. treatment New treatment vs. business-as- usual Department of Educational Psychology

Why Do an RCT? RCT designs support causal inferences; that is if there is a difference in outcomes for treatment and control, RCT supports conclusion that treatment caused the difference As an educator, you need to know what actions you can take that will CAUSE an improvement in outcomes –We need to understand the causal processes affecting outcomes if we are to do education well Not the only design that can support causal inferences

How to Do An RCT? Logic Model (or Theory of Change) Sampling Plan Measurements –Independent Variables –Dependent Variables –Covariates –Mediators –Moderaters Treatments, fidelity measures, and fidelity observations

Logic Model Theory of how intervention has its effects Include the various variables that you will include in study as measured variables. Text to accompany figure that explains how the variables are related to the change process

How to Do an RCT: Planning Process? Logic Model which includes –Intermediates steps between the intervention and the outcome (mediators) –Factors that may weaken or strengthen the effect of the intervention (moderators) –Alternative variables that may explain why the treatment group may have better outcomes than the control group (covariates)

RCT Variables Independent variable = treatment condition Dependent variable = outcome variable Covariates = variables to be statistically controlled in the analysis –Must be measured in data collection –E.g. prior achievement in studies of achievement Mediator variables = processes through which independent variables has its effects Moderator variables = variables that influence the size of the treatment effect Department of Educational Psychology

Logic Model Department of Educational Psychology

Random Sampling and the Independent Variable Random Assignment options (influence analysis strategy) –Randomly assign student When people receive the treatment independently Students in same classroom can receive different treatments Randomly assign by teacher or classroom When treatment is administered by classroom (all students in same classroom receive same treatment) When treatment is by teacher (all students with same teacher receive same treatment) Studies of classroom or teacher, policy, or interventions Randomly assign by school if treatment administered at school level (all students in same school receive same treatment) Studies of school policy Avoids cross-over effects Generally not feasible for district research (not enough schools)

Stratified Random Sampling Simple random sampling controls confounding variables only within limits of random sampling Stratified random sampling identifies variables to be controlled, called strata, and samples in way that controls –Sex as stratifying variable in reading Half males are randomly assigned to treatment and control; half females are randomly assigned to treatment and control

Stratified Random Sampling Sex and ESL as stratifying variables –Four groups: boys/ESL, boys/non-ESL, girls/ESL, girls/non-ESL –Half of each group is randomly assigned to treatment and controls Stratified random sampling actually controls stratifying variables, not just within limits of sampling

Sample Size Select sample size so power of detecting effect is large (e.g. >.8) Power is the probability of detecting an improvement resulting from the treatment if there is one Consult applied statistician and websites – based/optimal_design_softwarehttp://sitemaker.umich.edu/group- based/optimal_design_software

Dependent Variable If students are not the target of the intervention, do you want a student outcome variable? For example, –Teacher intervention designed to improve reading achievement Teacher measures e.g. pedagogical knowledge, fidelity of implementation in classroom –Student outcome Reading achievement measure, number of books read independently

Covariates Alternative explanations for an effect (if it occurs) and that were not controlled by stratification Used to determine if randomization well-controlled the variable Used to statistically control for alternative explanation –Test hypothesis of whether there is still a treatment effect controlling for differences in gender, prior achievement, etc. In studies of achievement, use prior achievement as covariate; increases power and controls for alternative explanation

Mediators Variables that represent intermediate steps in the processes leading from the intervention to the outcome School processes, teacher processes, student processes Mediator variables are used to assess whether the intervention led to the intermediate processes and whether intermediate process are related to outcome

Moderators Used to test hypotheses about whether the intervention is equally effective for all E.g. wonder whether intervention will be equally effective for boys and girls? Include sex as a moderator? Used to test hypotheses about whether intervention will help close gaps? E. g. Wonder if intervention will close gaps between blacks and whites? Include race as a moderator

Treatments Describe each treatment Treatment fidelity –Develop or select a measure of how well treatment is administered in treatment and control conditions Quantitative treatment fidelity index can be used as a mediator and to see if outcome increases as function of how well it is implemented Observe classrooms (treatment and control) to see what goes on in each treatment and how different they really are Qualitative observations of treatment and control –RCTs tend to be mixed quantitative and qualitative studies

When RCT? RCT canNOT be used if –Independent variable is not malleable (it is not under the control of educators) E.g. sex of students, race of students –Manipulation of independent variable by researchers would be ethically unacceptable or practically not feasible E.g. randomly assigning students to a suspension and no suspension condition E.g. randomly assigning people to high income and low income conditions

When RCT? Alternatives to RCT Regression discontinuity designs Assignment to treatment is based on a strict cut-off on a continuous variable Single-subject designs, such as ABA designs in which intervention is turned off and on at designated points to see if outcome changes as the intervention is turned off and on –Recognized by special education more than regular education (NCSER, but not NCER) –Must distinguish between single subject designs that show effects in one-on-one settings vs those that show effectiveness in class settings –Used when treatments have strong, immediate effects that are reversible

When RCT? As Research Capstone My opinion. RCT’s are best used –After “bugs” have been worked out of the intervention – 20VACCINE%20DISASTER –After research methods have been piloted in less rigorous quasi-experimental studies –After pilot studies (quasi-experimental studies or small RCTs) have provided some evidence for the intervention –In short, after a lengthy program of intervention development and pilot studies

Summary What? –RCT is an experiment in which people are randomly assigned to treatment groups Why? –RCT permits causal inference, and educators need to know what actions they can take to cause an improvement in student outcomes?

Summary How –Based on plan including logic model, sampling plan, variables (independent, dependent, mediators, moderators, covariates), treatment description, and fidelity measures and observations When and where –If feasible and ethically acceptable –If independent variable is malleable educational factor –If regression discontinuity or single subject design is less appropriate –If research design and treatment implementation have been well worked out

References Booth, J. L., Oyer, M. H., Pare-Blagoev, E. J., Elliott, A. J., Barbieri, C., Augustine, A., & Koedinger, K. R. (2015). Learning algebra by example in real-world classrooms. Journal of Research on Educational Effectiveness, 8(4), 530 – 551. Borman, G. D. & Dowling, N. M. (2006). Longitudinal achievement: Effects of multi-year summer school: Evidence from the Baltimore randomized field trial. Educational Evaluation and Policy Analysis, 28, 25 – 48.

References Fehr, C. N., Davison, M. L., Graves, M. G., Sales, G. C. & Seipel, B. E. (2012). The effect of individualized, online vocabulary instruction on picture vocabulary scores: An efficacy study. Computer Assisted Language Learning, 25(1), Food and Nutrition Service, U. S. Department of Agriculture (February, 2004). Evaluation of the school breakfast program pilot project. Washington, D. C.: Author.

What Works Clearinghouse ies.ed.gov/ncee/wwc Contains evaluations of research evidence on various interventions –Evidence for a causal effect of the intervention

Group Work: Research Plan Statement of research question Logic Model Research Plan –Sampling Plan –Independent Variable –Dependent Variable(s) –Covariates –Mediator variables –Moderator variables –Treatment conditions –Fidelity measure