1. WHY LARGE-SCALE RANDOMIZED CONTROL TRIALS? David Myers Senior Vice President IES 2006 Research Conference David Myers Senior Vice President IES 2006 Research Conference

2. Presentation Road Map Characteristics Benefits relative to small-scale RCTs Implementation and design challenges Characteristics Benefits relative to small-scale RCTs Implementation and design challenges

3. Characteristics Random assignment of schools/classrooms: –Multiple school districts (Reading Comprehension has about 10 districts; P4K has 27) –10 to 20 schools per “intervention arm” –200+ classrooms per “intervention arm” Random assignment of schools/classrooms: –Multiple school districts (Reading Comprehension has about 10 districts; P4K has 27) –10 to 20 schools per “intervention arm” –200+ classrooms per “intervention arm”

4. Characteristics (continued) Random assignment of students to treatment or control status with many schools/sites: –Upward Bound evaluation included a probability sample of 67 grantees and more than 3,000 students. –Charter school evaluation includes about 40 schools and 4,000 students. Random assignment of students to treatment or control status with many schools/sites: –Upward Bound evaluation included a probability sample of 67 grantees and more than 3,000 students. –Charter school evaluation includes about 40 schools and 4,000 students.

5. Characteristics (continued) Multiple years of data collection: –Baseline –One or more follow-up assessments Multiple interventions: –Reading Comp. and P4K: four each –Math Curriculum: three to four Multiple years of data collection: –Baseline –One or more follow-up assessments Multiple interventions: –Reading Comp. and P4K: four each –Math Curriculum: three to four

6. Benefits Impacts of school interventions, curricula, instruction, and professional development implemented in typical schools: –Often smaller impacts in large-scale studies (P4K) than in efficacy trials Impacts from a wider range of contexts than generally found in efficacy trials: –Students –Teachers –Schools –Greater opportunity for subgroup analyses Impacts of school interventions, curricula, instruction, and professional development implemented in typical schools: –Often smaller impacts in large-scale studies (P4K) than in efficacy trials Impacts from a wider range of contexts than generally found in efficacy trials: –Students –Teachers –Schools –Greater opportunity for subgroup analyses

7. What Questions Can an RCT Address? Intention to treat – yes Treatment on the treated – yes, with assumptions Ingredients and dosage – no design answers particularly well Intention to treat – yes Treatment on the treated – yes, with assumptions Ingredients and dosage – no design answers particularly well

8. Intention to Treat Impact of the opportunity to participate for all schools/teachers/students Simply, the difference in the means for the treated group and the control group Control group represents what we expect for treated group in absence of intervention Most policy relevant Subgroup analysis shows who benefits most Impact of the opportunity to participate for all schools/teachers/students Simply, the difference in the means for the treated group and the control group Control group represents what we expect for treated group in absence of intervention Most policy relevant Subgroup analysis shows who benefits most

9. Treatment on the Treated More assumptions Impact of intervention on those who actually receive treatment Adjusts for those who did not participate and control who did Less policy relevant because it does not show the impact of an implemented policy – TOT separates the impact on participation from the effect of the treatment Subgroup analysis shows who benefits most More assumptions Impact of intervention on those who actually receive treatment Adjusts for those who did not participate and control who did Less policy relevant because it does not show the impact of an implemented policy – TOT separates the impact on participation from the effect of the treatment Subgroup analysis shows who benefits most

10. Ingredients/Dosage Many assumptions, some unrealistic Like other designs, cannot address these questions with the same level of rigor Random assignment to different “intervention arms” of ingredients/doses needed for defensible impact estimates Many assumptions, some unrealistic Like other designs, cannot address these questions with the same level of rigor Random assignment to different “intervention arms” of ingredients/doses needed for defensible impact estimates

11. Study Implementation Challenges Objections involve interventions, not design, so negotiation teams must have substantive expertise More innovation in schools in recent years – less inclined to adopt yet another approach Districts want consistent instruction across schools Alignment of curriculum with state testing NCLB waivers – will schools and districts be held accountable (AYP) if the intervention fails? Objections involve interventions, not design, so negotiation teams must have substantive expertise More innovation in schools in recent years – less inclined to adopt yet another approach Districts want consistent instruction across schools Alignment of curriculum with state testing NCLB waivers – will schools and districts be held accountable (AYP) if the intervention fails?

12. Study Implementation Challenges (cont.) How much “intervention from the evaluator” is OK when it comes to fidelity? –Let policy and practice dictate decisions – what would developer or school do in the absence of the evaluation? Maintain normal school, program, and intervention operations – make the design accommodate the program When oversubscription is needed, programs and schools often overestimate student interest. How much “intervention from the evaluator” is OK when it comes to fidelity? –Let policy and practice dictate decisions – what would developer or school do in the absence of the evaluation? Maintain normal school, program, and intervention operations – make the design accommodate the program When oversubscription is needed, programs and schools often overestimate student interest.

13. Generic Design Challenges External validity – random effects design requires larger samples for same precision as fixed effects – demos vs. “national evaluations” Spillover or contamination from random assignment of schools to interventions Effect of giving some students treatment on the outcomes for the control group (P4K) Identifying the counterfactual External validity – random effects design requires larger samples for same precision as fixed effects – demos vs. “national evaluations” Spillover or contamination from random assignment of schools to interventions Effect of giving some students treatment on the outcomes for the control group (P4K) Identifying the counterfactual

14. Design Challenges: Case Study Charter schools evaluation: –Research question concerns the impact of attending a charter school  How would a child attending a charter school have performed in the absence of that school? –Design is random assignment to charter/regular public schools (lottery) –All designs are limited here:  A nonrandom sample of students left the public school, which may change what is taught there  Different than the thought experiment of asking what would happen with no charter schools Charter schools evaluation: –Research question concerns the impact of attending a charter school  How would a child attending a charter school have performed in the absence of that school? –Design is random assignment to charter/regular public schools (lottery) –All designs are limited here:  A nonrandom sample of students left the public school, which may change what is taught there  Different than the thought experiment of asking what would happen with no charter schools

15. Another Case Study: P4K What is the impact of being assigned to a pullout program? –How would students perform in the absence of the pullout program? Pullout program: –RA of schools to one of four interventions –Within schools, RA of eligible students to the pullout program or the regular classroom Change in the composition of the regular classroom and perhaps, change in curriculum and instruction when some struggling readers are removed What is the impact of being assigned to a pullout program? –How would students perform in the absence of the pullout program? Pullout program: –RA of schools to one of four interventions –Within schools, RA of eligible students to the pullout program or the regular classroom Change in the composition of the regular classroom and perhaps, change in curriculum and instruction when some struggling readers are removed