1. Open Forum: Scaling Up and Sustaining Interventions Moderator: Carol O'Donnell, NCER Carol.Odonnell@ed.gov 202-208-3749

2. Agenda Introductions Purpose of forum Brief background from moderator and panel –Definitions of “scale-up” –Goal 4 vs. Goal 3 –Issues of scalability –Factors that influence effectiveness at scale –What does successful scale-up look like? Open Forum - Questions from the Audience

3. Introductions – Quick Survey of Audience How many Goal 2? Goal 3? Goal 4 What do you hope to get out of this forum? (record audience questions; panel will address them in their comments; see last slides)

4. Purpose of Forum A discussion of what it takes to scale-up an intervention and what it takes to sustain the fidelity of the implementation of an intervention.

5. Definitions Scale-up is the transition from idiosyncratic adoption of interventions to broad, effective implementation across a large and diverse school system. Scale-up can be demonstrated by showing a plan for the gradual and systematic implementation of the intervention.

6. Definitions Scale-up is the practice of introducing proven interventions into new settings with the goal of producing similarly positive effects in larger, more diverse populations. Scale-up research examines factors that influence the effectiveness of interventions as they are brought to scale across settings.

7. Goal 4: Scale-up Evaluations Are fully developed interventions effective when they are implemented under conditions that would be typical if a school district or other education delivery setting were to implement them (i.e., without special support from the developer or the research team) across a variety of conditions (e.g., different student populations, different types of schools)?

8. Difference Between Goal 3 and Goal 4 Key differences have to do with delivery of the intervention (“at a distance” from the researcher or developer) and the diversity of the sample (scale-up evaluations require sufficient diversity in the sample of schools, classrooms, or students to ensure appropriate generalizability).

9. Difference Between Goal 3 and Goal 4 Small-scale efficacy studies provide examples of successful interventions for some groups. Do interventions with strong prior evidence of efficacy produce a net positive increase in student learning relative to the comparison group under typical conditions in real, complex, and varied educational environments?

10. Theoretical, Empirical, and Practical Issues of Scalability What educationally meaningful effects are expected at scale? What is the intervention's theory of change? How might experience moderate outcomes? Under what conditions will the intervention be implemented? How feasible is it to implement the critical components of the intervention with fidelity at scale?

11. Theoretical, Empirical, and Practical Issues of Scalability What are the practical concerns of measuring and observing fidelity of implementation in both the treatment and comparison groups at scale, in order to identify and document critical differences between the intervention and its counterfactual?

12. Theoretical, Empirical, and Practical Issues of Scalability Is the intervention feasible (practical? affordable—cost feasibility?) for schools and other education entities to implement under normal conditions (i.e., without any support from the researchers or developers of the intervention that would not typically be available to entities wanting to implement the intervention outside of a research study)?

13. Theoretical, Empirical, and Practical Issues of Scalability Interventions that are effective at scale are those that can produce the desired effects across a range of education contexts. Under what conditions will the intervention be implemented? What methods will be used to document conditions and critical variables that affect the success of a given intervention?

14. Factors That Influence Effectiveness of Interventions at Scale Threats to validity –Diluted professional development –Leakage –Changes in the student population within a district over time –Improved experimental procedures –Maintaining fidelity of implementation over time

15. Factors That Influence Effectiveness of Interventions at Scale Experience of the teacher, school, or district with the intervention and with the study. –Does scaling-up a unit incrementally (adding more schools each year) lead to increasing effects over time, or are the effects diluted? –Ask not only, “Is the intervention effective?” but, “To what degree do the refinements at the system level and school level due to experience produce better outcomes?”

16. Effects of Scale vs Experience The effects of scale are distinguished from the effects of experience in that the effects of scale are assessed by comparing outcomes achieved in schools at different levels of implementation-- 1 st year implementers vs 2 nd year implementers during the same year of the study. Effects of experience assessed by examining whether the intervention is effective in newly implementing schools as they go to scale--1 st time implementers in Year 2 vs Year 5 (GWU).

17. What does successful scale-up look like? (RAND, 2004) Widespread implementation Deep changes in classroom and school practices Sustainability – intervention is sustained over time Sense of ownership of new practices and policies among teachers and school leaders Intervention is adaptable to local contexts

18. What does successful scale-up look like? (RAND, 2004) District and school-wide support provided for implementation Methods for ensuring high quality implementation are in place (good PD) Financial support given (intervention is cost- feasible) Building of organizational capacity Marketing of curriculum materials so they are sustainable

19. What are the important characteristics of maintaining fidelity of implementation? How do you sustain the intervention after the research is finished? How many schools are needed in a scale-up study? How do I ensure power in an RCT? In a clustered RT? (see Optimal Design Software) Questions from the Participants

20. How do you get teachers to implement with fidelity? How do you gain access to schools, especially high numbers of schools? How do you maintain relationships with schools over the course of the 5-year scale- up study? Questions from the Participants

21. What are some of the data collection strategies for studying fidelity of implementation across schools/districts and at a distance? What do we need to do in a Goal 2 to prepare for a Goal 3 (identify critical components of the intervention by the end of your Goal 2)? What do we need to do in a Goal 3 to prepare for a Goal 4?– Building bridges. Questions from the Participants

22. What does theory-driven implementation science look like? How do fidelity of implementation measures differ when studying whole school programs versus interventions used in individual classrooms? What does successful scale-up look like? Questions from the Participants

23. What does Professional Development (PD) look like at full-scale? Can PD stand on its own? Can we pay for PD in a Goal 4? Others? Questions from the Participants