1. Scaling up and Sustaining Evidence-based Practices Glen Dunlap, George Sugai, Tim Lewis, Steve Goodman, Rob Horner www.pbis.org

2. Goals  Define the features and procedures for moving evidence-based educational practices from demonstrations to large-scale adoptions.  Use School-wide Positive Behavior Support as one example of large-scale implementation

3. Main Themes  To take educational innovations to scale begin with Valued Outcomes  The outcomes need to be valued  The outcomes need to be comprehensive

4. Phases in Scaling of Evidence-based Practices Emergence Establishing Demonstrations/Capacity Elaboration Systems Adoptions

5. Phases of Implementation  Emergence  Define Innovation with precision  Define Supporting Systems  Define Implementation Process  Awareness dissemination What is the innovation? Is it evidence-based? Is it conceptually coherent? Why is it effective? How is it more efficient than what we currently do?

6. Phases of Implementation  Demonstration  Documentation that innovative can be implemented locally with (a) fidelity, and (b) effect on valued outcomes.  Provide demonstrations (1-50)  Repeated demonstrations in multiple contexts (parts of the state, urban centers, different grade levels) may be needed.  Demonstrations typically are done at greater expense than is sustainable or scalable, but are justified as examples that the innovation “can be done here”  Build infra-structure for scaling  State policy  State training and support capacity  Information systems

7. Phases of Implementation  Elaboration  Shift from demonstration to broad implementation Use local trainers Presentation by local demonstration sites Many distributed (more cost effective) trainings Training at multiple organizational levels  Administrators  School boards  Instructional staff  Specialists (e.g. behavior specialists, school psychologists, social workers, counselors)  Families  Disseminate outcome data  Conduct and disseminate comparative cost data

8. Phases of Implementation  System Adoption/Sustainability Innovation is integrated into policy  Job descriptions  Hiring announcements  Annual personnel orientation Regular reporting of data  Are we implementing evidence-based practices  Are we producing the effects we want for children Investment in continuous regeneration  Implement  evaluate  adapt

9. Sustaining SWPBS Implementation Jennifer Doolittle University of Oregon 2006

10. Method  285 schools who have been involved in implementing SWPBS for at least 3 years. 71 not to criterion yet 74 met 80%/80% criterion on SET but did NOT sustain for two years. 140 met 80%/80% criterion on SET and did sustain.  School-wide Evaluation Tool  Logistic Regression

11. Doolittle (2006) SET SubscaleSustainability StatusPair-wise Comparison Effect Sizes1 Non-Imp (N = 71) Non-Main (N = 74) Main (N =140) MSDM M F-valueNon-Imp vs. Non-Main Non-Imp vs. Main Non-Main vs. Main Expectations defined0.58a0.280.81b0.210.89c0.1651.13***0.941.410.43 Behavioral expectations taught 0.470.280.810.190.900.13119.15*** 1.452.10 0.56 On-going behavioral reward system 0.530.370.810.240.950.1276.29***0.921.71 0.78 System responding to behavioral violations 0.630.210.710.200.810.1426.13***0.391.030.59 Monitoring and decision making 0.680.250.890.170.950.1159.13*** 1.001.50 0.43 Management0.660.260.840.150.930.1060.39***0.881.50 0.72 District-level support0.750.270.830.250.870.236.07**0.310.480.17 *p <.05; **p <.01; ***p <.001 1Effect sized used is d-statistic and interpreted as.2 = small effect,.5 = medium effect,.8 = large effect (Cohen, 1988). Notes. SET = School-wide Evaluation Tool; M = mean; SD = standard deviation. Means in the same row with different subscripts significantly differ at p <.05.

12. Results: Predictors of Implementation * Effect sized used is d-statistic and interpreted as.2 = small effect,.5 = medium effect,.8 = large effect (Cohen, 1988).

13. Results: Predictors of Sustained Implementation * * Effect sized used is d-statistic and interpreted as.2 = small effect,.5 = medium effect,.8 = large effect (Cohen, 1988).

14. Summary  The variables that were most relevant for initial implementation were DIFFERENT from the variables that affected sustainability.

15. Summary for Sustaining and Scaling  Begin with the Valued Outcomes  Innovations need to be more than effective: Comprehensive Efficient Research-based Dramatic improvement over what already exits.  The process of implementation changes as the scale increases Increased efficiency Increased emphasis on local capacity  Large scale implementation requires sustained effect Continuous regeneration.