1. Scaling Up Evidence-based Practices Dean Fixsen, Karen Blase, Rob Horner, and George Sugai

2. Current Challenges Effective practices exist for literacy, social behavior and safety Traditional TA approaches (dissemination and compliance) are insufficient Methods are available to reliably produce benefits to students statewide © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008

3. Currently EBPs now are boutique operations ◦ Now have convincing demonstrations that EBPs can work in the real world ◦ This is encouraging, but EBPs are not used on a sustainable scale sufficient to solve social problems

4. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Future Start with the end in mind What will it take to: ◦ make statewide use of education innovations ◦ that produce increasingly effective outcomes ◦ for the next 50 years?

5. Problem Most States already have attempted to implement a variety of these science-based practices Some attempts achieve their purposes, but others fall short © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008

6. Key Issues Four big issues have emerged: ◦ Select what to scale up ◦ Determine how to implement new education practices so they actually produce the intended benefits to students (effective) ◦ Define how to scale up effective practices so they are available to all students who could benefit from them (effective, accessible) ◦ Define how to align system structures and functions to fully support scale up efforts as part of “education as usual” (effective, accessible, sustainable)

7. When is a “practice” ready for TA Investment Defining a “practice” ◦ A “practice” is a procedure, or set of procedures, designed for use in a specific context, by individuals with certain skills/features, to produce specific outcomes for specific individuals. Operationally defined procedures Target population/ Context Implementer Characteristics Defined outcomes

8. The scope/size of a practice matters Practices can be small, medium or large in scope ◦ Time delay ◦ Functional behavioral assessment ◦ First Step to Success Scale up practices only when they are bundled to a level where they have impact on a core social outcome. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008

9. When is a “practice” ready for TA Investment Wisdom from:  Karin Guldbrandsson  H. Gotham  Dean Fixsen and Karen Blase  George Sugai  Bruce Joyce and Beverly Showers Eight Features of a “practice” that make it “ready for implementation” investment

10. When is a “practice” ready for TA Investment Substantive impact on highly valued outcome.  Practice is comprehensive enough that it addresses a highly valued outcome (not just a piece)  There is a perceived need for the outcome  Effect is dramatic (both in relative and absolute terms) The practice is defined with precision  The practice can be described with operational precision and substance.  What is the “it” Compatibility with existing practices

11. When is a “practice” ready for TA Investment Evidence-based  Experimental documentation of relationship between practiced and impact published in peer- reviewed journal. (Effect Size of at least.50)  Emerging  Promising  Evidence-based Efficiency  Minimal additional resources to perform new practice  Minimal additional resources to transition to new practice

12. When is a “practice” ready for TA Investment Social Validity  The people required to behave differently are comfortable with the new behavior(s) they are being asked to perform  Documentation of effectiveness trials Adaptability  The practice can be adapted to “fit” different contexts  The challenge of being precise and manualized versus being flexible and adaptive (be clear about core). Generalizability  The practice is effective across the array of contexts normally encountered.

13. Valued Outcomes Practice Implementation Identifying & Modifying Practices Efficiency EffectivenessPriority Continuous Regeneration Continuous Measurement Data- Based Prob. Solving Capacity Building

14. What “packaging features” are necessary for successful adoption? Materials for exploration and awareness  What is the need, what is the practice, what is the documented link between the practice and valued outcome? Materials for Installation and Initial Implementation  Materials for teaching people to master the new practice  Protocol for implementation

15. What “package features” are necessary for successful adoption? Evaluation tools  Is practice being implemented with fidelity  Is practice producing valued outcome  Are training segments producing implementation of core elements of “practice”

16. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Research Measures Annual Self- Assessment Measures Progress Monitoring Measures Universal Tier of SWPBS -School-wide Evaluation Tool (SET) -Self-Assessment Survey (SAS) -Benchmarks of Quality (BoQ) -Team Implementation Checklist (TIC) Secondary and Tertiary Tiers of SWPBS -Individual Student School- wide Evaluation Tool (I-SSET) -Benchmarks of Advanced Tiers (BAT) (To be developed) Overall Summary of Implementation -Implementation Phases Inventory (IPI) -Phases of Implementation (POI)

17. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 ActivityActivity Task AnalysisWhoWhen 1.Establish Commitment  Administrator  Top 3 goal  80% of faculty  Three year timeline a. b. c. d. e. 2. Establish Team  Representative  Administrator  Effective team operating procedures  Audit of teams/initiatives a. b. c. d. e. Evaluation Guided Action Planning

18. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Key Issues Interventions that are and remain effective through several generations of teachers, principals, superintendents, and State and national leaders Implementation supports that are and remain effective through several generations of trainers, coaches, evaluators, administrators, and State leaders

19. Teaching–Family Replications © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 0 100 200 300 400 500 600 700 800 900 ≤197219741976197819801982 0 50 100 150 200 250 300 Cumulative Couples Cumulative Homes Fixsen, Blase, Timbers, & Wolf (2001)

20. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Follow Through Programs Figure 1: This figure shows the average effects of nine Follow Through models on measures of basic skills (word knowledge, spelling, language, and math computation), cognitive-conceptual skills (reading comprehension, math concepts, and math problem solving) and self-concept. This figure is adapted from Engelmann, S. and Carnine, D. (1982), Theory of Instruction: Principles and applications. New York: Irvington Press.

21. School Adopting School-wide Positive Behavior Support

22. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Insufficient Methods Excellent experimental evidence for what does not work Diffusion/dissemination of information by itself does not lead to successful implementation (research literature, mailings, promulgation of practice guidelines) Training alone, no matter how well done, does not lead to successful implementation

23. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Insufficient Methods Excellent experimental evidence for what does not work Implementation by laws/ compliance by itself does not work Implementation by “following the money” by itself does not work Implementation without changing supporting roles and functions does not work Paul Nutt (2002). Why Decisions Fail

24. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Sobering Observation "All organizations [and systems] are designed, intentionally or unwittingly, to achieve precisely the results they get." R. Spencer Darling Business Expert Rise in Incidence of Autism Reduction in Incidence of Mental Retardation and Learning Disabilities The Oregon Department of Education has released graduation rates for all public high schools. Nearly one-third of all high school students don't receive a diploma after four years of study. by Betsy Hammond, The Oregonian Monday June 29, 2009,

25. Sobering Observations Innovative practices do not fare well in old organizational structures and systems Organizational and system changes are essential to successful implementation ◦ Expect it ◦ Plan for it © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008

26. Build on local strengths, context and accomplishments Implement 1-3 evidence-based practices at a sustainable scale of social importance. Document value of implementation to achieve valued student outcomes © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Benefits to focusing on Scaling Implementation

27. Develop capacity of your State to implement any evidence-based practice with fidelity and good outcomes De-silo implementation of evidence- based practices, innovations, legislative mandates in the State © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008

28. Scale Up To scale up interventions we must first scale up implementation capacity Building implementation capacity is essential to maximizing the use of EBPs and other innovations Large scale, real time change © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008

29. Academic SystemsBehavioral Systems 1-5% 5-10% Intensive, Individual Interventions Individual Students Assessment-based High Intensity Of longer duration Intensive, Individual Interventions Individual Students Assessment-based Intense, durable procedures Targeted Group Interventions Some students (at-risk) High efficiency Rapid response Targeted Group Interventions Some students (at-risk) High efficiency Rapid response 80-90% Universal Interventions All students Preventive, proactive Universal Interventions All settings, all students Preventive, proactive Multi-tier Model Attention, Effort, Precision

30. States Implementing SWPBS 9000+ schools in 44 states States Number of Schools Maryland Illinois

31. Hawaii Scott Spaulding, Claudia Vincent Pbis.org/evaluation/evaluation briefs Illinois

32. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Capacity Development State Management Team State Transformation Team Regional Implementation Teams

33. State Management Group State Transformation Team Regional Implementation Team N = 50 – 200 Schools Regional Implementation Team N = 50 – 200 Schools Regional Implementation Team N = 50 – 200 Schools Regional Implementation Team N = 50 – 200 Schools IMPLEMENTATION CAPACITY FOR SCALING UP EBPs SISEP Support & 2 FTE

34. State Management Group First Regional Implementation Team N = 9 Staff Policy Enabled Practice (PEP) Practice Informed Policy (PIP) System Change SISEP Support & 2 FTE

35. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Transformation Zone State Department Districts Schools Teachers/ Staff Effective Practices ALIGNMENT Federal Departments SISEP/ Implementation Teams FORM SUPPORTS FUNCTION

36. Context Many initiatives to improve education Initiatives too often conflicting and/or competing Early Intervention Literacy Math Wraparound Positive Behavior Support Family Support Response to Intervention Applied Behavior Analysis

37. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Using RTI to Align Systems Literacy Wraparound Math Family Support Behavior Support ALIGNMENT Early Intervention Response to Intervention/Prevention Student Outcomes Primary Prevention Universal Screening Multi-tiered Support Early Intervention Progress Monitoring Systems to support practices

38. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Implementation Team A group that knows the innovation very well (formal and practice knowledge) A group that knows how to implement that innovation with fidelity and good effect A group that accumulates data & experiential knowledge -- more effective and efficient over time (information economics, K. Arrow)

39. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Implementation Team Prepare Communities Prepare schools faculty, staff Work with Researchers Assure Implementation Prepare Districts Assure Student Benefits

40. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Capacity Development Start with too many overqualified people “Generation 1” RIT members become: ◦ Generation 2 expanded RIT members ◦ Generation 3 State Transformation Team leaders ◦ Generation 4 State and federal officials

41. State Management Group State Transformation Team 2 FTE & Others Regional Implementation Team N = 50 – 200 Schools Regional Implementation Team N = 50 – 200 Schools Regional Implementation Team N = 50 – 200 Schools Regional Implementation Team N = 50 – 200 Schools IMPLEMENTATION CAPACITY FOR SCALING UP EBPs [Phase 2] SISEP Support

42. Intensive Development Saturation Intensive Development Saturation 44/51

43. Coaching Defined Coaching is the active and iterative delivery of: ◦ (a) prompts that increase successful behavior, and ◦ (b) corrections that decrease unsuccessful behavior. ◦ Coaching is done by someone with credibility and experience with the target skill(s) ◦ Coaching is done on-site, in real time ◦ Coaching is done after initial training ◦ Coaching is done repeatedly (e.g. monthly) ◦ Coaching intensity is adjusted to need

44. Outcomes of Coaching Fluency with trained skills Adaptation of trained concepts/skills to local contexts and challenges  And new challenges that arise Rapid redirection from miss-applications Increased fidelity of overall implementation Improved sustainability  Most often due to ability to increase coaching intensity at critical points in time.

45. Training Outcomes Related to Training Components Training Outcomes Training Components Knowledge of Content Skill Implementation Classroom Application Presentation/ Lecture Plus Demonstration Plus Practice Plus Coaching/ Admin Support Data Feedback 10% 5% 0% 30% 20% 0% 60% 60% 5% 95% 95% 95% Joyce & Showers, 2002

46. Coaching within SWPBS Implementation Context: ◦ 9600 schools implementing SWPBS nationally Defining the Role  Internal vs External Selecting Coaches Training and support for coaches Assessing Impact

47. Rehearsal and Coaching After initial training, a majority of participants (211 of 213) demonstrated poor implementation. Decision-makers should pair training prior to implementation with on-going rehearsal and performance feedback (coaching) ◦ Test et al 2008 © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008

48. Example of the Impact of Coaching on Student Outcomes: Average Major Discipline Referrals per Day per Month Coach returns from leave

49. Leadership Team Funding Visibility Political Support TrainingCoachingEvaluation Local Demonstration Schools Active Coordination Behavioral Expertise

50. “ Even if you’re on the right track, you’ll get run over if you just sit there” - Will Rogers