1. Implementing Evidence-based Practices at Scales of Social Significance Rob Horner University of Oregon www.pbis.org

2. Goals Considerations for the emergence of “evidence-based practices” Six features for taking EBP technology to scale Define “implementation” as a unique technology. Use School-wide Positive Behavior Support as one example

3. Considerations in the Emergence of Evidence-based Practices Defining “practices” Defining “evidence-based” Why is it useful to define “Evidence-based Practices?”

4. Considerations in the Emergence of Evidence-based Practices Defining a “practice” ◦ A “practice” is a procedure, or set of procedures,  designed for use in a specific context,  by individuals with certain skills/features,  to produce specific outcomes  for specific individuals. ◦ A practice can be at any of an array of sizes:  Individual technique  Intervention package  Intervention program

5. Are these “practices?” For whom, in what context, to what outcome? Phonics-based early reading instruction Whole Language Reading Instruction Positive reinforcement Inclusion NCLB Discrete trial training Generalization Positive behavior support Functional analysis Applied behavior analysis

6. Why should we be invested in the emergence of Evidence-based Practices? Researchers should continually debate and discuss the definition and details. Implementers need to be able to identify practices that are validated. Take great care in distinguishing between ◦ Documentation that a practice is evidence-based, and ◦ Documentation that one practice is better than some other practice.

7. Taking Evidence-based Practices to Scales of Social Significance The value of a science of human behavior ◦ Basic principles that help us describe, interpret and establish effective patterns of behavior While behavior analysis is among the most powerful approaches for achieving social change, too often ABA is viewed as relevant only within a narrow range of applications  (we are a niche or boutique technology)

8. Key Features to Achieve Large-scale Implementation of Evidence-based Practices 1. Focus on comprehensive outcomes defined by the values of the social system 2. Expand the unit of analysis 3. Measure process as well as outcome 4. Use ABA principles to build effective and accessible practices 5. Establish a technology for implementation 6. Define practices for scaling up practices that are evidence- based.

9. What is School-wide Positive Behavior Support? School-wide PBS is:  A systems approach for establishing the social culture and individualized behavioral supports needed for schools to achieve both social and academic success while preventing problem behavior Evidence-based features of SW-PBS  Prevention  Define and teach positive social expectations  Acknowledge (reward) positive behavior  Arrange consistent consequences for problem behavior  On-going collection and use of data for decision-making  Continuum of intensive, individual interventions.  Administrative leadership – Team-based implementation (Systems that support effective practices)

10. School-wide Positive Behavior Support: Current Implementation School-wide Positive Behavior Support 7500 schools in 44 states  Team  Coach  Curriculum emphasizing prevention: Define and teach appropriate social behavior to all students  Formal system for rewarding appropriate behavior  Intensive, individual interventions based on behavioral function  On-going data collection and use of data for active decision-making

11. 7500 Schools across 44 states implementing school-wide positive behavior support

12. Primary Prevention: School-/Classroom- Wide Systems for All Students, Staff, & Settings Secondary Prevention: Specialized Group Systems for Students with At-Risk Behavior Tertiary Prevention: Specialized Individualized Systems for Students with High-Risk Behavior ~80% of Students ~15% ~5% SCHOOL-WIDE POSITIVE BEHAVIOR SUPPORT

13. SWIS summary 07-08 (Majors Only) 2,532 schools; 1,300,140 students; 1,139,119 ODRs Grade RangeNumber of Schools Mean Enrollment per school Mean ODRs per 100 per school day K-61762444.35 (sd=.46) (1 /300 / day) 6-9482653.92 (sd=1.42) (1/ 110 / day) 9-121769141.06 (sd=.1.57) (1/105 / day) K-(8-12)3124011.00 (sd=.1.86) (1/ 155 / day

14. N = 1679 443 163 246 Elementary Middle High K (8-12) 10%

15. N = 1679 443 163 246 Elementary Middle High K (8-12)

16. School-wide PBS 1. Focus on comprehensive valued outcomes  Social competence  Academic achievement  Safety 2. Expand the unit of analysis ◦ Whole school  Classroom  Groups of “at-risk” students  Individual students needing intensive support

17. School-wide PBS 3. Measurement/ Evaluation ◦ Include both process and outcome measures  Outcomes: Office Discipline Referrals/Academics  Process (implementation): Team Checklist  Research: System-wide Evaluation Tool (SET) SWIS TIC data SET data Comprehensive Measures Progress Monitoring Measures Fidelity Measures Outcomes Measures XX (ABA) Comprehensive Measures Progress Monitoring Measures Fidelity Measures XX Outcomes Measures XXXX (ABA)

21. School-wide PBS 4. Use ABA principles to Establish Accessible Evidence- based Practices ◦ Use the language of the implementation context ◦ Combine technologies needed to achieve valued outcomes.  ABA + Person-centered planning + Organizational Systems + Bio-Medical ◦ Collaborate with other disciplines  Mental Health, Juvenile Justice, School Psychology, Sociology ◦ Use Single-case Designs to Document Evidence-based Practices

22. Academic, Behavioral, and Functional Predictors of Chronic Problem Behavior in Elementary Grades Kent McIntosh University of Oregon 40

23. Non-target Students

26. School-wide PBS 4. Use ABA principles to Establish Accessible Evidence- based Practices ◦ Use the language of the implementation context ◦ Combine technologies needed to achieve valued outcomes.  ABA + Person-centered planning + Organizational Systems + Bio-Medical ◦ Collaborate with other disciplines  Mental Health, Juvenile Justice, School Psychology, Sociology ◦ Use Single-case Designs to Document Evidence-based Practices

27. Embedding Bully-Proofing in School-wide PBS Scott Ross Rob Horner University of Oregon www.pbis.org

28. A Functional Approach to Bully Prevention: Remove the social reinforcers Define, teach and reward school-wide behavior expectations. Teach all children to identify and label inappropriate behavior.  Not respectful, not responsible, not safe Teach all students a “stop signal” to give when they experience problem behavior.  What to do if you experience problem behavior (victim, recipient)  What to do if you see someone else in a problem situation (bystander) Teach all students what to do if someone delivers the “stop signal”

29. 29 BaselineAcquisitionFull BP-PBS Implementation Number of Incidents of Bullying Behavior School Days School 1 Rob Bruce Cindy Scott Anne Ken School 2 School 3 3.14 1.88.88 72%

30. BP-PBS, Scott Ross30 28% increase 19% decrease

31. BP-PBS, Scott Ross31 21% increase 22% decrease

32. School-wide PBS 4. Use ABA principles to Establish Accessible Evidence- based Practices ◦ Use the language of the implementation context ◦ Combine technologies needed to achieve valued outcomes.  ABA + Person-centered planning + Organizational Systems + Bio-Medical ◦ Collaborate with other disciplines  Mental Health, Juvenile Justice, School Psychology, Sociology ◦ Use Single-case Designs to Document Evidence-based Practices

33. Use single-case research to document evidence-based practices. Define protocol for measuring “effect size” Define professional standards for defining a practice as “evidence-based” using single case research.

34. School-wide PBS 5. Build a functional technology of Implementation ◦ Define conditions for implementation ◦ Define conditions for high fidelity  Coaching, Policies, Administrative Contingencies ◦ Establish implementation with low cost ◦ Establish procedures for sustainability and continuous regeneration right from the beginning.

35. The Role of Behavior Specialists in the Development of Function-based Behavior Support Plans Leah Benazzi University of Oregon

36. Three knowledge areas needed for a behavior support plan Behavior Support Plan Knowledge about The Student Knowledge about The Setting Knowledge about Behavioral Theory

37. Design Three conditions in which behavior support plan recommendations were built from simulated cases (descriptive information, functional assessment information)  5 counterbalanced simulations ◦ Team alone12 plans ◦ Specialist alone12 plans ◦ Team with Specialist12 plans

38. Measures  Technical Adequacy: Are elements of behavior support plan consistent with functional assessment hypothesis? (1- 6) (range 3-18) ◦ ABA Experts (published studies employing functional analysis)  Contextual Fit: Are elements of plan consistent with values, skills, resources, administrative support? ◦ 16 questions (8 factors): ◦ Scored on 1-6 scale: Total (16-96)  Team member Ranking of Plans based on preference for implementation (1,2,3)

39. Results: Technical Adequacy Mean Expert Rating (6-18) * Team alone plans were statistically different from plans that included behavior specialist. *Team + Specialist and Specialist Alone were not statistically significantly different.

40. Table 3 Repeated Measures Analysis of Variance Summary Table for the Effects of Plan Developer on Technical Adequacy Scores SourcedfSSMSF Plan Developer Beh Spec Invol Unpredicted 211211 285.88 279.27 6.62 142.94 279.27 6.62 32.89* 64.26* 1.52 Team1145.514.14 Error2295.614.35 Total35427.00 *p <.01.

41. Post-hoc analysis of Technical Adequacy BehaviorTeamTeam + FSpecialistAloneSpecialist Problem Behavior3.380.950.680.78 Antecedents identified4.401.831 Identified Function14.14**10.71 Prevention Strategies16.2**0.980.580.98 Teaching Strategies5.510.920.680.92 Extinction Strategies55.3**0.930.280.84 Positive Reinforcement5.570.940.780.98 Person Responsible134.16**0.930.110.83 Assess Fidelity1.50.0300.05 Assess Impact163.83**0.930.130.84 ** p< Bonferroni family-wise alpha.05

42. Results: Contextual Fit Mean Team Rating (0-100) * Specialist Alone plans were statistically different from plans that included team members. * Team Alone and Team + Specialist plans were not statistically significantly different

43. Post-hoc Analysis of Contextual Fit Contextual Fit DomainFBehaviorTeamTeam + SpecialistAloneSpecialist Knowledge of BSP Elements15.99**5.085.765.67 Skills to perform BSP9.725.355.625.56 Values consistent with BSP52.62**4.695.855.76 Resources to implement2.594.624.894.9 Administrative Support10.684.935.235.32 BSP expected to be effective29.78**4.295.255.4 BSP in best interest of student30.21**4.785.745.77 BSP Efficient to Implement13.10**4.325.044.97 ** p< Bonferroni family-wise alpha.05

44. Implementation Technology Role of Coaching Importance of Policy

45. Training Outcomes Related to Training Components Training Outcomes Training Components Knowledge of Content Skill ImplementationClassroom Application Presentation/ Lecture Plus Demonstration Plus Practice Plus Coaching/ Admin Support Data Feedback 10% 5% 0% 30% 20% 0% 60% 60% 5% 95% 95% 95% Joyce & Showers, 2002

46. Discipline Foundation Policy: LAUSD School-Wide Positive Behavior Support NUMBER: BUL-3638.0 ISSUER: Donnalyn Jaque-Antón, Executive Officer, Educational Services DATE: March 27, 2007 POLICY: Every student, pre-school through adult, has the right to be educated in a safe, respectful and welcoming environment. Every educator has the right to teach in an atmosphere free from disruption and obstacles that impede learning. This will be achieved through the adoption and implementation of a consistent school-wide positive behavior support and discipline plan for every school in LAUSD. All school level discipline plans will be consistent with the Culture of Discipline: Guiding Principles for the School Community (Attachment A) and Culture of Discipline: Student Expectations (Attachment B). This will include: teaching school rules and social-emotional skills; reinforcing appropriate student behavior; using effective classroom management and positive behavior support strategies by providing early intervention for misconduct and appropriate use of consequences.

47. School-wide PBS 6. Define practices for scaling up  Efficacy  Effectiveness  Dean Fixsen and Karen Blase Scale Model Fixsen et al

48. Leadership Team Funding Visibility Political Support TrainingCoachingEvaluation Local Demonstration Schools Active Coordination Behavioral Expertise

49. School-wide PBS Document effects in multiple formats ◦ Single case, Descriptive, RCT, etc. Documentation via randomized control-group Design ◦ Provide research outcomes that address multiple audiences  Families  Administrators  Teachers  Scientist from all disciplines

50. The Effects of School-wide PBS within a Randomized Control Effectiveness Trial The Effects of School-wide PBS within a Randomized Control Effectiveness Trial Rob Horner, George Sugai, Keith Smolkowski, Lucille Eber, Jean Nakasato, Anne Todd, Jody Esperansa OSEP TA Center on Positive Behavior Support www.pbis.org In press in the Journal of Positive Behavior Intervention

51. Research Questions Can SWPBS be implemented to criterion by typical state trainers? If SWPBS is implemented are schools perceived as safer settings? If SWPBS is implemented do students benefit academically?

52. Method Randomized Control Trial ◦ 30 Elementary Schools in Illinois, and 30 Elementary Schools in Hawaii ◦ Random assignment of schools to  (Initial SWPBS training; and Delayed SWPBS training) ◦ Replacement (7 schools) randomly assigned Data collected across three years ◦ Time 1: No SWPBS training for any schools ◦ Time 2: Initial Treatment Schools get training ◦ Time 3: Delay Schools get training (problems…)

53. Measures Office Discipline Referrals (ODR) Implementation of SWPBS ◦ School-wide Evaluation Tool (SET)  Sugai et al. Perceived School Safety ◦ School Safety Survey (SSS)  Sprague, Colvin & Irvin Academic Success ◦ Proportion of Students Meeting State Reading Standards (SAT – 9 in Hawaii; ISAT in Illinois)

54. Randomized Waitlist Controlled Trial Assessment Time Period GroupT 1T 2T 3 Treatment(N = 30)OXOO Control/Delay (N = 30)OOXO (T = time (by year), O = observation, X = implementation of SWPBS training)

55. Finding #1: Implementation by regular personnel Mean SET score (Total) [ Bold indicates post intervention ] T1T2T3 Treatment Group.381 N = 33.785 N = 30.823 N = 30 Control/Delay Group.388 N= 26.459 N= 27.640 N= 23 Random coefficients analysis (Murray, 1998; Singer & Willett, 2003): Time X Condition p <. 0001; r =.67; d = 1.78 Random coefficients analysis (Murray, 1998; Singer & Willett, 2003): Time X Condition p <. 0001; r =.67; d = 1.78

56. Results: With training by regular state trainers, schools are able to implement SWPBS to criterion. Random coefficients analysis: p <.0001; d = 1.78 Initial Training Delay Training **

57. Finding #2: SWPBS is associated with increased perception of safety: School Safety Survey: Risk Factor T1T2T3 Treatment Group.370 N= 24.344 N = 29.343 N = 25 Control/Delay Group.387 N = 19.415 N= 24.358 N = 20 Time X Condition p =.0154 r = -.40 d = -.86

58. Results: Perceived Social Risk Factors decreased when SWPBS was implemented with fidelity. Random coefficients analysis p =.0154; d = -.86 * *

59. Finding #3: SWPBS associated with increase in proportion of students meeting state reading standard T1T2T3 Treatment Group.455 N = 33.529 N= 31.536 N = 31 Control/Delay Group.38 N = 28.402 N= 27.436 N= 23 T2 Treatment vs. Control: p =.032 r =.28 d =.58

60. Results: The percentage of 3 rd graders meeting the state reading standard increased with SWPBS implementation N.S. p =.032; d =.58 **

61. Summary ◦ Never stop development of the rigorous, precise science of human behavior. ◦ Expand the unit of analysis to address socially relevant outcomes  Address the full set of outcomes defined as important for a context/ community ◦ Expand the research methods/questions to address socially important concerns.  Sustainability  Scalability

62. Summary Invest in a technology of “implementation” that will focus on taking evidence-based practices to scale.