“Sustaining & Expanding Effective Practices: Lessons Learned from Implementation of School-wide Positive Behavior Supports” Susan Barrett, Cyndi Boezio, Rob Horner, George Sugai Shepperd-Pratt Health Systems MD, Colorado Department of Education, University of Oregon, University of Connecticut

Goals  Themes of note  National implementation data  Lessons learned for states planning to implement educational innovations at scale.

Themes of Note  Implementation to criterion  Measure fidelity as an implementation activity Fixsen et al., 2005  Measure Organizational Capacity  Readiness  Sustainability  School Capacity– District Capacity– State Capacity  Collect and use data for decision-making  Emphasis on value of iterative self-assessment  Integration of initiatives  Mental Health, Behavior Support, Academics

Lessons Learned from PBS  National data 4772 schools currently working with the TA Center on implementation of School-wide PBS  State Leadership Teams  State Trainers, Cadre of Coaches  District Leadership Teams  School Implementation Teams Agreements Team Data-based Action Plan ImplementationEvaluation

Major State Implementation Efforts Maryland 494 schools Alabama 219 schools Illinois 611 schoolsColorado 405 schools Florida 250 schoolsNew York 322 schools Michigan 181 schoolsOhio 221 schools New Mexico 130 schoolsWest Virginia 215 schools Oregon 229 schoolsLouisiana 285 schools Missouri 183 schoolsGeorgia 171 schools

SWIS summary (Majors Only) 1668 schools: 838,184 students swis.org Grade Range Number of Schools Number of Students Mean ODR /100 / day K ,932 Mean = (sd=.50) ,129 Mean = (sd=1.06) ,325 Mean = (sd=1.37) K-(8-12)23990,198 Mean = (sd=1.56)

Randomized Controlled Trial (Preliminary Findings)  Assessment Time Period  GroupT 1T 2T 3  Treatment(N = 30)OXOO  Control/Delay (N = 30)OOXO  (T = time (by year), O = observation, X = implementation of SWPBS training)

Finding #1: Implementation by regular personnel Mean SET score (Total) [ Bold indicates post intervention ] T1T2T3 Treatment Group.381 N = N = N = 30 Control/Delay Group.388 N= N= N= 23 Random coefficients analysis (Murray, 1998; Singer & Willett, 2003): Time X Condition p <. 0001; r =.67; d = 1.78 Random coefficients analysis (Murray, 1998; Singer & Willett, 2003): Time X Condition p <. 0001; r =.67; d = 1.78

Finding #2: SWPBS is associated with increased perception of safety: School Safety Survey: Risk Factor T1T2T3 Treatment Group.370 N= N = N = 25 Control/Delay Group.387 N = N= N = 20 Time X Condition p =.0154 r = -.40 d = -.86

Finding #3: SWPBS associated with increase in proportion of students meeting state reading standard T1T2T3 Treatment Group.455 N = N= N = 31 Control/Delay Group.38 N = N= N= 23 T2 Treatment vs. Control: p =.032 r =.28 d =.58

Sustainability  Eber et al., (Illinois)

High fidelity implementation associated with higher yr 1 & 2 sustainability

Sustainability  Jennifer Doolittle, Ph.D. 285 schools implementing SWPBS Post-hoc analysis using SET scores  Schools that met and sustained SWPBS criterion  N = 140  Schools that met but did NOT sustain SWPBS  N = 74  Schools that had not yet met SWPBS criterion  N = 71

Doolittle (2006) SET SubscaleSustainability StatusPair-wise Comparison Effect Sizes1 Non-Imp (N = 71) Non-Main (N = 74) Main (N =140) MSDM M F-valueNon-Imp vs. Non-Main Non-Imp vs. Main Non-Main vs. Main Expectations defined0.58a b c *** Behavioral expectations taught *** On-going behavioral reward system *** System responding to behavioral violations *** Monitoring and decision making *** Management *** District-level support ** *p <.05; **p <.01; ***p <.001 1Effect sized used is d-statistic and interpreted as.2 = small effect,.5 = medium effect,.8 = large effect (Cohen, 1988). Notes. SET = School-wide Evaluation Tool; M = mean; SD = standard deviation. Means in the same row with different subscripts significantly differ at p <.05.

Summary  States proposing to invest in large scale educational reforms 1. Require Efficacy Evidence documenting impact 2. Require Effectiveness Evidence documenting practical application 3. Require Sustainability Evidence that practices sustain over time.

Sustainability  Organizational capacity (readiness)  Implementation to criterion (mastery)  Not dosage  Implementation of sustainability systems as well as daily practices  Regeneration cycle  Collection and use of data  Iterative system of re-training; re-introduction Capacity Building  Formal decision-making structure

Valued Outcomes Practice Implementation Identifying & Modifying Practices Efficiency Efficacy, and Effectiveness Priority Continuous Regeneration Continuous Measurement Data- Based Prob. Solving Capacity Building

Valued Outcomes Practice Implementation Identifying & Modifying Practices Efficiency Efficacy & Effectiveness Priority Continuous Regeneration Continuous Measurement Data- Based Prob. Solving Capacity Building

Core Outcomes Organizational Capacity (Readiness) Effective and Practical Practices Implementation Process Cycle of Continuous Regeneration * Iterative data collection * Decision-making * Adaptation School Context McIntosh, Horner & Sugai Sustainability Systems Planning for Large Scale Implementation

Summary  Scaling up implementation of effective practices is the next major challenge facing education. Current approaches for scaling up are not effective  The investment in research will have limited impact without a formal structure for moving evidence-based practices to scale.  Sustainability will be one of the key variables affecting successful scaling up.  States investing in large scale educational reform should require documentation of (a) efficacy, (b) effectiveness and (c) sustainability.

Questions/Comments