1. Promises and Pitfalls in School-wide Positive Behavioral Interventions and Supports (PBIS)
Catherine Bradshaw, Ph.D., M.Ed.
Johns Hopkins Center for the Prevention of Youth Violence (CDC)
Johns Hopkins Center for Prevention and Early Intervention (NIMH)
Department of Mental Health
Penn State IES Fellows: May 17, 2011

2. PBIS Model: Whole-school Prevention
Application of behavioral, social learning, & organizational behavioral principles
Clear behavioral expectations (e.g., ready, responsible, and respectful)
Positive rewards
Procedures for managing disruptions
(Horner & Sugai, 2001; Lewis & Sugai, 1999; Sugai & Horner, 2006)

3. MacArthur Middle School
MacArthur Middle School
Eagle Code
Be Responsible
Respect Yourself
Respect Others

5. Eagle Tickets

6. PBIS Model: Whole-school Prevention
Application of behavioral, social learning, & organizational behavioral principles
Clear behavioral expectations (e.g., ready, responsible, and respectful)
Positive rewards
Procedures for managing disruptions
Focus on changing adult behavior
Emphasizes staff buy-in
Team-based & data-based process
Consistency in discipline practices
(Horner & Sugai, 2001; Lewis & Sugai, 1999; Sugai & Horner, 2006)

7. Office vs. Classroom Managed

10. PBIS Model: Whole-school Prevention
Application of behavioral, social learning, & organizational behavioral principles
Clear behavioral expectations (e.g., ready, responsible, and respectful)
Positive rewards
Procedures for managing disruptions
Focus on changing adult behavior
Emphasizes staff buy-in
Team-based & data-based process
Consistency in discipline practices
Can be implemented in any school level, type, or setting
Non-curricular model – flexible to fit school culture & context
Coaching to ensure high fidelity implementation
On-going progress monitoring
Public health approach (universal / selective / indicated)
Requires a shift from punitive/reactive to preventive
(Horner & Sugai, 2001; Lewis & Sugai, 1999; Sugai & Horner, 2006)

11. Maryland’s Tiered Instructional and Positive Behavioral Interventions and Supports (PBIS) Framework
Academic Systems
Behavioral Systems
Intensive, Individually Designed Interventions
Address individual needs of student
Assessment-based
High Intensity
Intensive, Individually Designed Interventions
Strategies to address needs of individual students with intensive needs
Function-based assessments
Intense, durable strategies
1-5%
1-5%
Targeted, Group Interventions
Small, needs-based groups for
at risk students who do not respond
to universal strategies
High efficiency
Rapid response
5-10%
5-10%
Targeted, Group Interventions
Small, needs-based groups for at risk students who do not respond to universal strategies
High efficiency/ Rapid response
Function-based logic
Andrea—
As you all know, this is the public health framework for behavior and instructional supports in Maryland through PBIS and RtI..
Core Curriculum and
Differentiated Instruction
All students
Preventive, proactive
School-wide or classroom
systems for ALL students
80-90%
Core Curriculum and
Universal Interventions
All settings, all students
Preventive, proactive
School-wide or classroom systems for ALL students and staff
80-90%
(MSDE, 2008) 11 11 11

12. Maryland’s PBIS Organizational Model
School Level
826 PBIS Teams (one per school) ≈ 59%
Team leaders (one per school)
Behavior Support Coaches (560)
District Level (24)
District Coordinators
State Level
State Leadership Team
Maryland State Department of Education (MSDE)
Sheppard Pratt Health System
Johns Hopkins University
24 Local school districts
Department of Juvenile Services, Mental Hygiene Administration
University of Maryland
Management Team
Advisory Group
National Level
National PBIS Technical Assistance Center
University of Oregon, University of Connecticut, & University of Missouri
State
District
School
Classroom
Student
(Barrett, Bradshaw & Lewis-Palmer, 2008; JPBI)

13. Group Randomized Trial of SW-PBIS
Funding
Centers for Disease Control & Prevention
National Institute of Mental Health
Institute of Education Sciences
Sample
37 voluntary elementary schools across 5 school districts
Enrollment ; 60% Caucasian; 48% suburban; 41% urban fringe; 49% Title I
Design
Group randomized effectiveness trial
21 PBIS & 16 “Focus/Comparison”
Baseline plus 4 years (spring spring 2007)
Data from 29,423 students & 3,563 staff
29,423 students and 3,563 staff
Q: % title I, 48% Suburban, 41% Urban, 11% Rural
Project Target
(Bradshaw et al., Prevention Science, 2009; School Psychology Quarterly, 2008; JPBI, 2010)

14. SET: PBIS Implementation Fidelity
The SET implementation fidelity data indicate the state’s procedures for training are effective. The schools trained in PBS received significantly higher scores on the SET at all study years. Prior research by the program developers indicate that a score of 80% or higher is associated with reductions in office disciplinary referrals and enhanced academic performance. Most of the trained schools received an 80% or higher 1 year following training. All by 1 of the 21 trained schools had made an 80% two years after receiving training.
Notes. No significant differences between groups at baseline, but differences at all other years at p<.05. Overall SET score: Wilks’ Λ = .38, F (4,32) = 13.36, p <.001, partial η2 = .63, d = (Bradshaw, Reinke et al., ETC, 2007)

15. Brief Summary of SW-PBIS Training Effects on Fidelity
High fidelity implementation and sustainability of PBIS
All trained schools reached high fidelity within 4 years (66% by the end of year 1)
All trained schools sustained high fidelity
(Bradshaw, Reinke et al., ETC, 2007; Bradshaw et al. Prevention Science, 2009)

16. Analysis of Climate Data: Organizational Health Inventory (OHI)
OHI: 37 item staff-report measure of 5 aspects of a healthy functioning school (Hoy et al., 1991)
academic emphasis - students are cooperative in the classroom, respectful of other students who get good grades, and are driven to improve their skills
staff affiliation - warm and friendly interactions, commitment, trust
collegial leadership - principal’s behavior is friendly, supportive, open
resource influence - principal’s ability to lobby for resources for the school
institutional integrity - teachers are protected from unreasonable community and parental demands
overall OHI score (average of all items)
Analyses
Longitudinal analyses were conducted using a 3-level approach in Mplus (Muthén & Muthén, 2006)
Intent to treat approach (Lachin, 2000) & moderated by fidelity (SET)
Adjusted for staff (sex, race, age) and school (FARMs, student mobility, faculty turnover, & school enrollment) covariates on intercept and slope

17. Effect of PBIS on Overall OHI*
Sig. change (.05) \
Note. Adjusted means from 3-level model. * Intervention effect on slope of overall OHI significant at p<.05.

18. Effect of PBIS on Collegial Leadership*
Sig. difference (.05) * \
Note. Adjusted means from 3-level model. * Intervention effect on slope of overall OHI significant at p<.05.
Note. Adjusted means from 3-level model. * Intervention effect on slope significant at p<.05.

19. Brief Summary of SW-PBIS Training Effects on Climate
PBIS training associated with significant improvements in staff members’ report of school climate / organizational health
Principal leadership, collegial relationships, academic emphasis, recourse influence, institutional integrity, and overall OHI
Effect sizes ranged from .24 (AE) to .35 (RI)
OHI intercept and slope negatively correlated
Schools starting with lower levels of OHI tended to take longer to reach high fidelity, but improved the most
(Bradshaw, et al., SPQ, 2008; Bradshaw et al. Prevention Science, 2009)

20. Brief Summary of Impacts of SW-PBIS on Student Outcomes
Significant impacts for students:
Significant reduction in school-level suspensions among the PBIS schools
Students in PBIS schools were 32% less likely to receive an office discipline referral
A positive trend for school-level MSA academic performance was observed
(Bradshaw et al., JPBI, 2010)

21. Impact of SW-PBIS on Bullying and Rejection: HLM 3-Level Results
Teacher-Reported Bullying
Teacher-Reported Rejection
Coefficient SE
Intercept
1.4029***
0.0242
1.8174***
0.0359
Mobility
0.0001
0.0028
0.0021
0.0032
Student Teacher Ratio
0.0053
0.0088
Faculty Turnover
0.0034
0.0018
Enrollment
0.0089
0.0115
0.0309*
0.0141
Special Education Status
0.1176***
0.0268
0.3646***
0.0367
Ethnicity (Black)
0.2317***
0.0316
0.1545***
0.0288
Grade Cohort
0.0473 *
0.0481
FARMS
0.0846***
0.0165
0.2347***
0.0241
Gender
0.2261***
0.0183
0.2127***
0.0176
Slope (Growth)
0.0326***
0.0099
0.0767***
0.0158
SWPBIS Intervention * *
0.0145
0.0015
0.0009
0.0011
0.0043*
0.0016
0.0042
0.0022
0.0023
0.0013
0.0028* *
0.0037 *
0.0048
0.0187
0.0121
0.0333***
0.0059
0.0078
0.0127
0.0161
0.0181
0.0218***
0.0051
0.0079
Gender (male)
0.0188***
0.0064
0.0056
0.0077
Measures. At each of the five post-training data collections, teachers completed the Teacher-Observation of Classroom Adaptation-Checklist (TOCA-C; Koth, Bradshaw, & Leaf, 2009; Werthamer-Larsson et al., 1991). Teacher-reports of bullying was assessed through the Teacher-Observation of Classroom Adaptation-Checklist (TOCA-C; Koth, Bradshaw, & Leaf, 2009; Werthamer-Larsson et al., 1991) a Likert-scale of 1=Never to 6=Almost Always. Four items (teases classmates  , yells at others, harms others, and fights) were averaged (α=.87). Teacher-reports of rejection were assessed through three TOCA items (is rejected, does not have many friends, and is not liked by classmates; α=.84), which were averaged.
Note. Measure is Teacher Observation of Classroom Adjustment TOCA (teacher-reported) * p<.05, ** p <.01, *** p <.001

22. Impact of SW-PBIS on Teacher-Reported Perpetration of Bullying Behaviors
Four items (teases classmates  , yells at others, harms others, and fights) were averaged (α=.87).
(N = 12,334, γ = -.02, t = -2.60, p<.05, ES= -.11)

23. Impact of SW-PBIS on Teacher-Reported Student Rejection
Teacher-reports of rejection were assessed through three TOCA items (is rejected, does not have many friends, and is not liked by classmates; α=.84), which were averaged
(N = 12,334, γ = -.03, t = -2.32, p<.05, ES= -.14)

24. SW-PBIS Intervention Status by Grade Cohort on Rejection
Control Younger

25. Impact of SW-PBIS on Teacher-Reported Disruptive Behaviors
Results indicate the after adjusting for both individual and school level characteristics, there was a significant positive intervention effect on disruptive behaviors (γ = -.02, t = -2.39, p<.05, ES = -.12), with students in the intervention having lower levels of disruptive behaviors as compared to those in the control group.
(N = 12,334, γ = -.02, t = -2.39, p<.05, ES = -.12)

26. Impact of SW-PBIS on Teacher-Reported Concentration Problems
The effect of SWIPIS on concentration problems was marginally significant (γ = -.03, t = -2.08, p=.046).
(N = 12,334, γ = -.03, t = -2.08, p=.046, ES=.09)

27. Impact of SW-PBIS on Teacher-Reported Emotion Regulation Skills
The effect of SWIPIS on emotion regulation skills was significant (γ = .03, t = -2.30, p=.045, ES = .10), with students in the intervention having higher emotion regulation skills as compared to those in the control group.
(N = 12,334, γ = .03, t = -2.30, p=.045, ES = .10)

28. Impact of SW-PBIS on Teacher-Reported Positive Behaviors
Results indicate the after adjusting for both individual and school level characteristics, there was a significant intervention effect on positive behaviors (γ = .03, t = -2.11, p<.05, ES = .15), with students in the intervention having higher levels of positive behaviors as compared to those in the control group.
(N = 12,334, γ = .03, t = -2.11, p<.05, ES = .15)

29. Variations in the Impact of SW-PBIS
Is there variation in the impact of SW-PBIS based on the child’s baseline pattern of risk?
Is SW-PBIS more impactful for some students than others?
Parallel to ‘green’, ‘yellow’, and ‘red’ zone framework?
Used Latent Class Analysis (LCA) to examine variation
LCA is a person centered approach
Groups participants with similar patterns among indicator variables into latent classes (McCutcheon, 1987)
Analysis conducted in Mplus 6.1 (Muthén & Muthén, )
Fitting the classes
Substantive theory as well as statistical support (Nylund et al., 2007)
5 indices: Akaike Information Criterion (AIC), Baysian Information Criterion (BIC; Schwartz, 1978), Sample Size Adjusted Baysian Information Criterion (SSA BIC; Sclove, 1987), Lo-Mendell- Rubin adjusted likelihood ratio test (LMR; Lo, Mendell, & Rubin, 2001), and the sample size adjusted LMR (Muthén & Muthén, )

30. Latent Class Analyses: Mean Baseline TOCA Scores
Mean (Centered)
CATHERINE: I am not loving the “”Problem Behaviors” Group title maybe high risk since they are only 1st graders?
Baseline Teacher Ratings of Student Behavior (TOCA) N = 12,334

31. Variation in Impact by LCA Membership: % within Class Experiencing (Untoward) Outcome
High Risk
At Risk
Normative
Socially-Emotionally Skilled
TOCA item
PBIS
Control
Sent to Principals’ Office
71A
78 A
41 B
46 B 22 20 10 11
Rcvd Counseling for
Inappropriate Behavior 57 54
25 C
30 C 5
Social Skills Development 50 56
30D
36D 18 9E
13E
Grade Retained 8 7 3 4 2F 4F
Referred to Special Education 36 39
23G
27G 13 6
Note. Numbers represent percents experiencing that outcome.
Those sharing letters are significantly different at p<.05. N = 12,334

32. Conclusions of Main Effects
Several significant impacts of SW-PBIS on the school environment, staff, and students
Main effects on teacher reported adjustment (i.e., concentration problems, aggressive/disruptive behavior, bullying, rejection, emotion regulation, prosocial behavior)
Although students in both groups (PBIS and Comparison) tended to display higher rates of problem behaviors over time, students in PBIS schools faired better than those in comparison schools
Some indication that the intervention effects are strongest the earlier students are exposed to PBIS

33. Conclusions from Variations Results
Four classes (not 3) of risk patterns emerged
High risk (6.6%), At risk (23.3%), Normative (36.5%), Social-emotional Skilled (33.6%)
Those in the ‘high risk’ and the ‘at-risk’ faired the worst in the Control schools than in the SW-PBIS schools
Shows that main effects models may ‘wash out’ some significant program impacts
Consistent with an RtI framework, however even the higher risk students are doing ‘better’ in an SW-PBIS environment

34. Examining Contextual Factors Associated with Scale-up
Research Aims
To identify school- and district-level characteristics which predict initial training, adoption, and the quality with which PBIS is implemented.
Greater need at the school- and district-level would be associated with initial training, but lower odds of adoption or quality implementation.
Greater district infrastructure and wealth would be associated with greater odds of all three outcomes.
(Bradshaw & Pas, accepted pending revision)

35. Participating Schools
All 24 MD districts participate in the PBIS initiative.
825 traditional elementary schools across 22 districts in the state were eligible to be included (i.e., two districts had three or fewer schools trained) of which 312 were trained.
236 schools across 17 districts had been trained AND provided data, indicating active participation and could be analyzed for the implementation outcome.
A “traditional” elementary school is defined as a school that educates both regular and special education students, excludes alternative schools and those which just provide services to special education students, and serves kindergarten through grades five or six.
Further districts (i.e., from the 22 included in the training model) were excluded from the adoption and implementation analyses because they had two or fewer schools that submitted implementation data, and thus an insufficient number of schools with which to estimate the parameters with confidence.

36. Outcome Variables Training in school-wide PBIS
A school team of at least 4 individuals, including an administrator, attended the state’s two-day training event.
Adoption of school-wide PBIS
School was trained AND submitted implementation data in the spring of 2008
Implementation of school-wide PBIS
The Implementation Phases Inventory (IPI; Bradshaw, Debnam, Koth, & Leaf, 2009):implementation and maintenance scales and overall score.
IPI SCALES
Preparation (Cronbach’s alpha [α] =.65, e.g., “PBIS team has been established,” “School has a coach”)
Initiation (α = .80, e.g., “A strategy for collecting discipline data has been developed,” “New personnel have been oriented to PBIS”)
Implementation (α = .90, e.g., “Discipline data are summarized and reported to staff,” “PBIS team uses data to make suggestions regarding PBIS implementation”), Maintenance (α = .91, e.g., “A set of materials has been developed to sustain PBIS,” “Parents are involved in PBIS related activities”).
Indication of the extent to which that core feature was in place at the school on a 3-point scale from 0 (not in place) to 2 (fully in place).
Schools receive a percentage of implemented elements for each stage, such that a higher score indicates greater implementation.
A total score (α = .94) was calculated by averaging the percent implemented for each stage, weighting each scale equally.
IPI scores from the spring of 2008 ( school year) were used in the current study.
The preparation and initiation scales focus on pre-cursors to implementation, and thus there tends to be less variability in these scales, due to most schools achieving very high scores on these scales (Bradshaw, Debnam et al., 2009). As a result, we focused on the two latter phases of implementation (i.e., implementation and maintenance), along with the overall IPI score.

37. Conclusions
Schools with greater need were more likely to receive training, and in some cases also adopt SW-PBIS.
Maryland schools self-identify for training; lower-performing schools seem to seek SW-PBIS as a way to improve their school.
Indicators of school disorganization do not serve as an obstacle to successful implementation of SW-PBIS.
The number of years since training and percent of certified teachers is associated with better implementation.
District variables are related to training and adoption, but not implementation.
(Bradshaw & Pas, accepted pending revision)

38. Next Steps: Integrating PBIS with…
PATHS to PAX & PBIS (NIMH, IES; Ialongo & Bradshaw)
Integrate an evidence-based classroom management program (PAX/Good Behavior Game) and a social-emotional learning curricula (PATHS) with PBIS
Pilot work in 6 Baltimore City PBIS schools over 2 years
PBISplus Project (IES; Leaf & Bradshaw)
45 MD elementary schools all implementing school-wide PBIS
Provides support to SSTs and teachers related to selection of evidence-based practices
3 years of on-site support and ‘coaching’ provided through a PBISplus Liaison
Safe and Supportive Schools Grant (MDS3) (USDOE; MSDE)
Collect data on climate and student safety
Conduct a 60 high school randomized trial of the integration PBIS with evidence-based violence prevention, drug prevention, and truancy prevention programs
Double Check Cultural Proficiency & Student Engagement (IES; Bradshaw)
Develop school-wide and classroom –based management system s to reduce disproportionality in PBIS elementary and middle schools
Provide professional development, adapt the Classroom Check-up (Reinke, 2007), support data- based decision-making

39. Potential Challenges to the PBIS Model
What impact does the focus on a team-based adaptive process have on variation in implementation quality?
What are some strengths and limitations of the training model?
What is the effect on the classroom?
Is it a process or a program?
Controversial role of rewards
What is changing – students or environment?
Too behaviorally focused? What about internalizing or social- emotional factors?
Is it too adult focused/driven?
Cost benefit analysis – effect size vs. investment
What to do when the universal isn’t enough?
Will this work for urban schools or high schools?

40. Acknowledgements Maryland State Department of Education Milt McKenna
Andrea Alexander
Sheppard Pratt Health System
Susan Barrett
Jerry Bloom
Johns Hopkins
Phil Leaf
Katrina Debnam
Chrissy Koth
Mary Mitchell
Elise Pas
Contact Information
Catherine Bradshaw, Ph.D.
Johns Hopkins Center for the Prevention of Youth Violence
Johns Hopkins Bloomberg School of Public Health
PBIS Resources
Supported by NIMH (1R01MH A, P30 MH08643), CDC (1U49CE & K01CE ), IES (R324A07118, R305A090307, & R324A ) & USDOE