SCHOOL-WIDE POSITIVE BEHAVIORAL INTERVENTIONS AND SUPPORT: ADDRESSING THE BEHAVIOR OF ALL STUDENTS Benchmarks of Quality KENTUCKY CENTER FOR INSTRUCTIONAL DISCIPLINE

Participant Expectations Be Responsible  Return promptly from breaks  Be an active participant  Use the law of two feet Be Respectful  Maintain cell phone etiquette  Listen attentively to others  Limit sidebars and stay on topic Be Kind  Enter discussions with an open mind  Respond appropriately to others’ ideas

Training Objectives To become familiar with the Benchmarks of Quality (BoQ) and understand why it is important to use it To understand how to complete, submit, and use the BoQ

Why is Evaluation of PBIS Implementation Important? To examine the fidelity of implementation − “Are we really doing what we think we are doing?” To document effectiveness of implementation − “Is what we’re doing working?” To identify and examine strengths and weaknesses of implementation − Celebrate successes − Identify areas to improve

PBIS Fidelity Measures Team Implementation Checklist  Completed during October team meeting PBIS Self-Assessment Survey  Completed between March 1 and April 30 Benchmarks of Quality  Completed between February 1 and March 31

Team Implementation Checklist The Team Implementation Checklist (TIC) is a self-assessment completed by the school PBIS Leadership Team. It serves as a guide in appraising the status of PBIS start-up, team functioning, development of key components, and evaluation.

PBIS Self-Assessment Survey The PBIS Self-Assessment Survey (SAS), is used by school staff for initial and annual assessment of Positive Behavior Support systems in schools. The survey examines the status and need for improvement of four behavior support systems: (a) school-wide discipline systems, (b) non-classroom management systems (c) classroom management systems, and (d) systems for individual students. Each question in the survey relates to one of the four systems. Survey results are summarized and used for a variety of purposes including: annual action planning; internal decision-making; assessment of change over time; increasing awareness of staff; and team validation.

Benchmarks of Quality The Benchmarks of Quality (BoQ) is a research- validated measure that assesses the development and implementation of PBIS across 10 critical elements. The BoQ is completed annually by school PBIS Leadership Teams to assess strengths and identify areas of need. Results are used for action planning. Minimal implementation of universal systems is occurring when a school obtains a score of 70%.

Benchmarks of Quality Developed by Florida PBIS at the University of South Florida Assesses development and implementation of school- wide PBIS Completed by school teams annually to identify areas of strength and weakness Used by state and districts to guide TA and training and evaluate outcomes related to level of implementation

53 Items Aligned with PBIS Process Addressing 10 Critical Elements 1. PBIS team 2. Faculty commitment 3. Effective procedures for dealing with discipline 4. Data entry and analysis plan 5. Expectations and rules/procedures developed

53 Items Aligned with PBIS Process Addressing 10 Critical Elements 6. School-wide recognition system established 7. Lesson plans for teaching expectations and rules/procedures 8. Implementation plan 9. Classroom systems 10. Evaluation

Initial BoQ Validation Process Expert review Pilot studies in Florida and Maryland Reliability: Test-retest, inter-rater both >.01 Concurrent Validity – SET/ODR’s For more details see JPBI, Fall 2007

Completing the Benchmarks of Quality

Timeline January – training for coach/designee on the BoQ February to March – complete the BoQ assessment By March 31 – enter BoQ scores into school PBIS Assessment account

Two Methods for Completion Method 1: Coach uses scoring rubric and team members use separate rating form (original way) Method 2: Coach uses scoring rubric and team members also use scoring rubric (modified version)

Method 1 Steps 1. Coach completes rating using the Scoring Form and Scoring Rubric 2. Team members complete ratings using the Team Member Rating Form 3. Coach tallies most common responses from team and marks on Scoring Form 4. Coach determines areas of discrepancy and marks on Scoring Form 5. Team meets and resolves any discrepancies; determines final score 6. Findings are discussed with team; action plan items are generated

Pros Cons Less time for team members to complete ratings Allows for deeper discussion among team members Different scoring scale for coach vs. team members Determining discrepancies can be confusing Reviewing discrepancies can be time consuming Method 1

Method 2 Steps 1. Coach completes rating using the Scoring Form and Scoring Rubric 2. Team members complete rating using the Scoring Form and Scoring Rubric 3. Coach tallies most common responses from team and marks on Scoring Form 4. Findings are discussed with team; action plan items are generated

Pros Cons More streamlined, less confusing scoring method Everyone uses the same scoring form Eliminates the issue of discrepancies More time for team members to complete rating May result in less intense discussion by team about scores and meaning Method 2

Method 1: Specifics Step 1: Coach completes rating The Coach uses the Benchmarks of Quality Scoring Guide to score each of the 53 items on the Benchmarks of Quality Scoring Form

Method 1: Specifics Step 2: Team Member Ratings The Coach gives the Benchmarks of Quality Team Member Rating Form to each PBIS Leadership Team member to be filled out and returned to the Coach upon completion  All team members should complete their form independently

Method 1: Specifics Step 2: Team Member Ratings (continued) Team members should be instructed to rate each of the 53 items according to whether the component is In Place, Needs Improvement, or Not in Place. Team members place a check in the box to indicate their rating. Note: Some of the items relate to product and process development, others to action items; in order to be rated as In Place, the item must be developed and implemented (where applicable).

Method 1: Specifics Step 3: Record the Team Member Ratings Coach collects the Team Member Rating Forms and tallies the teams’ most frequent response using the Tally Sheet. Record the most frequent response for each item on the Benchmarks of Quality Scoring Form in the Step 2 Column, using ++ for In Place, + for Needs Improvement, and – for Not In Place.

Method 1: Specifics Step 4: Determine discrepancies In the Step 3 Column, place a check in the box where any discrepancies in scoring occur between the team and coach ratings

Discrepancy Guidelines For items with a range of 0 to 1:  1 = In Place  0 = Needs Improvement or Not in Place For items with a range of 0 to 2:  2 = In Place  1 = Needs Improvement  0 = Not in Place For items with a range of 0 to 3:  3 = In Place  2 or 1 = Needs Improvement  0 = Not in Place

Discrepancy Examples Coach scores Item 1 a 2 and the team rating is “In Place”  Discrepancy or No Discrepancy? Coach scores Item 8 a 0 and the team rating is “In Place”  Discrepancy or No Discrepancy? Coach scores Item 19 a 1 and the team rating is “Needs Improvement”  Discrepancy or No Discrepancy?

Method 1: Specifics Step 5: Team Report The Coach completes the Team Summary on p. 3 of the Benchmarks of Quality Scoring Form, recording areas of discrepancy, strength, and weakness.

Method 1: Specifics Step 6: Reporting back to the team The coach reports back to the team using the Team Report page of the Benchmarks of Quality: Scoring Form. If needed, address items of discrepancy and adjust the score.

Discrepancies If there were any items for which the team’s most frequent rating varied from the Coach’s rating based upon the Scoring Guide, the descriptions and exemplars from the guide should be shared with the team. If upon sharing areas of discrepancy, the Coach realizes that there is new information that according to the Scoring Guide would result in a different score, the item and the adjusted final score should be recorded on the Scoring Form.

Method 1: Specifics Step 6: Reporting back to the team (continued) The coach then leads the team through a discussion of the identified areas of strength (high ratings) and weakness (low ratings). The team identifies celebrations to share with staff. The team identifies areas of focus and documents on the PBIS Tier 1/Universal Action Plan.

Method 2: Specifics Step 1: Coach completes rating The Coach uses the Benchmarks of Quality Scoring Guide to score each of the 53 items on the Benchmarks of Quality Scoring Form

Method 2: Specifics Step 2: Team members complete ratings Team members use the Benchmarks of Quality Scoring Guide to score each of the 53 items on the Benchmarks of Quality Scoring Form When completed, team members turn in their Scoring Form to the Coach

Method 2: Specifics Step 3: Record the ratings on the Scoring Form Coach collects all the Scoring Forms and tallies the teams’ most frequent response using the Tally Sheet. The Coach transfers the most frequent response for each item onto a Scoring Form.

Method 2: Specifics Step 4: Team Report The Coach completes the Team Summary on p. 3 of the Benchmarks of Quality Scoring Form, recording areas of strength and weakness. The coach leads the team through a discussion of the identified areas of strength (high ratings) and weakness (low ratings). The team identifies celebrations to share with staff. The team identifies areas of focus and documents on the PBIS Tier 1/Universal Action Plan.

Submitting Your BoQ Method 1 and Method 2 The Coach logs into the school’s PBIS Assessment account and records the final score for each item of the BoQ. This is a new feature. You no longer have to turn in a copy of your BoQ to your KyCID Area Coordinator

Final Notes District level PBIS team members or KYCID staff may validate BoQ findings using the PBIS Walkthrough or the SET In addition, your team can use the PBIS Walkthrough anytime throughout the year for a quick self-assessment

Credits