Benchmarks of Quality (BOQ) Training Thank you for agreeing to complete the Benchmarks of Quality During this training, I will walk you through the process of completing the benchmarks

Objectives Understand purpose of Benchmarks (BoQ) Understand how to administer the BoQ

Why is Evaluation Important? To gain an understanding of how the program is functioning “Are we really doing what we think we are doing?” To document program effectiveness “Is what we’re doing working?” To identify and examine strengths and weaknesses of the program Celebrate success Identify areas to improve Why is evaluation Important? “Are we really doing what we think we are doing?” “Is what we’re doing working?” Celebrate success Identify areas to improve

School-wide Benchmarks of Quality Assesses development and implementation of school-wide PBS Lists 53 benchmarks of quality in school-wide PBS programs within 10 critical elements Completed by school teams periodically to identify areas of strength and weakness Used by state and district to guide TA and training, identify model schools, evaluate outcomes related to level of implementation The School-wide Benchmarks of Quality is the tool we are using for evaluation Lists over 50 benchmarks of quality school-wide PBS programs that address 10 critical elements To be completed by school teams on a yearly basis to assess how they score on a 100 point scale with regard to developing and implementing school-wide PBS Is one of the measures used by the PBS Project to determine model schools

BoQ (Revised) Eliminated the Crisis Section Added Classroom Section (7 items) Few items removed, rearranged, or reworded for clarity New total score is 107 Be sure to use the new version! Are you classroom items last? If so, you’ve got an old version.

Completing the Benchmarks

3 Elements of the Benchmarks of Quality Team Member Rating Form Completed by team members independently Returned to coach/facilitator Scoring Form Completed by coach/facilitator using Scoring Guide Used for reporting back to team Scoring Guide Describes administration process Rubric for scoring each item Take a minute to pull out each of these forms 1. The Team Member Rating form (columns say –in place, needs improvement, not in place) This is the form for the team members to complete. 2. Scoring Form (columns with numbers) this is the form for the coach or designated person to complete 3. Scoring Guide (large packet with directions on front) this is the packet to help guide your decisions about which rating to chose For the remainder of the training, the top of the slide will always tell you which form I am discussing at the time

Method of Completion Coach/facilitator uses Scoring Guide to ascertain the appropriate score for each item, collects Team Member Rating forms, resolves any discrepancies, and reports back to team Alt. Option – Scoring Form is completed at a team meeting with all members reaching consensus on the appropriate score for each item using the Scoring Guide rubric. The team identifies areas of strength and need.

Completion of BoQ Step 1 – Coach’s Scoring The Coach/facilitator will use his or her best judgment based on personal experience with the school and the descriptions and exemplars in the Benchmarks of Quality Scoring Guide to score each of the 53 items on the Benchmarks of Quality Scoring Form (p.1 & 2). Do not leave any items blank. Look at the scoring guide The procedures are listed on the first page Read Step 1 - Coaches Scoring with me “The Coach will use his or her best judgment based on personal experience with the school and the descriptions and exemplars in the Benchmarks of Quality Scoring Guide to score each of the 53 items on the Benchmarks of Quality Scoring Form (p.1 & 2). Do not leave any items blank. “

Benchmarks Practice: Scoring Form, Scoring Guide Critical Elements STEP 1 STEP 2 ++, +, or _ STEP 3 1 Team has administrative support 3 2 1 2 Team has regular meetings (at least monthly) 3 Team has established a clear mission/purpose You will see that the items on the scoring form and scoring guide correspond Read the description for item 1 on page two of the the scoring guide One point: Includes all of the following: SAC team member, Administrator (i.e., principal, asst. principal or dean), reg. ed. teacher, spec. ed. teacher, member with behavior expertise, and a coach/district-level representation. Some groups are not represented on the team. Zero points: Some groups are not represented on the team. Decide for you team which response best fits Let’s do one more. Look at item #2. Team has administrative support: 3 points: Administrator(s) attended training, play an active role in the PBS process, actively communicate their commitment, attends all team meetings, and supports the decisions of the PBS Team. 2 points: Administrator(s) support the process but do not attend all meetings or take as active a role as the rest of the team. 1 point: Administrator(s) support the process but attend only a few meetings or doesn’t take as active a role as the rest of the team. 0 points: Administrator(s) do not actively support the PBS process. Decide which best fits your team There may be some items that do not fit your school 100%. In this case, make your best guess.

Benchmarks Practice: Scoring Form, Scoring Guide Critical Elements STEP 1 STEP 2 ++, +, or _ STEP 3 1 Team has administrative support 3 2 1 2 Team has regular meetings (at least monthly) 3 Team has established a clear mission/purpose You will see that the items on the scoring form and scoring guide correspond Read the description for item 1 on page two of the the scoring guide One point: Includes all of the following: SAC team member, Administrator (i.e., principal, asst. principal or dean), reg. ed. teacher, spec. ed. teacher, member with behavior expertise, and a coach/district-level representation. Some groups are not represented on the team. Zero points: Some groups are not represented on the team. Decide for you team which response best fits Let’s do one more. Look at item #2. Team has administrative support: 3 points: Administrator(s) attended training, play an active role in the PBS process, actively communicate their commitment, attends all team meetings, and supports the decisions of the PBS Team. 2 points: Administrator(s) support the process but do not attend all meetings or take as active a role as the rest of the team. 1 point: Administrator(s) support the process but attend only a few meetings or doesn’t take as active a role as the rest of the team. 0 points: Administrator(s) do not actively support the PBS process. Decide which best fits your team There may be some items that do not fit your school 100%. In this case, make your best guess.

Completion of BoQ Step 2 – Team Member Rating The coach/facilitator will give the Benchmarks of Quality Team Member Rating Form to each SWPBS Team member to be completed independently and returned to the coach upon completion. Members should be instructed to rate each of the 53 items according to whether the component is “In Place,” “Needs Improvement,” or “Not in Place.” Some of the items relate to product and process development, others to action items; in order to be rated as “In Place;” the item must be developed and implemented (where applicable). Coaches will collect and tally responses and record on the Benchmarks of Quality Scoring Form the team’s most frequent response using ++ for “In Place,” + for “Needs Improvement,” and – for “Not In Place.” These are the directions for step two of the scoring guide. Give the team member rating forms to all the members of your team. Have the members complete them. Members should be instructed to rate each of the 53 items according to whether the component is “In Place”, “Needs Improvement”, or “Not in Place”. Some of the items relate to product and process development, others to action items; in order to be rated as “In Place;” the item must be developed and implemented (where applicable). It is best to give them a deadline to make sure you get them back. Collect all the forms. Now, once you have the forms to review, you must decide what is the most frequent response for your team. Let’s see an example:

Benchmarks Practice: Scoring Form, Team Members Rating Form In Place (++) Needs Improvement (+) Not In Place (-) STEP 1 STEP 2 ++, +, or _ STEP 3 2. Team has administrative support 3 2 1 ++ + Team Member A 2. Team has administrative support X My responses to items 1 and 2 are at the top of the page. Three sample team member’s responses to items 1 and 2 are at the bottom of the page. You now need to decide which is the most frequent response For item one-in blue, it was unanimous so I entered two pluses to indicate the team’s most frequent score (click to see pluses) For item two, two people said “needs improvement,” while One person said “in place” Since 2 of 3 said “needs improvement” what would you score?? I would score item 2 as a “+” (Click to see ) If any item does not have a most frequent response (e.g. 2 team members rated the item as “in place,” 2 as “needs improvement” And 2 as “not in place”, leave it blank or write a note indicating no consensus. Team Member B 2. Team has administrative support X Team Member C 2. Team has administrative support X

Completion of BoQ Step 3 - Team Report The coach will then complete the Team Summary on p. 3 of the Benchmarks of Quality Scoring Form recording areas of discrepancy, strength and weakness. Discrepancies - If there were any items for which the team’s most frequent rating varied from the coaches’ rating based upon the Scoring Guide, the descriptions and exemplars from the guide should be shared with the team. This can happen at a team meeting or informally. If upon sharing areas of discrepancy, the coach realizes that there is new information that according to the Scoring Guide would result in a different score, the item and the adjusted final score should be recorded on the Scoring Form Now for Step 3, you will check your scores with your team’s scores for any discrepancies Follow along as I read the directions. “The coach will then complete the Team Summary on p. 3 of the Benchmarks of Quality Scoring Form recording areas of discrepancy, strength and weakness. “ Discrepancies - If there were any items for which the team’s most frequent rating varied from the coaches’ rating based upon the Scoring Guide, the descriptions and exemplars from the guide should be shared with the team. This can happen during the team report so you do not need an additional meeting. If upon sharing areas of discrepancy, the coach realizes that there is new information that according to the Scoring Guide would result in a different score, the item and the adjusted final score should be recorded on the coaches Scoring Form

Benchmarks Practice: Scoring Form, Team Members Rating Form In Place (++) Needs Improvement (+) Not In Place (-) STEP 1 STEP 2 ++, +, or _ STEP 3 Team has broad representation 1 2. Team has administrative support 3 2 ++  + Team Member A Team has broad representation X 2. Team has administrative support My responses to items 1 and 2 are at the top of the page. Three sample team member’s responses to items 1 and 2 are at the bottom of the page. You now need to decide which is the most frequent response For item one-in blue, it was unanimous so I entered two pluses to indicate the team’s most frequent score (click to see pluses) For item two, two people said “needs improvement,” while One person said “in place” Since 2 of 3 said “needs improvement” what would you score?? I would score item 2 as a “+” (Click to see ) If any item does not have a most frequent response (e.g. 2 team members rated the item as “in place,” 2 as “needs improvement” And 2 as “not in place”, leave it blank or write a note indicating no consensus. Team Member B Team has broad representation X 2. Team has administrative support Team Member C Team has broad representation X 2. Team has administrative support

Completion of BoQ Step 4 – Reporting Back to Team After completing the remainder of the Benchmarks of Quality: Scoring Form, the coach will report back to the team using the Team Report page of the Benchmarks of Quality: Scoring Form. If needed, address items of discrepancy and adjust the score. The coach will then lead the team through a discussion of the identified areas of strength (high ratings) and weakness (low ratings). This information should be conveyed as “constructive feedback” to assist with action planning. Follow along as I read the directions for Step 4. After completing the remainder of the Benchmarks of Quality: Scoring Form, the coach will report back to the team using the Team Report page of the Benchmarks of Quality: Scoring Form. If needed, address items of discrepancy and adjust the score. the coach will lead the team through a discussion of the identified areas of strength (high ratings) and weakness (low ratings). This information should be conveyed as “constructive feedback” to assist with action planning. This element of the Benchmarks is an opportunity for the team to understand how they are doing, celebrate their success, and make plans to improve.

Benchmarks Team Summary: Scoring Form Areas of Discrepancy Item # Team Response Coach’s Score Scoring Guide Description ++, ++, + Administrator does not actively support the process 2 Areas of Strength Critical Element Description of Areas of Strength Turn to Page 3 of the Scoring Form. This is called the Team Summary. Now return to the example on the screen. Remember that I noted item 2 as an area of discrepancy. So I listed Item #2, the team’s responses, my score for the item, and the scoring guide description for my score. This area of discrepancy will generate a discussion with the team about why they put a higher rating than I did. As it turns out, although the administrator did not attend meetings, the team leader met with him often and he frequently addressed the faculty in support of the PBS team’s plans. So, I changed the score I gave to Item 2 to reflect this information and gave them a 1. Let’s move on to the next part of the Team Summary. This is where you choose the top areas of strength and top areas in need of development. The team may have more than space allows, just use your best judgment in selecting which items to include. Areas in Need of Development Critical Element Description of Areas in Need of Development

Alternative Option for Completion of BoQ *(method has been validated empirically)

Alternative Option Step 1 – Team Member Scoring The team member uses personal experience with PBS and the descriptions and exemplars in the Benchmarks of Quality Scoring Guide ) for each of the 53 items on the Benchmarks of Quality Scoring Form (p.1 & 2). The team will meet and reach consensus on the appropriate score for each item. Look at the scoring guide The procedures are listed on the first page Read Step 1 - Coaches Scoring with me “The Coach will use his or her best judgment based on personal experience with the school and the descriptions and exemplars in the Benchmarks of Quality Scoring Guide to score each of the 53 items on the Benchmarks of Quality Scoring Form (p.1 & 2). Do not leave any items blank. “

Alternative Option Step 2 – Team Summary After completing the Benchmarks of Quality: Scoring Form, the team should use the Team Report page of the Benchmarks of Quality: Scoring Form to guide a discussion of the identified areas of strength (high ratings) and weakness (low ratings). This information should be used as “constructive feedback” to assist with action planning. Look at the scoring guide The procedures are listed on the first page Read Step 1 - Coaches Scoring with me “The Coach will use his or her best judgment based on personal experience with the school and the descriptions and exemplars in the Benchmarks of Quality Scoring Guide to score each of the 53 items on the Benchmarks of Quality Scoring Form (p.1 & 2). Do not leave any items blank. “

Submitting Your Evaluation Step 5 – Reporting/Entering Data The coach/facilitator will enter the final scores from the Benchmarks of Quality: Scoring Form along with other End-Year Evaluation items on PBSES, the web-based reporting system accessed through the Project’s website: http://flpbs.fmhi.usf.edu. Step 5 - Reporting to District Coordinator or whoever you report to. . . The coach will forward a copies of the Benchmarks of Quality: Scoring Form and all of the Team Member Rating Forms to the to the district coordinator or the person that has been designated in Maryland.