1. Evaluation Tools, On-Line Systems, and Data-Based Decision Making Version 3.0, Rev. 12.2.11  This is a presentation of the Illinois PBIS Network. All rights reserved. Adapted from the Illinois PBIS Network

2. Agenda Evaluation Tools:  Team Implementation Checklist (TIC)  Benchmarks of Quality (BoQ)  Self Assessment Survey (SAS) PBIS Assessment @ www.pbisapps.orgwww.pbisapps.org

3. Outcomes of Session  Introduction of Tier 1 Evaluation Tools  Understand purpose of Tools  Understand process of Tool completion  Practice using the Tools  Know where to complete surveys and generate reports on www.pbisapps.orgwww.pbisapps.org

4. Tier 1 Evaluation Tools…  Team Implementation Checklist (TIC)  Benchmarks of Quality (BoQ)  Self-Assessment Survey (SAS)

5.  Effort data (Are we working the plan?)  Team Implementation Checklist (TIC)  Self-Assessment Survey (SAS)  Outcome data (Is it having an effect?)  “Big 5” Graphs  Triangle %  Fidelity data (Are we following the plan?)  School Evaluation Tool (SET)  Benchmarks of Quality (BoQ)

6. Purpose of PBIS Evaluation Tools  Getting Started Self Assessment Survey (SAS) Team Implementation Checklist (TIC)  Getting to Implementation Fidelity Team Implementation Checklist (TIC) Benchmarks of Quality (BoQ)  Sustaining Improvement SAS, BoQ, SET

7. PBIS Assessment Account  Team Implementation Checklist 3.1  Benchmarks of Quality  Self Assessment Survey

8. Your PBIS Assessment Account Your PBIS Technical Assistance Coordinator or External Coach will:  Give each school a unique account number  Provide written directions on how to complete the surveys  Set open/close time periods convenient for school(s) to complete the survey  When a survey period has been closed the survey results and graphs are generated automatically  Closed surveys can be reopened  You can take a survey more than once

9. A Brief Overview of PBIS Assessment: www.pbisassessment.org

10. School Account and Login

11. Select “Surveys” Access your open survey by clicking on “Launch”

12. PBIS Assessment Reports Three reports are available for the Tools: Total Score Subscale Score Individual Item

13. Team Implementation Checklist (TIC)

14.  Self-assessment tool for monitoring the implementation of school-wide PBS  Team self-assessment in six core areas Establish commitment Establish and maintain team Self-assessment Establish school-wide expectations Establish information system Build capacity for function-based support

16. Team Implementation Checklist (TIC)  WHO: Completed by the Universal PBIS Team  WHAT: Guides the development, implementation, monitoring and revision process for building a positive school culture  WHERE: At your school during a Universal PBIS Team meeting  WHEN: 3x per year  HOW: One person, usually the Internal PBIS Coach, enters the data online at www.pbisapps.orgwww.pbisapps.org

17. Online View of Team Checklist Enter date the survey was completed

18. Enter the TIC Data

19. TIC Total Score Report

20. TIC Subscale Report

21. TIC Individual Items Report

22. TIC Activity In a Small Group…  Review the results of the TIC Total Score, Subscale & Individual Item Reports  What do the survey data results say about this Universal team’s functioning?  What next steps would you recommend this team include in their Action Plan?  Why?

23. School-Wide Evaluation Tool SET

24.  The SET is a research-validated instrument that is designed to assess and evaluate the critical features of school-wide effective behavior support across an academic school year. School-Wide Evaluation Tool (SET)

25.  To determine the extent to which schools are already using School-wide Positive Behavior Support (PBIS)  To determine if training and technical assistance efforts result in change when using PBIS  To determine if use of PBIS procedures is related to valued change in the safety, social culture, and violent behavior in schools The Set was Designed To:

26.  Assess features that are in place  Determine annual goals  Evaluate on-going efforts  Design and revise procedures  Compare year to year efforts in the area of PBIS Information Gathered Can Be Used To:

27.  Expectations defined  Behavioral expectations taught  Acknowledgement procedures  Correction procedures  Monitoring and evaluation  Management  District-level support SET Evaluates Questions Across Seven Featured Areas:

28. SWIS

29. What are our “major” and “minor” behavior referral forms indicating? What data options are available when a student needs additional social interventions by a teacher in the classroom (minor) or when other school personnel need to me informed (major), how data gets collected. SWIS – Major & Minor Other district based system

30. Report Options using SWIS:  Average Referrals Per Day Per Month  Referrals By Problem Behavior  Referrals By Location  By Student  Referrals By Time

31. How are we doing? Look for trends Plan time to re-teach

32. Are there target behaviors? Behavior of the week or Staff training/development By Problem Behavior

33. Look for hot spots Review the rules for the locations Review the lessons for teaching the rules By Location

34. By Time

35. By Student

36. How are we doing? Look for trends Plan time to re-teach

38. Do we have a solid green zone? 80%?

39. Self Assessment Survey (SAS)

40.  SAS assesses the extent to which PBIS practices and systems (essential components) are in place within a school School-wide Non-classroom Classroom Individual Student  SAS provides entire staff (certified and non-certified) perspective

42. Self-Assessment Survey (SAS )  WHO: Completed by all teachers, staff and administrators in your school building  WHAT: Examines the status and need for improvement of four behavior support systems (school-wide, classroom, non classroom, individual students)  WHERE: At your school  WHEN: Annually, usually at the beginning or towards the end of each school year  HOW: All school staff take the survey online at PBIS Assessment: www.pbisapps.orgwww.pbisapps.org

43. Online View of Self-Assessment Survey (SAS)

45. School-wide Non-classroom Classroom Individual SAS Total Score Report

46. SAS Subscale Report

47. SAS Individual Items Report

48. Self-Assessment Survey Activity with a Small Group…  Use the SAS Individual Items Report as a reference.  Discuss how you might approach problem solving with your team if the staff identified an item or items as HIGH for Not In Place but LOW for Improvement Priority.

49. Data-Based Decision Making

50. Data-Based Decision-Making SWIS or SWIS Like Systems  Developing Precise Statements  Solution Development Prevention, Teaching, Reward, Extinction, Corrective Consequence, and Data Collection  Action Plan for Results

51. Using Data for On-Going Problem-Solving  Start with the perceptions not the data  Use data in “decision layers” Is there a problem? (overall rate of ODR) Localize the problem  (location, problem behavior, students, time of day) Get specific  Don’t drown in the data  It’s “OK” to be doing well  Be efficient

52. Six Things to Avoid  Define a solution before defining the problem  Build solutions from broadly defined, or fuzzy problem statements  Failure to use data to confirm/define problem  Agree on a solution without building a plan for how to implement or evaluate the solution  Agree on a solution but never assess if the solution was implemented  Serial problem solving without decisions

53. Uses of Data  Review current status and identify problems early. Use data on a regular basis (every two weeks) to monitor key indicators, and identify problems before they become difficult.  Refine a problem statement to a level of precision that will allow functional solutions.  Use data to test possible solutions.  Use data to assess if solutions are working If many students are making the same mistake it typically is the system that needs to change not the students.

54. Benchmarks of Quality (BoQ)

55.  The Benchmarks of Quality measures the degree to which a school is implementing the universal level of school-wide positive behavior support.  The results provide a mechanism for school teams to identify strengths and weakness based on fidelity criteria for use in establishing future action plans.

60. Benchmarks of Quality (BoQ)  WHO: The PBIS Universal Team  WHAT: Lists 10 critical elements of quality for Universal School-wide PBIS implementation. This tool identifies strengths and opportunities for change and growth  WHERE: At your school  WHEN: The spring of each school year Mar/Apr/May  HOW: Online directions at www.pbisillinois.orgwww.pbisillinois.org PoI Narrated Instructions (PPT 2003 & 2007)

61. Technical Assistance Coordinator Enters BoQ Online

62. BoQ Total Score Report

63. BoQ Subscale Report

64. BoQ Individual Items Report

65. BoQ Activity In a Small Group…  Review the results of the BoQ Subscale and BoQ Individual Item Reports.  What action items would you recommend this team consider for improving their Faculty Commitment and Discipline Procedures?

66. Collect and Use and UseData Review Status and Identify Problems Develop and Refine Hypotheses Discuss and Select Solutions Develop and Implement Action Plan Evaluate and Revise Action Plan Problem Solving Foundations Team Initiated Problem Solving (TIPS) Model

67. Identifying Problems/Issues  What data to monitor ODR per day per month OSS, ISS, Attendance, Teacher report Team Checklist/ SET (are we doing what we planned to do?)  What question to answer Do we have a problem?  What questions to ask of Level, Trend, Peaks How do our data compare with last year? How do our data compare with national/regional norms? How do our data compare with our preferred/expected status?  If a problem is identified, then ask What are the data we need to make a good decision?

68. Using Data to Refine Problem Statement  The statement of a problem is important for team-based problem solving. Everyone must be working on the same problem with the same assumptions.  Problems often are framed in a “Primary” form, that creates concern, but that is not useful for problem-solving. Frame primary problems based on initial review of data Use more detailed review of data to build “Solvable Problem Statements.”

69. Precise Problem Statements (What are the data we need for a decision?)  Precise problem statements include information about the Big Five questions: What is problem, and how often is it happening Where is it happening Who is engaged in the behavior When the problem is most likely Why the problem is sustaining

70. Primary vs. Precision Statements  Primary Statements Too many referrals September has more suspensions than last year Gang behavior is increasing The cafeteria is out of control Student disrespect is out of control  Precision Statements There are more ODRs for aggression on the playground than last year. These are most likely to occur during first recess, with a large number of students, and the aggression is related to getting access to the new playground equipment.

71. Primary vs. Precision Statements  Primary Statements Too many referrals September has more suspensions than last year Gang behavior is increasing The cafeteria is out of control Student disrespect is out of control  Precision Statements There are more ODRs for aggression on the playground than last year. These are most likely to occur during first recess, with a large number of students, and the aggression is related to getting access to the new playground equipment.

72. Precise or Primary Statement?  Children are using inappropriate language with a high frequency in the presence of both adults and other children. This is creating a sense of disrespect and incivility in the school.  James D. is hitting others in the cafeteria during lunch, and his hitting is maintained by peer attention.

73. Precise or Primary Statement?  ODRs during December are higher than in any other month.  Minor disrespect and disruption are increasing over time, and are most likely during the last 15 minutes of our block periods when students are engaged in independent seat work. This pattern is most common in 7 th and 8 th grades, involves many students, and appears to be maintained by escape from work (but may also be maintained by peer attention… we are not sure).

74. Precise or Primary Statement?  Three 5 th grade boys are name calling and touching girls inappropriately during recess in an apparent attempt to obtain attention.  Boys are engaging in sexual harassment.