School-wide Evaluation Tool (SET) Version 2.1 Kathleen Strickland-Cohen Ph.D., BCBA-D TCU

Training Goals Define Purpose of the School-wide Evaluation Tool (SET) Process Who participates? Duration and Data-collection protocol What permanent products will you need to see/collect? Item Analysis Considerations for each of the items Scoring Practice

Purpose of the SET The purpose of the SET is to provide an efficient and valid index of the extent to which PBIS core features are in place within a school. Tier I (Universal PBIS) Whole School Universal Prevention Conducted annually by external coach/evaluator

Uses of the SET Formative Assessment Annual Self-Assessment Determine current PBIS practices in place and needed prior to launching implementation Annual Self-Assessment Self-assess annually to facilitate sustained implementation of PBIS Progress monitoring Build action plan to focus implementation efforts District/State Recognition Determine schools warranting recognition for their fidelity of PBIS implementation Research Studies on Effectiveness of SW-PBIS

SET Related Research The SET is a research-validated tool that has been shown to be: Reliable Test-Retest (M = 97.3%) IOA (M = 99%) Valid Construct Validity (Pearson r = .71) Internal Consistency 27 of 28 items are highly correlated “Crisis plan readily available” kept in SET for safety and legal reasons

School-wide Evaluation Tool How to Use the SET School-wide Evaluation Tool

Administering the SET Interviews Observations/Walk-through Administrator 10-15 randomly selected staff A minimum of 3 PBIS team members 15 Randomly selected students Observations/Walk-through School rules posted in 10 locations Crisis procedure posted in 7 locations Permanent products

Features of the Scoring Tool   Feature Evaluation Questions Data Source Score: 0-2 Essential Features of Universal Support Collection of Evidence to Document the Implementation of Tier I Essential Features P = Product I = Interview O = Observation 0 = descriptors of the feature not being implemented 1 = descriptors of the feature being partially implemented 2 = descriptors of the feature being fully implemented

Permanent Product Review Discipline Handbook School Improvement Plan Goals Annual Action Plan Social Skills Instructional Materials/Lesson Plans Behavior Incident Summaries or Reports ODR Forms

Essential Features of Universal Support School-wide Evaluation Tool

SET Features (# of evaluation questions) Expectations Defined (2) Expectations Taught (5) Acknowledgement Procedures (3) Corrective Procedures (4) Monitoring & Evaluation (4) Management (8) District-level Support (2)

A & B: Expectations Defined and Taught

Behavioral Expectations: Permanent Products Is there documentation that staff has agreed to 5 or fewer positively stated school rules/ behavioral expectations? (0=no; 1= too many/negatively focused; 2 = yes) Is there agreement between information from administrator and staff interviews and publicly posted rules? Is there a documented SYSTEM for teaching behavioral expectations to students on an annual basis? (0= no; 1 = states that teaching will occur; 2= yes) Let’s look at some examples…

Example: Score of 2

Example: Score of1 What is Missing? How Could We Make this Example a 2?

C: Rewarding Behavioral Expectations School-wide Evaluation Tool

C: Rewarding Behavioral Expectations The key feature is a Documented SYSTEM for acknowledging appropriate student behaviors

Example: Score of 2

Example: Score of1

D: Responding to Behavioral Violations

Example: Score of 2

E: Monitoring & Decision Making

Activity Does the discipline referral form list: (1) student/grade, (2) date, (3) time, (4) referring staff, (5) problem behavior, (6) location, (7) persons involved, (8) probable motivation, & (9) administrative decision? 0= 0-3 items 1= 4-6 items 2= 7-9 items Using the ODR forms provided, work with a partner to identify any missing information and score the forms as: 0=0-3 items 1= 4-6 items 2= 7-9 items

F & G: Management and District Level Support

Management: Permanent Products F1: Does the school improvement plan list improving behavior support systems as one of the top 3 school improvement plan goals? Note: Academic Goals can be combined/collapsed F8: Does the team have an action plan with specific goals that is less than one year old? Let’s look at an example…

Example: Annual Action Plan

Interviews & Observations

Typical Sequence of Administration Schedule date and define expectations—time, people, materials Interviews/Observations Administrator interview should be conducted first Allow for 30-45 min for administrator interview and document review Best to schedule 30 to 45 minutes before a natural break in the schedule (e.g., lunch, recess) Walk-through Observations, Student/Staff Interviews Can be conducted simultaneously and/or divided up among a pair of evaluators Should be conducted when academic instruction is not occurring Action Plan Development May be done at same time or at a later meeting

Possible Pre-Administration Documentation School/Discipline Handbook Documentation of systems for: Teaching behavioral expectations Rewarding appropriate student behaviors Addressing and reporting specific behavioral violations Office Discipline Referral Form School Improvement Plan School/PBIS Team Action Plan

Administrator Interview Discipline System Do you collect and summarize office discipline referral information? Yes No (If no, skip to #4) What system do you use for collecting and summarizing office referrals? (E2) What data do you collect? Who collects and enters the data? What do you do with the ODR information? (E3) Who looks at the data? How often do you share it with other staff? What type of problems do you expect teachers to refer to the office rather than handling in the classroom? (D2) What is the procedure for handling extreme emergencies in the building (i.e., stranger with a gun)? (D4)

Administrator Questions, cont’d School-wide Expectations 6. Do you have school rules or a motto? Yes No (If no, skip to 10) 7. How many are there? 8. What are the rules/motto? (B4, B5) 9. What are they called? (B4, B5) 10. Do you acknowledge students for doing well socially? Yes No 11. If yes, what are the social acknowledgements/activities/ routines called (student of the month, positive referral, letter home, stickers, high 5’s)? (C2, C3)

Administrator Questions, cont’d Data Systems and Team Has the team taught/reviewed the school-wide program with staff this year? Yes No (B3) Is your school-wide team representative of your school staff? Yes No (F3) Are you on the team? Yes No (F5) How often does the team meet? (F6) Do you attend team meetings consistently? Yes No (F5) Who is your team leader/facilitator? (F4) Does the team provide updates to faculty on activities & data summaries? Yes No If Yes, how often? (E3, F7) Do you have an out-of-school liaison in the state or district to support you on positive behavior support systems development? Yes No (G2) If yes, who? What are your top 3 school improvement goals? (F1) Does the School budget contain an allocated amount of money for building and maintaining school-wide behavior support? Yes No (G1)

Staff and Student Interviews

Staff/Faculty (at least 10) Questions What are the __________________ (school rules, high 5's, 3 bee’s)? (B5) (Define what the acronym means) Have you taught the school rules/behavioral expectations this year? (B2) Have you given out any (rewards for appropriate behavior) since (2 month ago)? (C3) What types of student problems do you or would you refer to the office? (D2) What is the procedure for dealing with a stranger with a gun? (D4) Is there a school-wide team that addresses behavioral support in your building? Are you on the team?

Team Member (3-4) Questions Does your team use discipline data to make decisions? (E4) Has your team taught/reviewed the school-wide program with staff this year? (B3) Who is the team leader/facilitator? (F4)

Student (15) Questions *Important to interview a representative sample of students 1. What are the _________________ (school rules, high 5's, 3 bee’s, define what the acronym means)? (B4) 2. Have you received a _____________ since (2 months ago)? (C2) Hint: When talking to young students, select a recent holiday or break to refer back to (e.g., Have you received a High-Five Card from your teacher since Spring Break/Halloween/etc.)

Tip: Scoring Behavioral Expectations Taught B4: Can at least 70% of 15 or more students state 67% of the school rules? (0= 0-50%; 1= 51-69%; 2= 70-100%) B5: Can 90% or more of the staff asked list 67% of the school rules? (0= 0-50%; 1= 51-89%; 2=90%-100%) Translation: Can at least 11 of 15 students and 14 of 15 staff interviewed state: 2 out of 3; 3 out of 4; or 4 out of 5 school-wide expectations

Let’s Practice…

SAMPLE INTERVIEWS/WALKTHROUGH GO HERE ALONG WITH MATCHING SCORING SHEET

Your Turn! Small Group Activity Provide participants with examples of completed admin interview and walk-through, have them score. Small Group Activity

SET Overall Mean Score When entering SET data into PBIS Assessment, there is no need to calculate Mean scores by hand… Understanding how scores are determined: Calculate percentage pts for each of seven subscales Total the percentage pts Divide total by 7 to get overall mean

Calculating SET Overall Mean Scores Example 1 Example 2 A = 3 /4 = 75% B = 5/10 = 50% C = 3/6 = 50% D = 4/8 = 50% E = 4/8 = 50% F = 4/16 = 25% G = 2/4 = 50% Total = 350 Mean = 350/7 = 50% A = 4 /4 = 100% B = 6/10 = 60% C = 4/6 = 67% D = 8/8 = 100% E = 4/8 = 50% F = 9/16 = 56% G = 3/4 = 75% Total = 508 Mean = 508/7 = 72.5%

What is Meant by 80/80? A = 3 /4 = 75% A = 4 /4 = 100% B = 5/10 = 50% Mean of at least 80% on Subscale B “Behavioral Expectations Taught” and Overall Implementation Score Would the following scores meet the 80/80 criteria? Example 1 Example 2 A = 3 /4 = 75% B = 5/10 = 50% C = 3/6 = 50% D = 4/8 = 50% E = 4/8 = 50% F = 4/16 = 25% G = 2/4 = 50% Total = 350 Mean = 350/7 = 50% A = 4 /4 = 100% B = 6/10 = 60% C = 4/6 = 67% D = 8/8 = 100% E = 4/8 = 50% F = 9/16 = 56% G = 3/4 = 75% Total = 508 Mean = 508/7 = 72.5%

For more information: Horner, R. H., Todd, A. W., Lewis-Palmer, T., Irvin, L. K., Sugai, G., & Boland, J. B. (2004). The school-wide evaluation tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6(1), 3-12. www.pbis.org/evaluation www.pbisassessment.org

Kathleen Strickland-Cohen, Ph.D., BCBA-D Thank You! Kathleen Strickland-Cohen, Ph.D., BCBA-D m.k.strickland@tcu.edu