1. School-wide Evaluation Tool (SET)
Version 2.1
Kathleen Strickland-Cohen Ph.D., BCBA-D
TCU

2. Training Goals Define Purpose of the School-wide Evaluation Tool (SET)
Process
Who participates?
Duration and Data-collection protocol
What permanent products will you need to see/collect?
Item Analysis
Considerations for each of the items
Scoring Practice

3. Purpose of the SET
The purpose of the SET is to provide an efficient and valid index of the extent to which PBIS core features are in place within a school.
Tier I (Universal PBIS)
Whole School Universal Prevention
Conducted annually by external
coach/evaluator

4. Uses of the SET Formative Assessment Annual Self-Assessment
Determine current PBIS practices in place and needed prior to launching implementation
Annual Self-Assessment
Self-assess annually to facilitate sustained implementation of PBIS
Progress monitoring
Build action plan to focus implementation efforts
District/State Recognition
Determine schools warranting recognition for their fidelity of PBIS implementation
Research Studies on Effectiveness of SW-PBIS

5. SET Related Research
The SET is a research-validated tool that has been shown to be:
Reliable
Test-Retest (M = 97.3%)
IOA (M = 99%)
Valid
Construct Validity (Pearson r = .71)
Internal Consistency
27 of 28 items are highly correlated
“Crisis plan readily available”
kept in SET for safety and legal reasons

6. School-wide Evaluation Tool
How to Use the SET
School-wide Evaluation Tool

7. Administering the SET Interviews Observations/Walk-through
Administrator
10-15 randomly selected staff
A minimum of 3 PBIS team members
15 Randomly selected students
Observations/Walk-through
School rules posted in 10 locations
Crisis procedure posted in 7 locations
Permanent products

8. Features of the Scoring Tool
Feature
Evaluation Questions
Data Source
Score: 0-2
Essential Features of Universal Support
Collection of Evidence to Document the Implementation of Tier I Essential Features
P = Product
I = Interview
O = Observation
0 = descriptors of the feature not being implemented
1 = descriptors of the feature being partially implemented
2 = descriptors of the feature being fully implemented

9. Permanent Product Review
Discipline Handbook
School Improvement Plan Goals
Annual Action Plan
Social Skills Instructional Materials/Lesson Plans
Behavior Incident Summaries or Reports
ODR Forms

10. Essential Features of Universal Support
School-wide Evaluation Tool

11. SET Features (# of evaluation questions)
Expectations Defined (2)
Expectations Taught (5)
Acknowledgement Procedures (3)
Corrective Procedures (4)
Monitoring & Evaluation (4)
Management (8)
District-level Support (2)

12. A & B: Expectations Defined and Taught

13. Behavioral Expectations: Permanent Products
Is there documentation that staff has agreed to 5 or fewer positively stated school rules/ behavioral expectations?
(0=no; 1= too many/negatively focused; 2 = yes)
Is there agreement between information from administrator and staff interviews and publicly posted rules?
Is there a documented SYSTEM for teaching behavioral expectations to students on an annual basis?
(0= no; 1 = states that teaching will occur; 2= yes)
Let’s look at some examples…

14. Example: Score of 2

15. Example: Score of1 What is Missing?
How Could We Make this Example a 2?

16. C: Rewarding Behavioral Expectations
School-wide Evaluation Tool

17. C: Rewarding Behavioral Expectations
The key feature is a Documented SYSTEM for acknowledging appropriate student behaviors

18. Example: Score of 2

19. Example: Score of1

20. D: Responding to Behavioral Violations

21. Example: Score of 2

22. E: Monitoring & Decision Making

23. Activity
Does the discipline referral form list: (1) student/grade, (2) date, (3) time, (4) referring staff, (5) problem behavior, (6) location, (7) persons involved, (8) probable motivation, & (9) administrative decision?
0= 0-3 items = 4-6 items 2= 7-9 items
Using the ODR forms provided, work with a partner to identify any missing information and score the forms as:
0=0-3 items
1= 4-6 items
2= 7-9 items

24. F & G: Management and District Level Support

25. Management: Permanent Products
F1: Does the school improvement plan list improving behavior support systems as one of the top 3 school improvement plan goals?
Note: Academic Goals can be combined/collapsed
F8: Does the team have an action plan with specific goals that is less than one year old?
Let’s look at an example…

26. Example: Annual Action Plan

27. Interviews & Observations

28. Typical Sequence of Administration
Schedule date and define expectations—time, people, materials
Interviews/Observations
Administrator interview should be conducted first
Allow for min for administrator interview and document review
Best to schedule 30 to 45 minutes before a natural break in the schedule (e.g., lunch, recess)
Walk-through Observations, Student/Staff Interviews
Can be conducted simultaneously and/or divided up among a pair of evaluators
Should be conducted when academic instruction is not occurring
Action Plan Development
May be done at same time or at a later meeting

29. Possible Pre-Administration Documentation
School/Discipline Handbook
Documentation of systems for:
Teaching behavioral expectations
Rewarding appropriate student behaviors
Addressing and reporting specific behavioral violations
Office Discipline Referral Form
School Improvement Plan
School/PBIS Team Action Plan

30. Administrator Interview
Discipline System
Do you collect and summarize office discipline referral information? Yes No (If no, skip to #4)
What system do you use for collecting and summarizing office referrals? (E2)
What data do you collect?
Who collects and enters the data?
What do you do with the ODR information? (E3)
Who looks at the data?
How often do you share it with other staff?
What type of problems do you expect teachers to refer to the office rather than handling in the classroom? (D2)
What is the procedure for handling extreme emergencies in the building (i.e., stranger with a gun)? (D4)

31. Administrator Questions, cont’d
School-wide Expectations
6. Do you have school rules or a motto? Yes No (If no, skip to 10)
7. How many are there?
8. What are the rules/motto? (B4, B5)
9. What are they called? (B4, B5)
10. Do you acknowledge students for doing well socially? Yes No
11. If yes, what are the social acknowledgements/activities/ routines called (student of the month, positive referral, letter home, stickers, high 5’s)? (C2, C3)

32. Administrator Questions, cont’d
Data Systems and Team
Has the team taught/reviewed the school-wide program with staff this year? Yes No (B3)
Is your school-wide team representative of your school staff? Yes No (F3)
Are you on the team? Yes No (F5)
How often does the team meet? (F6)
Do you attend team meetings consistently? Yes No (F5)
Who is your team leader/facilitator? (F4)
Does the team provide updates to faculty on activities & data summaries? Yes No If Yes, how often? (E3, F7)
Do you have an out-of-school liaison in the state or district to support you on positive behavior support systems development? Yes No (G2)
If yes, who?
What are your top 3 school improvement goals? (F1)
Does the School budget contain an allocated amount of money for building and maintaining school-wide behavior support? Yes No (G1)

33. Staff and Student Interviews

34. Staff/Faculty (at least 10) Questions
What are the __________________ (school rules, high 5's, 3 bee’s)? (B5)
(Define what the acronym means)
Have you taught the school rules/behavioral expectations this year? (B2)
Have you given out any (rewards for appropriate behavior) since (2 month ago)? (C3)
What types of student problems do you or would you refer to the office? (D2)
What is the procedure for dealing with a stranger with a gun? (D4)
Is there a school-wide team that addresses behavioral support in your building?
Are you on the team?

35. Team Member (3-4) Questions
Does your team use discipline data to make decisions? (E4)
Has your team taught/reviewed the school-wide program with staff this year? (B3)
Who is the team leader/facilitator? (F4)

36. Student (15) Questions
*Important to interview a representative sample of students
1. What are the _________________ (school rules, high 5's, 3 bee’s, define what the acronym means)? (B4)
2. Have you received a _____________ since (2 months ago)? (C2)
Hint: When talking to young students, select a recent holiday or break to refer back to (e.g., Have you received a High-Five Card from your teacher since Spring Break/Halloween/etc.)

37. Tip: Scoring Behavioral Expectations Taught
B4: Can at least 70% of 15 or more students state 67% of the school rules?
(0= 0-50%; 1= 51-69%; 2= %)
B5: Can 90% or more of the staff asked list 67% of the school rules?
(0= 0-50%; 1= 51-89%; 2=90%-100%)
Translation:
Can at least 11 of 15 students and 14 of 15 staff interviewed state:
2 out of 3;
3 out of 4; or
4 out of 5 school-wide expectations

38. Let’s Practice…

39. SAMPLE INTERVIEWS/WALKTHROUGH GO HERE
ALONG WITH MATCHING SCORING SHEET

40. Your Turn! Small Group Activity
Provide participants with examples of completed admin interview and walk-through, have them score.
Small Group Activity

41. SET Overall Mean Score
When entering SET data into PBIS Assessment, there is no need to calculate Mean scores by hand…
Understanding how scores are determined:
Calculate percentage pts for each of seven subscales
Total the percentage pts
Divide total by 7 to get overall mean

42. Calculating SET Overall Mean Scores
Example 1
Example 2
A = 3 /4 = 75%
B = 5/10 = 50%
C = 3/6 = 50%
D = 4/8 = 50%
E = 4/8 = 50%
F = 4/16 = 25%
G = 2/4 = 50%
Total = 350
Mean = 350/7 = 50%
A = 4 /4 = 100%
B = 6/10 = 60%
C = 4/6 = 67%
D = 8/8 = 100%
E = 4/8 = 50%
F = 9/16 = 56%
G = 3/4 = 75%
Total = 508
Mean = 508/7 = 72.5%

43. What is Meant by 80/80? A = 3 /4 = 75% A = 4 /4 = 100% B = 5/10 = 50%
Mean of at least 80% on Subscale B “Behavioral Expectations Taught” and Overall Implementation Score
Would the following scores meet the 80/80 criteria?
Example 1
Example 2
A = 3 /4 = 75%
B = 5/10 = 50%
C = 3/6 = 50%
D = 4/8 = 50%
E = 4/8 = 50%
F = 4/16 = 25%
G = 2/4 = 50%
Total = 350
Mean = 350/7 = 50%
A = 4 /4 = 100%
B = 6/10 = 60%
C = 4/6 = 67%
D = 8/8 = 100%
E = 4/8 = 50%
F = 9/16 = 56%
G = 3/4 = 75%
Total = 508
Mean = 508/7 = 72.5%

44. For more information:
Horner, R. H., Todd, A. W., Lewis-Palmer, T., Irvin, L. K., Sugai, G., & Boland, J. B. (2004). The school-wide evaluation tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6(1), 3-12.

45. Kathleen Strickland-Cohen, Ph.D., BCBA-D
Thank You!
Kathleen Strickland-Cohen, Ph.D., BCBA-D