UNDERSTANDING, PLANNING AND PREPARING FOR THE SCHOOL-WIDE EVALUATION TOOL (SET)

UNDERSTANDING, PLANNING & PREPARING: WHAT IS THE SET?  The SET is an empirically validated tool used to assess level of PBIS implementation.  It was developed at the University of Oregon for the PBIS National Technical Assistance Center as a research tool.  SET scores are the national standard used to measure PIBS implementation (universals).

UNDERSTANDING, PLANNING & PREPARING: WHAT IS THE SET?  Provides objective information to schools about effectiveness of implementation.  Gives teams specific areas to focus on for improvement.  Establishes a baseline to compare implementation over time.  Allows NC PBIS to analyze trends in implementation in order to modify strategies of support.  Helps NC PBIS to understand discrepancies in school outcomes.

UNDERSTANDING, PLANNING & PREPARING: WHAT IS THE SET?  There are four components of the SET:  Administrator interview  Staff and student interviews  Observations in all settings  Document review and scoring  Completed annually (usually in spring).  Takes about 2 hours to complete the SET.  Usually completed between 9:30 am and 11:30 am (or 12:00-2:00).

UNDERSTANDING, PLANNING & PREPARING: WHAT IS THE SET?  SET is typically completed in pairs.  Primary interviewer does the majority of the interviewing and is responsible for completing all required tasks.  Secondary interviewer’s role is to check for agreement on scoring, and to discuss any discrepancies with the primary interviewer.

UNDERSTANDING, PLANNING & PREPARING: WHAT IS THE SET?  Prior to SET:  Contact school to schedule SET  Ensure the school is prepared and has list of products  Day of SET:  Complete the Administrator Interview while beginning to review products.  Complete the staff and student interviews while touring and doing observations.  Review documents and score on site.  After SET:  Write a summary of findings you would like to share with the school to ensure improvement and celebrate successes.

Scheduling the SET:  Identify the contact person at the school.  Contact this person to schedule the SET and ask that they collect each of the available products listed below. Products to Request:  Discipline handbook  School Improvement Plan  Annual action plan for meeting school-wide behavior support goals  Social skills instructional materials/ implementation time line (lesson plans and schedule for teaching the lessons)*  Behavioral incident summaries or reports (e.g., office referrals, suspensions, expulsions)  Office discipline referral form(s)  Disciplinary flowchart*  Budget*  Other related information UNDERSTANDING, PLANNING & PREPARING: WHAT IS THE SET?

ADMINISTERING THE SET Administrative Interview Student and Staff Interviews Observations

ADMINISTERING THE SET: ADMINISTRATOR INTERVIEW  The administrator interview must be completed prior to staff and student interviews.  Get clear answers to each question as staff interviews are scored based on agreement with administrator answers.  Interview typically takes 30 minutes.

ADMINISTERING THE SET: ADMINISTRATOR INTERVIEW TIPS  At the beginning of each section, it is helpful to orient the administrator to the types of questions coming.  It is appropriate to ask follow up questions if you need more information or to re- word questions if he or she doesn’t understand what you are asking.  If the administrator answers “No” to certain questions, you will be prompted to skip ahead.

ADMINISTERING THE SET: ADMINISTRATOR INTERVIEW TIPS  Question #5: Extreme Emergencies- “ What is the procedure called?”  Question #6: Rules or Motto- Clarify that this is the acronym or the way the school refers to expectations/rules.  Question #11: Reinforcers- Focus on the reinforcers that are most frequent (school-wide), and find out how staff and students refer to them.  Question #12: Team Teach Staff PBIS- If answer is yes, follow-up with questions about when and how the procedures were reviewed with staff.  Question #20: School Improvement Plan- Improving or sustaining behavioral practices must be part of a School Improvement Plan Goal, even if it is a Key Process or Action Item.  Question #21: Budget- If answer is yes, follow-up with how PBIS is sustained financially.

ADMINISTERING THE SET: STUDENT AND STAFF INTERVIEWS  Prior to beginning additional interviews and observations, fill in the blanks on the top of the Interview and Observation Form based on the administrator’s answers.  Write the administrator’s answers to questions 4, 5 and 17 on the Interview and Observation Form in the corresponding column.  If you skipped questions during the administrator interview, you need to agree on which additional interview questions you can complete.  Ask the administrator for reinforcers to provide students who participate in interviews (refer to expectations when delivering (ex. “Thank you for being responsible by knowing the 3 Bs.”).

ADMINISTERING THE SET: STUDENT AND STAFF INTERVIEWS  Move throughout the school to conduct interviews of staff members and students.  Strive to obtain a representative sample of students across grade level, sex, and race.  Interview staff members with various roles in the school.  Avoid showing interviewees the score summary sheet to prevent them from being influenced by other’s responses.  Three or four of the staff interviews should be be PBIS team members to answer additional questions.

ADMINISTERING THE SET: STUDENT AND STAFF INTERVIEWS  When you approach staff, you may have to remind them of your purpose and ask if they have time to answer a few questions.  Staff Interviews:  Consist of 7 questions, with 3 additional questions for PBIS team members.  Interview one person at a time.  Try to locate staff in various grade levels.  Attempt to minimize interruption to direct instruction or supervision.  Record number of expectations known(3/4, 1/3, 5/5, etc.), or document the letters of the acronym they know.  Circle either Y or N for yes/no questions.  Record responses for later analysis.

ADMINISTERING THE SET: STUDENT AND STAFF INTERVIEWS  Tell the student that you are conducting a survey and would like to ask a couple of questions. Move on if the student is hesitant.  Student Interviews:  Interview one person at a time. If students are standing in a group, ask only one student to answer or ask the student to move over to the side with you.  Attempt to minimize interruption to direct instruction or supervision.  Record number of expectations known(3/4, 1/3, 5/5, etc.).  Circle either Y or N for yes/no question.

ADMINISTERING THE SET: OBSERVATIONS While interviewing staff and students, complete the required observations in 10 areas.  Look for posted rules and expectations that are specific to the area.  Look for crisis procedures in the required areas (notebook, red bag containing procedures, cards in nametag, etc.). If you cannot locate the items, ask the teacher or staff member to show them to you.

SCORING THE SET

SCORING THE SET: DOCUMENT REVIEW  After completing the observations and interviews, take all notes and documents to a working space on-site.  Complete the scoring guide with the secondary interviewer.  Review documents as needed.  For items requiring documents to score, there will be a P (Product) in the “data source” column.

SCORING THE SET: DOCUMENT REVIEW  The role of the documents is to support sustainable implementation of PBIS.  Documents should contain details clear enough for future training of staff and students.  There are many different ways to document procedures that meet the criteria.

SCORING THE SET: DOCUMENT REVIEW  Most items are easy to score and have clear criteria.  Some items may require discussion between the primary and secondary interviewer to determine score.  Many items require multiple data sources to determine score.  Use the “data source” column on the scoring guide to assist you.

 Complete scoring.  Create a summary of scores and feedback for how to use the information for action planning. This can be done in a Plus/Delta format.  Send feedback to the school PBIS team.  Enter the score in the PBIS DMS (or ensure that the scores are entered by LEA coaches). SCORING THE SET: NEXT STEPS

 SET (revised layout for interview/observations)  Scoring the SET  Ensuring SET Scoring Consistency  Examples of SET summary and feedback forms This presentation adapted from materials created by the PBIS Coordinators in Wake County Public School System, NC. SET RESOURCES AND REFERENCES