Understanding and Administering the School- Wide Evaluation Tool (SET)

Agenda for today Understanding of the SET and its purpose Process for scheduling and completing Administering the 4 parts of the SET Sign up for dates

In your packet… Slides Supporting Handouts Blank SET Documents Activities

Please make note of time limits and watch your clocks! Trainer will raise his/her hand. Finish your thought/comment. Participants will raise a hand and wait quietly. Attention Signal

Be Responsible Return promptly from breaks Be an active participant Use laptop computers appropriately Be Respectful Maintain cell phone etiquette Listen attentively to others Limit sidebars and stay on topic Be Kind Enter discussions with an open mind Respond appropriately to others’ ideas Participant Expectations

Understanding and Planning for the School-Wide Evaluation Tool (SET)

Understanding and Planning: What is the SET? The SET is an empirically validated tool used to assess level of PBS implementation. It was developed at the University of Oregon for the PBIS National Technical Assistance Center as a research tool. SET scores are the national standard used to measure PBS implementation (universals).

Understanding and Planning: Why Administer the SET? Provides objective information to schools about effectiveness of implementation Gives teams specific areas to focus on for improvement Establishes a baseline to compare implementation over time Allows WCPSS PBS team to analyze trends in implementation in order to modify strategies of support Helps WCPSS to understand discrepancies in school outcomes

Understanding and Planning: How is the SET Administered? There are four components of the SET –Administrator interview –Staff and student interviews –Observations in all settings –Document review and scoring Takes about 2 hours to complete the SET Usually completed between 9:30 am and 11:30 am.

Understanding and Planning: Who administers the SET? SET is completed in pairs. Primary interviewer does the majority of the interviewing and is responsible for completing all required tasks. Secondary interviewer’s role is to check for agreement on scoring, and to discuss any discrepancies with the primary interviewer. Secondary interviewers can ask follow up questions if they need more information to document the answer.

Understanding and Planning: Scheduling Process The SET Scheduling Coordinator will: Create and maintain master schedule Notify all schools of the assigned date and obtain confirmation Re-confirm close to the SET date Handle all re-schedules Troubleshoot any day-of challenges Ensure that all materials are returned to the assigned PBS coach

Understanding and Planning: Scheduling Process Master schedule is on google docs, on laptops. Sign up for dates and choose primary or secondary interviewer. In order to meet our goal, 4 SETS must be completed every day between March 22 rd and April 20 th.

Understanding and Planning: Process for Completing the SET Once you have dates, Javier will be your main contact. Day of SET: –Show up –Complete the Administrator Interview first. –Complete the staff and student interviews while touring and doing observations. –Review documents and score on site. After SET –Write a brief plus/delta of observations you would like to share with the school. –Return all documents and notes to Javier. –Accept our gratitude!

Administering the SET Administrative Interview Student and Staff Interviews Observations Document Review and Scoring

Administering the SET: Administrator Interview The administrator interview must be completed prior to staff and student interviews. Get clear answers to each question as staff interviews are scored based on agreement with administrator answers. Interview typically takes minutes.

Administering the SET: Administrator Interview Tips At the beginning of each section, it is helpful to orient the administrator to the types of questions coming. It is ok to ask follow up questions if you need more information or to re-word if they don’t understand what you are referring to. If the administrator answers no to certain questions you will be prompted to skip ahead.

Administering the SET: Administrator Interview Tips Question #5: Find out how staff would refer to the procedure Question #6: If unclear, reference the acronym Question #11: Focus on the reinforcers that are most frequent, and find out how staff and students refer to them Question #12: If yes, follow-up with when and how procedures were reviewed with staff. Question #20: Key word in this question is goals. Improving or sustaining behavioral practices must be part of a goal Question #21: If yes, follow-up with how

Activity 1: SET Matching Exercise On your own, complete the SET matching exercise sheet by matching the interview questions with the evaluation questions. Compare answers with a partner. Refer to answer key if needed.

Administering the SET: Student and Staff Interviews Prior to beginning additional interviews and observations, fill in the blanks on the top of the interview and observation form based on administrator answers. Write the administrator’s answers to questions 4, 5 and 17 on the interview and observation form in the corresponding column. If you skipped questions on the administrator interview, you need to agree on which additional interview questions you can complete.

Administering the SET: Student and Staff Interviews Move throughout the school to conduct interviews of staff members and students. Strive to obtain a representative sample of students across grade level, sex, and race. Interview staff members with various roles in the school. Four of the staff interviews need to be PBS team members so that they can answer additional questions.

Administering the SET: Student and Staff Interviews When you approach staff, you may have to remind them of your purpose and ask if they have time to answer a few questions Staff Interviews: –7 questions with 3 additional for PBS team members –Interview one person at a time –Try to locate staff in various grade levels –Attempt to minimize interruption to direct instruction or supervision –Record number of expectations known(3/4, 1/3, 5/5, etc.) –Circle Y or N for yes/no questions –Put “+” when staff response matches administrator and a “-” when it does not.

Administering the SET: Student and Staff Interviews Tell the student that you are conducting a survey and would like to ask them 2 questions. Move on if the student is hesitant. Student Interviews: –Interview one person at a time. If students are standing in a group, ask only one student to answer or ask the student to move over to the side with you. –Attempt to minimize interruption to direct instruction or supervision –Record number of expectations known(3/4, 1/3, 5/5, etc.) –Circle Y or N for yes/no question

Activity 2: Role Play Form groups of two or three. Practice approaching both “staff” and “students” in role play. Note any issues or trouble spots.

Administering the SET: Observations While interviewing staff and students, complete the required observations in 10 areas. Look for posted rules and expectations that are specific to the area you are in Ask for documentation of the crisis procedures in the required areas (notebook, red bag containing procedures, cards in nametag, etc.)

Administering the SET: Document Review and Scoring After completing the observations and student and staff interviews take all notes and documents to a space on-site. Complete the scoring guide with the secondary interviewer. Review documents as needed For items requiring documents to score there will be a P in the “data source” column.

Administering the SET: Document Review and Scoring The role of documents is to support sustainable implementation of PBS. Documents should contain details clear enough for future training of staff and students. There are many different ways to document procedures that meet the criteria.

Administering the SET: Scoring Guidelines Most items are easy to score and have clear criteria for 0,1,2. Some items may require discussion between the primary and secondary interviewer to determine score. Many items require multiple data sources to determine score. Use the “data source” column on the scoring guide to assist you.

Administering the SET: Scoring Guidelines Common Scoring Challenges SIP Action Plans Budget When in doubt leave an item blank, make copies of necessary documents and send to Javier for scoring

Activity 3: Case Study On your own, review all the materials from the provided case study and complete the scoring guide. Compare answers with a partner and discuss any discrepancies.