Say: Welcome to today’s professional learning session on inter-rater reliability. Inter-rater reliability is essential to the teacher evaluation process to eliminate biases and sustain transparency, consistency, and impartiality.  This work will not only deepen our own understanding of TEAM, but also improve teacher perception of our local implementation of teacher evaluation. *Note to facilitator—survey responses from the Tennessee Educator Survey might be shared to reinforce the need of this module. See TM_E5.d on the Tennessee Educator Survey at http://educatorsurvey.tnk12.gov/. *Note to facilitator—if completing all modules of the professional learning package, Say: Over the next few professional learning sessions we will Build our understanding of performance levels, indicators, and descriptors Calibrate our ratings Calibrate our feedback with a structured framework Consider the impact of this work on your own evaluation (C1 on the TEAM Administrator rubric). We will be walking through this session as learners—active participants—so your engagement will be key for allowing you to construct new learning. *Note to facilitator—this session is designed to allow the learner to construct his/her own learning, so there will be several points of self-assessment and reflection vs. telling/instructing. You will be facilitating learning, not giving a lecture. TEAM Professional Learning Package Inter-Rater Reliability Module 1: Accurate Ratings Learning Session 1: Understanding Performance Levels, Indicators, and Descriptors

Module 1: Accurate Ratings Learning Session 1: Understanding Performance Levels, Indicators, and Descriptors Learning Session 2: Calibration of Ratings Say: In this module, we will start by revisiting the performance levels, indicators, and descriptors to deepen understanding of instructional practice and how it should be rated. Then we will engage in a calibration activity. Calibrating is the process of using a single set of rating criteria among multiple raters. If calibrating is successful, a particular body of evidence should receive the same rating regardless of who rated it. Without it, classroom observations will not yield reliable data. ”   However, before we can assign ratings, we need to look at the evidence collected and ask ourselves, “What is the learning goal for students by the end of this unit? What do the standard(s) call for the students to know and be able to do? What was the expectation for students in this lesson in relation to the standard(s)? Did all students meet the expectation? HOW do I know?” Keeping this information front and center will keep you focused on what important as you evaluate the evidence for each indicator.

Guiding Questions for Learning Session 1 How can my deepened understanding of performance levels, indicators, and descriptors increase the accuracy of observation ratings? How can increased accuracy of observation ratings impact teaching and learning? Say: Take a moment and read our guiding questions to yourself. [Pause for reading time.] Which of these is of greater interest to you? Take 90 seconds and share both your selection and why you chose it with a shoulder partner. *Note to facilitator—group share out is not necessary.

Learning Session 1: Understanding Performance Levels, Indicators, and Descriptors Say: The foundation of strong evaluations is a deep understanding of performance levels, TEAM indicators, and TEAM descriptors.

Understanding the Performance Levels Access the Performance Levels document. Individually, read the performance levels and highlight key phrases. With a partner or in a small group, compare highlights, identifying similarities and differences. Share with the larger group. Say: Our first step in this process is to strengthen our understanding of performance levels. Performance levels make up the numerical scale on which we base observation ratings. They allow us to differentiate between practices that are at expectations, approaching, or exceeding expectations. *Note to facilitator—if an all digital session, be sure participants have the digital materials accessible. If paper, be sure they can easily identify the Performance Levels document. Consider color coding the documents for easy identification. Say: First, access the Performance Levels document. Next, you will read the performance levels and highlight key phrases, individually. Then, you will compare your highlights with a partner and discuss differences and similarities. Finally, as a table, share your thoughts. We will take approximately 6 minutes to do this. Say: What questions might you have? [pause] Please begin. *Note to facilitator—Once participants have shared, you will reveal the next slide.

Understanding the Performance Levels Performance Level Guide 1 Significantly Below Expectations: A teacher at this level has limited knowledge of the instructional skills, knowledge, and responsibilities described in the rubric and struggles to implement them. He/she has little to no impact on student outcomes. 2 Below Expectations: A teacher at this level demonstrates some knowledge of the instructional skills, knowledge, and responsibilities described in the rubric but implements them inconsistently. His/her impact on student outcomes is below expectations. 3 At Expectations: A teacher at this level understands and implements most of the instructional skills, knowledge, and responsibilities described in the rubric. His/her impact on student outcomes is meeting expectations. 4 Above Expectations: A teacher at this level comprehends the instructional skills, knowledge, and responsibilities described in the rubric and implements them skillfully and consistently. He/she makes a strong impact on student outcomes. 5 Significantly Above Expectations: A teacher at this level exemplifies the instructional skills, knowledge, and responsibilities described in the rubric and implements them adeptly and without fail. He/she meets ambitious teaching and learning goals and makes a significant impact on student outcomes. Performance at this level should be considered a model of exemplary teaching. Say: Which words did you find most important as you considered the descriptor for each performance level? *Note to facilitator—accept all answers and encourage responses. Do not agree or disagree, simply accept. (Consider simply saying “thank you” to each response. Next slide reveals the “right” answers and participants will self-assess.)

Understanding the Performance Levels Performance Level Guide 1 Significantly Below Expectations: A teacher at this level has limited knowledge of the instructional skills, knowledge, and responsibilities described in the rubric and struggles to implement them. He/she has little to no impact on student outcomes. 2 Below Expectations: A teacher at this level demonstrates some knowledge of the instructional skills, knowledge, and responsibilities described in the rubric but implements them inconsistently. His/her impact on student outcomes is below expectations. 3 At Expectations: A teacher at this level understands and implements most of the instructional skills, knowledge, and responsibilities described in the rubric. His/her impact on student outcomes is meeting expectations. 4 Above Expectations: A teacher at this level comprehends the instructional skills, knowledge, and responsibilities described in the rubric and implements them skillfully and consistently. He/she makes a strong impact on student outcomes. 5 Significantly Above Expectations: A teacher at this level exemplifies the instructional skills, knowledge, and responsibilities described in the rubric and implements them adeptly and without fail. He/she meets ambitious teaching and learning goals and makes a significant impact on student outcomes. Performance at this level should be considered a model of exemplary teaching. Say: As you have identified, there are some key terms that help define each performance level. [Use the script below to quickly point out some key differences. Allow leaders time to “check” their answers.] Say: At level one you see limited knowledge struggles little to no impact on student achievement Say: At level two, you might see Inconsistency and outcomes below expectations

Understanding the Performance Levels What takeaways might you be having about the performance levels? How might your thinking about the rating scale be changing? How might this new understanding change your approach to observation? Say: Read the questions on the screen. [pause] Say: Take two minutes and share your thoughts with your shoulder partner. *Note to facilitator—group share out is not necessary.

Understanding the Indicators and Descriptors Say: Our next step is to look closely at the descriptors on the TEAM general educator rubric. *Note to facilitator—if an all digital session, be sure participants have the digital materials accessible. If paper, be sure they can easily identify the TEAM Teacher Evaluator Handbook. Consider color coding for easy identification.

Understanding the Indicators and Descriptors: Standards & Objectives Step 1 Individually read the descriptors at performance level 5. With a partner or in a small group, discuss the importance of each descriptor and how they work together to define the indicator. Share with the larger group. *Note to facilitator—the purpose of this activity is help participants understand what makes the indicator critical to student success. There may be descriptors that are particularly important that should be discussed. Encourage participants to think about the indicator holistically as well. *Note to facilitator—determine grouping arrangement before-hand. Consider using observation data to configure pairs or small groups. . *Note to facilitator—at this point, the responses may be a simple restatement of descriptors. You can ask probing questions to have the group describe their thinking. More discovery is forthcoming, so it’s ok not to push too hard. Say: We are going to look at this indicator and descriptors through the lens of performance level 5. As we discussed during the performance level session, practices at this level exemplify the instructional skills, knowledge, and responsibilities described in the rubric and they are implemented adeptly and without fail in the service of ambitious teaching and learning that leads to a significant impact on student outcomes. It should be considered a model of exemplary teaching. For example, the first descriptor states that all learning objectives are clearly and explicitly communicated, connected to state standard(s), and referenced throughout the lesson. Is that practice critical in order to have a significant impact on student outcomes? Why is it critical?

Understanding the Indicators and Descriptors: Standards & Objectives Step 2: With a partner or in a small group, differentiate between the performance levels by highlighting and considering the unique elements in each. *Note to facilitator—this slide is animated to display the rubric upon a click. Do not reveal the rubric until participants have marked their rubrics. *Note to facilitator—the purpose of this activity is not only to help participants differentiate between the levels. By making connections to the performance level descriptors, the connection should be made that the impact on student outcomes is paramount in assigning a rating. After the activity, say: What does it mean when something is clearly and explicitly communicated rather than just communicated? How would that difference look and sound? How does the learning change when objectives are known from life experiences and integrated with other disciplines? How does the learning change when expectations for student performance are vague? How can a teacher be successful promoting student learning when there is little evidence of mastery gathered from students? Thinking about the performance level descriptors, if the expectations for student performance are vague, would we expect to see students meeting those expectations? Would that practice lead to a significant impact on student achievement? What does this tell you about collecting evidence and rating practice? *Note to facilitator—if needed, prompt participants to make a connection between the performance levels and student outcomes.

Understanding the Indicators and Descriptors: Standards & Objectives Step 3 Individually read the TEAM handbook for this indicator which describes level 5 performance With a partner or in a small group, continue discussing the purpose of this indicator and how this information can help you differentiate level 5, level 3, and level 1 performance. Share with the larger group. *Note to facilitator—encourage as much depth as possible in these responses. Say: Now we are going to look at this indicator and descriptors more deeply by reading about performance at level 5 in the TEAM teacher evaluation handbook. Continue to think about the purpose of the indicator as well as how the information can assist in differentiating between level 5, level 3, and level 1 performance. *Note to facilitator—if needed as part of the debrief, say the summary below. The standards and objectives indictor is strongly tied to effective communication. In a teacher’s planning, the learning must be clearly crafted so they can be clearly communicated to students. When teachers connect learning to what students already know from their prior learning or life experiences, it becomes more clear to students. Also, the teacher collects evidence of mastery from the students so that he/she clearly understands student learning needs. This indicator is as much about students clearly communicating their understanding as it is about teachers clearly communicating their expectations.

Understanding the Indicators and Descriptors: Standards & Objectives Step 4: With a partner or in a small group, discuss the evidence that might be collected from observing students when performance is at level 5, at level 3, and at level 1. Share with the larger group. *Note to facilitator—the purpose of this activity is to ensure observers understand that they are evaluating the impact of teacher performance on student learning. Say: Because a teacher’s primary professional responsibility is to ensure that students learn, I want to reiterate that observation of student learning should play a predominant role in the evaluation of teacher performance. Through focused, rigorous observation of classroom practice and student performance as a result of those practices, it is possible to accurately distinguish exemplary practice from good practice from ineffective practice.

Understanding the Indicators and Descriptors: Standards & Objectives Step 5: With a partner or in a small group, chart 1-3 critical elements to keep in mind when determining a rating for this indicator. Share with the larger group. *Note to facilitator—the purpose of this activity is to solidify learning and establish consistent practices among observers. Have chart paper available for this activity. A verbal share out might be replaced with a gallery walk and debrief.

Understanding the Indicators and Descriptors: Standards & Objectives With a partner or in a small group, discussing the following: How is your understanding of the indicator changing? About which descriptor might you have most changed your thinking? How will this learning enhance your ability to provide more meaningful feedback to teachers? How might this new understanding change your approach to rating practice? How will your learning about this indicator shape the way you look at other indicators? *Note to facilitator—after this reflection, you might do the following: -repeat the 5 step process with other descriptors, particularly those are most critical based on local needs

Learning to Application *Note to facilitator—if moving to learning session 2 at this time, this learning to application may be omitted. Guidance for selecting a video can be found in package guidance document Say: This learning session closes with an activity titled, “Learning to Application.” It is designed to help everyone build on the knowledge they have developed during this learning session and prepare for the next one.”

Learning to Application Prior to our next learning session: Watch and script the assigned lesson video Assign a rating to the assigned indicators Craft a rational to support the rating