The Basic Components of Inter-Rater Reliability ODE SUMMIT = SIT IN RUBRIC-ALIKE GROUPS HANDOUT: Rubric Language Evidence Examples Collecting Observation Evidence IRR-Calibration Plan Supplies: Chart paper Markers Post-its Highlighters FOR ODE SUMMIT: Have extra copies of rubrics Danielson Marshall LEGENDS

Objectives Develop strong understanding of rubric language and performance levels Strengthen observation skills and explore importance of frequent observation Calibrating the collection of evidence through multiple measures Rate the level of performance and provide consistent feedback that builds self-directed learners 2 min Minimum IRR training should encompass 30-hours (this is akin to learning anything new)

Vision of Effectiveness Think of when you observed an excellent teacher in the classroom or a professional context. What did you see or hear that made you think that you were observing an effective practitioner? In a perfect world, what is your vision of educator effectiveness? Slides 3-6 = 15 min

Traits of Effectiveness Write two pieces of evidence of an effective practitioner on two post-its (one piece of evidence per post-it) As a table group sort your post-its into like categories Agree on a label for each category Slides 3-6 = 15 min Slides 3-4 = 5 min

Traits of Effectiveness Write two pieces of evidence of an effective practitioner on two post-its (one piece of evidence per post-it) As a table group sort your post-its into like categories Agree on a label for each category Label:___ Label:___ Label:___ Label:___ Slides 3-6 = 15 min This slide = 5 min

Traits of Effectiveness What happens if we add the four Domains of the InTASC Standards? Do you need to re-sort your post-its? The Learner Content Instructional Professional & Learning Practices Responsibilities Slides 3-6 = 15 min This slide = 5 min When you do this at home use the domains/categories specific to your rubric

Understanding the Rubric Set up Danielson Domains  Components Elements Evidence LEGENDS Standards Performance Indicators Marshall Criteria 3 min Localize! Teach the structure of your rubric – how is it set-up? How is it a resource for observers of practice? How is it a resource for teachers in their professional development?

Level of Performance Level 1- Does not meet standards; performs below the expectations for good performance under this standard; requires direct intervention and support to improve practice Level 2- Making sufficient progress toward meeting this standard; meets expectations for good performance most of the time and shows the continuous improvement; expected improvement through focused professional learning and growth plan 5 min

Level of Performance Level 3- Consistently meets expectations for good performance under the standard; demonstrates effective practices and impact on student learning; continues to improve professional practice through ongoing professional learning Level 4- Consistently exceeds expectations for good performance under this standard; demonstrates highly effective practices and impact on student learning; continued expansion of expertise through professional learning and leadership opportunities 5 min

Where do you live? It is expected that we strive and are supported to live at a Level 3 and visit Levels 2 & 4 You’re going to move between levels within your year and career! 3 min This may be a cultural shift for teachers and principals. Level “3” is where you really consistent control the practice in that standard – “4” may represent areas of passion or areas where we really shine, and “2” In our rubric, we all have a standard that maybe everyone in Oregon is learning and may be a “2” currently (like ones addressing teachers knowing content standards and CCSS is brand new to most of us!).

For Example: Designing Student Assessments 5-min Draw attention to 1F and talk about this one in particular and the creation of “ok to be at level 2”   Show them where the language supports their “newness” and growth and “allow” them to mark themselves basic knowing the plan to move towards proficient. Create the culture of “We got you (support you)” vs. “Gotcha!”

Differentiated Language Levels of Performance Using your rubric, pick a component, standard or criteria Highlight key words that show the difference between levels of practice Discuss and record responses on the organizer provided Share examples of evidence from your practice that matches the language you identified at each level HANDOUT: Need blank table of levels of performance for common language Slide 12 = 5 min The difference between a “1” and a “4” is vastly apparent; it is the difference between the language in “2” and “3” that is really important to come to a firm and calibrated understanding of prior to engaging in gathering evidence toward the rubric. This is the difference between a proficient practitioner and one that needs more job-embedded supports to grow, or a place that a practitioner will need to focus on for their professional growth goals, etc. Do this with as many components as you need to identify the patterns of differentiated language

Common Language Level 1 Level 2 Level 3 Level 4 No Never Not Clear Unaware Does Not Poor Little Low Inadequate Uses Few Some Attempts Limited Moderate Uneven Inconsistent Rarely Developing Partially Consistent High Quality Timely Accurate Appropriate Clear Effective Aligned Almost Always All Students Highly Effective Adapted For Always Fully Aligned Extensive Highly Developed 3 min You may have also noted shifts in practice: Example – from Danielson, as you go from left to right, the practitioner starts to have a more student-led style of teaching and learning

Identifying Evidence Read the evidence statements provided Based on your rubric, which component, standard or criteria is it evidence of? Which statements are evidence of observable practice? Non-observable? If non-observable, where would this evidence be found? What are examples of evidence from your practice that support components, standards and criteria that aren’t listed? List a few examples of your own in the blank spaces. Note: Levels of performance are not determined until multiple sources of evidence are collected. 15 min HANDOUT: Examples of Evidence Work with an elbow partner to fill-out the Examples of Evidence hand-out by following the prompts on the slides When your back at your district, cross score with folks from a different working table or other partner – did they sort the evidence into the same categories? Did they think it was observable or non-observable? Where did they think they could find the non-observable evidence?

Fact vs. Opinion When Collecting Evidence Table Talk: Think about a time that you received an evaluation based on both factual evidence and opinion. What were the differences in…. How you felt? The impact on your professional growth? The relationship with your evaluator? Why is it important to collect multiple pieces of factual evidence before trying to assess educator practice? How can factual evidence support educator development? 3 min Have volunteers share out

Which is Stronger? Why? Time Educator Actions Student Actions Time 1:01 1:03 Educator asked kids to stand and led them in “The Days of the Week” song. Educator asked “What day comes after Saturday?” 16 of the 27 kids stood up for the song. Steven shouted out, “Monday!” Most students laughed-2 boys physically rolled around and knocked over 2 girls. Steven walked away from the group, and sat in the opposite corner of the classroom 3 min The top one is stronger it gives direct examples with details. The teacher is going to be able to reflect more on this type of evidence. Have participants talk about why. Chart the differences/reasons. Time Educator Actions Student Actions 1:01 1:03 Educator reviewed the days of the week Educator asked the question about the days of the week Many students were not listening. Steven called out over and over again

Effective Observation Practices Eliminate effects of bias. Enter the classroom without judgment and work from evidence. Collect factual evidence. Write down only what teachers and students say and do. Look for evidence of learning. Remain, review, reflect. Pause to organize your evidence before aligning. Only rate after multiple sources of evidence are collected. Frequent, focused, varied with timely feedback. 3 min Eliminating bias is a training in and of itself, but it is of the utmost importance for a valid and reliable observations and evaluations.

Collecting Evidence What do you see and hear the teacher and students doing? What evidence can you gather? Time Educator Actions Student Actions HANDOUT: Collecting Evidence 1 min

Selecting Observation Focus When taking notes, come up with a short-hand system. Don’t worry if you miss something – practice makes perfect. Sample of how you could scaffold support for focusing your first 3 observations Cluster1: Managing classroom procedures, Managing student behavior, and Organizing physical space Cluster 2: Creating environment of respect & rapport, Establishing a culture for learning, Communicating with students Cluster 3: Using questioning and discussion techniques, Engaging students in learning, Using assessment in instruction Count off at your table by “3’s” – be ready to gather evidence on the video based on your cluster 3 min

Video 1 Ms. Warburton’s 8th grade math lesson: Sorting and classifying equations CCSS Math 8.EE.C.7a Collect evidence statements on your cluster https://www.teachingchannel.org/videos/sorting- classifying-equations-overview This slide = 12-mim Be sure that you are sitting with a like rubric group

Video 1 (follow up) Compare your evidence statements with a cluster partner With partner, use your rubric and practice aligning/coding evidence statements with components/standards/criteria language. Make sure each item shared is factual evidence Each cluster shares out at their table One piece of evidence from each cluster In whole group, each cluster will share out one piece of evidence. Add to your notes any factual evidence your table partners shared that you did not have written down This slide = 15 min AT ODE SUMMIT: For sake of training only do 2 at each table plus statement we provide for calibration

Calibrating Let’s all look at evidence that aligns with: At tables: Based on the evidence we have compiled, talk about the level of performance should be assigned for each component. On the following slides use your clicker to report the ranking for each of the three cluster areas we collected evidence for Cluster Danielson Marshall LEGENDS Cluster 1 2c Bh 3.3 This slide = 30 min First bullet point: Let’s all look at evidence that aligns with ____________. You may need to give participants more factual evidence statement (hopefully one that they have already) (Or rove and find a common one being talked about). Give participants 1-2 minutes to do this, then share out and chart Second bullet point: Have tables agree on and assign level of performance for given evidence statement, then have tables share out. Did we get the same level? Talk about the fact that you would never assign a level of performance after only one observation – you need multiple observations and evidence to do that(the importance of only using one piece during IRR training). 80% agreement prior to moving on (and the disagreement should not be more than one-level off).

What is the performance level for Cluster 1: 2c, Bh, and 3.3? 4 Expert Score:

Calibrating Let’s all look at evidence that aligns with: At tables: Based on the evidence we have compiled, talk about the level of performance should be assigned for each component. On the following slides use your clicker to report the ranking for each of the three cluster areas we collected evidence for Cluster Danielson Marshall LEGENDS Cluster 2 2a Bd 3.2 This slide = 30 min First bullet point: Let’s all look at evidence that aligns with ____________. You may need to give participants more factual evidence statement (hopefully one that they have already) (Or rove and find a common one being talked about). Give participants 1-2 minutes to do this, then share out and chart Second bullet point: Have tables agree on and assign level of performance for given evidence statement, then have tables share out. Did we get the same level? Talk about the fact that you would never assign a level of performance after only one observation – you need multiple observations and evidence to do that(the importance of only using one piece during IRR training). 80% agreement prior to moving on (and the disagreement should not be more than one-level off).

What is the performance level for Cluster 2: 2a, Bd, and 3.2? 1 2 3 4 Expert Score:

Calibrating Let’s all look at evidence that aligns with: At tables: Based on the evidence we have compiled, talk about the level of performance should be assigned for each component. On the following slides use your clicker to report the ranking for each of the three cluster areas we collected evidence for Cluster Danielson Marshall LEGENDS Cluster 3 3c Cg 5.2 This slide = 30 min First bullet point: Let’s all look at evidence that aligns with ____________. You may need to give participants more factual evidence statement (hopefully one that they have already) (Or rove and find a common one being talked about). Give participants 1-2 minutes to do this, then share out and chart Second bullet point: Have tables agree on and assign level of performance for given evidence statement, then have tables share out. Did we get the same level? Talk about the fact that you would never assign a level of performance after only one observation – you need multiple observations and evidence to do that(the importance of only using one piece during IRR training). 80% agreement prior to moving on (and the disagreement should not be more than one-level off).

What is the performance level for Cluster 3: 3c, Cg, and 5.2? 1 2 3 4 Expert Score:

Effective Feedback Preparing Sharing Align evidence to the rubric. Code evidence to the rubric and determine the level of performance for each component observed. 2. Prioritize. Select one or two parts of the teacher’s practice on which to focus. Sharing Invite dialogue. Welcome and value teacher input. 2. Refer to evidence. Discuss factual evidence from the observation and invite dialogue on the evidence or its alignment to the rubric. 3. Identify development areas. Define specific areas for development and invite the teachers to comment on those areas or to suggest additional areas. 4. Plan concrete action. Together, develop clear and measurable next steps, including resources and a plan for support. 5. Set a timeline. Agree on a plan for implementation and follow-up. 5 min Back at home, write feedback for practice or know that this is the next step in your

Planning for IRR Implementation Plan - Best practice = 30-hours minimum Rubric: Understanding Rubric Format Levels of Performance Language of Instruction Evidence: Multiple Measures Observable & Non-Observable Aligning/Coding to Rubric Fact vs. Opinion Awareness of Bias __ min ODE SUMMIT: Plan for Implementation

Planning for IRR Implementation Practice Observations: Observe Practice (Frequent, Focused, Varied) Organize Evidence (Remain, Review, Reflect) Aligning/Coding Evidence to Rubric Rate Performance 80% rule Effective Feedback Preparing / Sharing model What is the plan to come together multiple times to re-calibrate throughout the year(s)? How will you assess the effectiveness of this IRR/Calibration plan? __ min ODE SUMMIT: Plan for Implementation

Video 2 Ms. Bannon’s 3rd grade reading lesson: Understanding main idea CCSS ELA RL.3.2 & ELA.SL.3.2 Collect evidence statements on your cluster https://www.teachingchannel.org/videos/3rd-grade- ela-lesson Slides 25-27 = 35 min Slide 25 = 12-min WILL NOT DO AT ODE SUMMIT

Video 2 (follow up) Compare your evidence statements with a cluster partner With partner, use your rubric and practice coding evidence statements with components/standards/criteria language. Make sure each item shared is factual evidence Each cluster shares out at their table One piece of evidence from each cluster In whole group, each cluster will share out one piece of evidence. Add to your notes any factual evidence your table partners shared that you did not have written down Slides 25-27 = 35 min Slide 26 = 15-min WILL NOT DO AT ODE SUMMIT

Calibrating Let’s all look at evidence that aligns with: At tables: Based on the evidence we have compiled, talk about the level of performance should be assigned for each component. On the following slides use your clicker to report the ranking for each of the three cluster areas we collected evidence for Cluster Danielson Marshall LEGENDS Cluster 1 3e Aj 3.1 This slide = 30 min First bullet point: Let’s all look at evidence that aligns with ____________. You may need to give participants more factual evidence statement (hopefully one that they have already) (Or rove and find a common one being talked about). Give participants 1-2 minutes to do this, then share out and chart Second bullet point: Have tables agree on and assign level of performance for given evidence statement, then have tables share out. Did we get the same level? Talk about the fact that you would never assign a level of performance after only one observation – you need multiple observations and evidence to do that(the importance of only using one piece during IRR training). 80% agreement prior to moving on (and the disagreement should not be more than one-level off).

What is the performance level for Cluster 1: 3e, Aj, and 3.1? 2 3 4

Calibrating Let’s all look at evidence that aligns with: At tables: Based on the evidence we have compiled, talk about the level of performance should be assigned for each component. On the following slides use your clicker to report the ranking for each of the three cluster areas we collected evidence for Cluster Danielson Marshall LEGENDS Cluster 2 3a Cc 3.2 This slide = 30 min First bullet point: Let’s all look at evidence that aligns with ____________. You may need to give participants more factual evidence statement (hopefully one that they have already) (Or rove and find a common one being talked about). Give participants 1-2 minutes to do this, then share out and chart Second bullet point: Have tables agree on and assign level of performance for given evidence statement, then have tables share out. Did we get the same level? Talk about the fact that you would never assign a level of performance after only one observation – you need multiple observations and evidence to do that(the importance of only using one piece during IRR training). 80% agreement prior to moving on (and the disagreement should not be more than one-level off).

What is the performance level for Cluster 2: 3a, Cc, and 3.2? 1 2 3 4

Calibrating Let’s all look at evidence that aligns with: At tables: Based on the evidence we have compiled, talk about the level of performance should be assigned for each component. On the following slides use your clicker to report the ranking for each of the three cluster areas we collected evidence for Cluster Danielson Marshall LEGENDS Cluster 3 3b Cg 8.4 This slide = 30 min First bullet point: Let’s all look at evidence that aligns with ____________. You may need to give participants more factual evidence statement (hopefully one that they have already) (Or rove and find a common one being talked about). Give participants 1-2 minutes to do this, then share out and chart Second bullet point: Have tables agree on and assign level of performance for given evidence statement, then have tables share out. Did we get the same level? Talk about the fact that you would never assign a level of performance after only one observation – you need multiple observations and evidence to do that(the importance of only using one piece during IRR training). 80% agreement prior to moving on (and the disagreement should not be more than one-level off).

What is the performance level for Cluster 3: 3b,__, and__? 1 2 3 4

Wrapping Up First of many professional learning opportunities Bend Summit – October 22nd CCSS Regional Series Pendleton – October 29 & 29 Wilsonville – November 4 & 5 Redmond – November 7 & 8 Resources SLG Goal Guidance and samples Updated Framework, FAQs, “Who is evaluated?” Toolkit Implementation Web Page www.ode.state.or.us/search/page/?id=3904 Today’s summit is just the beginning of the professional learning opportunities that will be available. COSA’s regional series “Common Core in Action” will have fall and spring dates. Where the connections between CCSS and Educator Effectiveness will be explored more fully. Implementation Resources page of the toolkit is where to find all the resources we have talked about today, including the template and sample SLG goals

Thank you for attending today! Providing Feedback Summit Evaluation General Questions/Feedback ode.evaluation@state.or.us Thank you for attending today! Please take a moment to complete the summit evaluation. We will use this information to inform the Bend session. We’ve tried to answer throughout the session today, but if any additional questions surface, or if you want to give feedback on the SLG Goals guidance document, please email ode.evaluation@state.or.us This email address will be monitored and questions answered either through direct feedback or through updated FAQs. Thank you for attending!