2. Copyright © Tulsa Public Schools 2011 © 2012, Tulsa Public Schools Teacher and Leader Effectiveness (TLE) Observation and Evaluation System / Process EVALUATOR TRAINING – 2012 – Segment 5 Getting the Rankings Right

3. 2 Calibration Sensation!! Getting the Scores Right

4. Importance of Each and Every Training Piece… 3 Each piece of the training represents an EQUALLY important component of the TLE system and process. One missing component can derail the process.

5. TLE Evaluation System Observations Evaluations Conferences Feedback / Support 4

6. Rubrics and the Horse’s Tale Which direction is the horse’s head facing? 5

7. The Goal of Rater Accuracy To move from subjectivity to higher levels of objectivity and insight. How? Practice, discipline and intentional study 6

8. Warm Up Question Why is it important that raters be calibrated correctly with regard to the rankings/expectations in the rubric? 7

9. Calibration Exercise Basics Focusing (separately) on indicators 1, 6, 7, 11, 12 & 15, we will… – reflect upon the requirements of the indicator – watch a short clip of a teacher performance – individually rank the performance – discuss our rankings, attempting to reach agreement on a score – compare our score against the target score 8

10. Context of Calibration Only one performance/observation – Disadvantages: not exactly comparable to how principals evaluate teachers (i.e., doesn’t allow us the opportunity to draw upon multiple observations and our history/personal knowledge of the teacher) – Advantages: forces us to critically examine the rubric’s expectations and our understanding of those expectations 9

11. Determining an Indicator’s Rating Each Indicator often has several definitional narratives for each level of proficiency. However, Principals must enter only one (1) score as to each Indicator (1, 2, 3, 4, 5, NO, NA). 10 3

12. Determining an Indicator’s Rating “Best fit” approach 11

13. 12 Indicator 1

14. Indicator 1— Planning and Preparation at a “3” Plans for instructional strategies that encourage the development of performance skills. Ensures materials and equipment are ready at the start of the lesson or instructional activity (most of the time). 13

15. Planning and Preparation: What Do you Expect… teachers to be doing/saying if they are performing at an “effective” level… students to be doing/saying if teachers are performing at an “effective” level… 14

16. 15 Indicator 1… Ready to Cue Music?

17. What Level of Planning and Preparation is This? (Clip 1, Indicator 1) 16 Click on the link to view: Clip 1 - Indicator 1--revo, reaction, reform.wmv

18. What Level of Planning and Preparation is This? (Clip 1, Indicator 1) If the video doesn’t play for some reason, you should be able to access it another way: Click on this link to spawn clip from folder: Clip 1 - Indicator 1--revo, reaction, reform.wmv FYI, the video is also accessible on the Teaching Channel website at : www.teachingchannel.org/videos/3-rs-revolution-reaction-and- reform?resume=0 www.teachingchannel.org/videos/3-rs-revolution-reaction-and- reform?resume=0 (Revolution, Reform, Reaction) (Clip is from 0:40 – to 9:59) 17

19. What do you think? Study the rubric for Indicator 1 and write down your score for this Indicator relying on the language in the rubric. Show your score to the table. What was the most frequent (“modal”) score? Discuss our modal scores Is there a consensus for the room after our discussion? 18

20. Target score…. Clip 1, Indicator 1 5 - Superior 19

21. 20 Indicator 15

22. Indicator 15— Closure at a “3” Ends the day’s learning / activity by summarizing the lesson and or asking students to summarize the lesson. Connects what is learned to prior learning. 21

23. Indicator 15: What Do you Expect… teachers to be doing/saying if they are performing at an “effective” level… students to be doing/saying if teachers are performing at an “effective” level… 22

24. 23 Indicator 15… Ready to Cue Music?

25. What Level of Closure is this? 24 Click on the link to view: Clip 2_Indicator 15--absent kid.wmv

26. What Level of Closure is this? If the video doesn’t play for some reason, you should be able to access it another way: Click on this link to spawn clip from folder: Clip 2_Indicator 15--absent kid.wmv FYI, the video is also accessible on the Teaching Channel website at : http://www.teachingchannel.org/videos/mr-mccloud- discovering-surface-area-of-a-cylinder?fd=0# http://www.teachingchannel.org/videos/mr-mccloud- discovering-surface-area-of-a-cylinder?fd=0# (Mr. McCloud—area of cylinder) (from 24:47- 26:53) 25

27. What do you think? Study the rubric for Indicator 15 and write down your score for this Indicator relying on the language in the rubric. Show your score to the table. What was the most frequent score? Is there a consensus for the room after our discussion? 26

28. Target score…. Clip 2, Indicator 15 2 or 3* – Needs Improvement or Effective * depending on whether he confirmed that the students had actually summarized the lesson. (We can’t tell for sure from the clip whether the true summary of the lesson was in the students’ written work.) 27

29. 28 Indicator 7

30. Indicator 7— Embedding Literacy at a “3” Literacy, the practice of reading, writing, spelling, listening, and speaking, is embedded in ALL content as an explicit learning objective. Displays basic recognition of the importance of literacy as the “bonding agent” for all learning. 29

31. Indicator 7: What Do you Expect… teachers to be doing/saying if they are performing at an “effective” level… students to be doing/saying if teachers are performing at an “effective” level… 30

32. 31 Indicator 7… Ready to Cue Music?

33. What Level of Embedded Literacy is This? (Clip 3, Indicator 7) 32 Click on the link to view: Clip 3-Indicator 7--math manipulatives.wmv

34. What Level of Embedded Literacy is This? (Clip 3, Indicator 7) If the video doesn’t play for some reason, you should be able to access it another way: Click on this link to spawn clip from folder: Clip 3-Indicator 7--math manipulatives.wmv FYI, the video is also accessible on the Teaching Channel website at : http://www.teachingchannel.org/videos/what-fraction-of-this-shape-is- red?fd=0 (fractions/geometry using manipulatives ) (from 0:23 to end) 33

35. What do you think? Study the rubric for Indicator 7 and write down your score for this Indicator relying on the language in the rubric. Show your score to the table. What was the most frequent score? Is there a consensus for the room after our discussion? 34

36. Target score…. Clip 3, Indicator 7 4 – Highly Effective 35

37. 36 Indicator 11

38. Indicator 11—Directions at a “3” Provides directions and procedures, in a variety of delivery modes, e.g., verbal, modeling, visual, demonstration, etc., that are clearly stated / presented and relate to the learning objectives. Gives students directions for transitions and includes transitioning in the planning process to optimize academic learning time. Uses spoken and written language that is clear and correct, conforms to standard English, vocabulary, and is appropriate to students’ ages and interests. 37

39. Indicator 11: What Do you Expect… teachers to be doing/saying if they are performing at an “effective” level… students to be doing/saying if teachers are performing at an “effective” level… 38

40. 39 Indicator 11… Ready to Cue Music?

41. What Level of Directions is This? (Clip 4, Indicator 11) 40 Click on the link to view: Clip 4-Indicator 11--evaporation.wmv

42. If the video doesn’t play for some reason, you should be able to access it another way: Click on this link to spawn clip from folder: Clip 4-Indicator 11--evaporation.wmv FYI, the video is also accessible on the Teaching Channel website at : https://www.teachingchannel.org/videos/content-differentiation- science?fd=1https://www.teachingchannel.org/videos/content-differentiation- science?fd=1. (Evaporation) (20:25 - 24:00) 41 What Level of Directions is This? (Clip 4, Indicator 11)

43. What do you think? ( Clip 4, Indicator 11 ) Study the rubric for Indicator 11 and write down your scores for this Indicator relying on the language in the rubric. Show your score to the table. What was the most frequent score? Is there a consensus for the room after our discussion? 42

44. Target score…. Clip 4, Indicator 11 2 –Needs Improvement 43

45. 44 Indicator 6

46. Indicator 6—Teacher’s Interactions with and Expectations of her Students at a “3” Oral, written and nonverbal communications with students are considerate and respectful. Consistently conveys a generally positive view of learning and of the curriculum, demonstrating high expectations for most students. 45

47. Indicator 6: What Do you Expect… teachers to be doing/saying if they are performing at an “effective” level… students to be doing/saying if teachers are performing at an “effective” level… 46

48. 47 Indicator 6… Ready to Cue Music?

49. What Level of Interactions and Expectations Is This? (Clip 5, Indicator 6) 48 Click on the link to view: Clip 5- Indicator 6--star of the week.wmv

50. What Level of Interactions and Expectations is This? (Clip 5, Indicator 6) If the video doesn’t play for some reason, you should be able to access it another way: Click on this link to spawn clip from folder: Clip 5- Indicator 6--star of the week.wmv FYI, the video is also accessible on the Teaching Channel website at : www.teachingchannel.org/videos/our-selves-our-classroom-our- families?fd=1 (Star of the Week) (from 1:00 – 6:13) 49

51. What do you think? Study the rubric for Indicator 6 and write down your score for this Indicator relying on the language in the rubric. Show your score to the table. What was the most frequent score? Is there a consensus for the room after our discussion? 50

52. Target score…. Clip 5, Indicator 6 5 - Superior 51

53. 52 Indicator 12

54. Indicator 12 —Modeling at a “3” Provides demonstrations and modeling of the desired skill or process that are clear and precise to students. 53

55. Indicator 12: What Do you Expect… teachers to be doing/saying if they are performing at an “effective” level… students to be doing/saying if teachers are performing at an “effective” level… 54

56. 55 Indicator 12… Ready to Cue Music?

57. What do you think? (Clip 6, Indicator 12) 56 Click on the link to view: Clip 6-Indicator 12--Climate Change in a bottle.wmv

58. If the video doesn’t play for some reason, you should be able to access it another way: Click on this link to spawn clip from folder: Clip 6-Indicator 12--Climate Change in a bottle.wmv Clip 6-Indicator 12--Climate Change in a bottle.wmv FYI, the video is also accessible on the Teaching Channel website at : https://www.teachingchannel.org/videos/climate-change-lesson-3 (Climate Change in a Bottle) (from 0:30 – 5:22) 57 What do you think? (Clip 6, Indicator 12)

59. What do you think? Study the rubric for Indicator 12 and write down your score for this Indicator relying on the language in the rubric. Show your score to the table. What was the most frequent score? Is there a consensus for the room after our discussion? 58

60. 4 – Highly Effective 59 Target score…. Clip 6, Indicator 12

61. Our Closure: Interview a Friend! 60 Pick one of the following questions – What behaviors and processes support accurate calibration? – What did this exercise reveal about my expectations? Was I calibrated a little low/high? Just right? Ideas? – How might evaluators deconstruct the words of the rubric with videos, or other materials, with their teachers to ensure that have agreement in their expectations?

62. Final Notes How does the Tulsa model and its processes support accurate calibration? 61

63. Questions and Next Segment… The 6 th Segment of Evaluator Information File: PP6 Modeling & Presentation Topic: Modeling / Presentations 62

64. For more information: www.tulsaschools.org Office of Teacher and Leader Effectiveness 918-746-6800 Tulsa Framework: http://sde.state.ok.us; left column: Teacher and Leader Effectiveness; Scroll to Frameworks; Tulsa Frameworkhttp://sde.state.ok.us Talia Shaull – shaulta@tulsaschools.org Gene S. Kleindienst - kleinge@tulsaschools.org Jana Burk – burkja@tulsaschools.org 63