1. 1 You’re the Judge Now…Getting the Scores Right Calibration Training, continued

2. Determining an Indicator’s Rating Each Indicator often has several definitional narratives for each level of proficiency. However, Principals must enter only one (1) score as to each Indicator (1, 2, 3, 4, 5, NO, NA). 2 3

3. Determining an Indicator’s Rating “Best fit” approach 3

4. The Goal of Rater Accuracy To move from subjectivity to higher levels of objectivity and insight. 4

5. How Do You Become More Accurate? Practice, discipline, and intentional study of the rubric. 5

6. Let’s Review!! Indicator 1 Indicator 6 Indicator 7 Indicator 11 Indicator 12 “I think that’s a Revulsion!” “I like Chuck E. Cheese too!” “If you can express your answer in the numeric way, you can also express it in the geometric way and in words.” “I luv puzzles!!” “In science, you have to be flexible. Everyone be flexible” 6

7. Calibration Activity One 7

8. Your First Job… Using your rubric, working in PAIRS, sort observation evidence into their appropriate indicator category or categories: – Indicator 1 – Indicator 6 – Indicator 7 – Indicator 11 – Indicator 12 8

9. Answer Key to Sorting the Evidence, Calibration Activity #1 9 Evidence Indicator 1B, O, R, T Indicator 6C, F, H, L, Q (also I) Indicator 7D, E, I, M, S (also H) Indicator 11N, P (also A, G, J and S) Indicator 12A, G, J, K

10. Your Second Job… Using your rubric, working in PAIRS, determine the “best fit score” for each piece of evidence. – Note that the evidence typically relates to just one component of the indicator. Right now, you should ignore the other components of the indicator/rubric that are not addressed on the evidence sheet. 10

11. Suggested* Scores 11 Evidence Indicator Score A 11, 12 3 B 1 1-2 C 6 1 D 7 2 E 7 5 F 6 2 G 11, 12 4 H 6, 7 5 (Ind. 6), 3-4 (Ind. 7) I 6, 7 1-2 J 11, 12 1 (Ind. 12); 2 (Ind. 11)

12. Suggested* Scores 12 Evidence Indicator Score K 12 5 L 6 4 M 7 4 N 11 2 O 1 1-2 P 11 2-3 Q 6 3-4 R 1 S 7, 11 5 T 1 3

13. Wait…there’s something we haven’t talked about! 13

14. Make Your Own “Professional Preference” Lists Pet Peeves and Preferences 14

15. Common Pet Peeves/Preferences Pet Peeves/Negative Bias --messy rooms --sloppy attire --food/drink in the classroom --accents, loud speech --typohs --particular strategies Preferences/Positive Bias --neat room --attractive appearance --humor --casual demeanor --students in uniforms --particular strategies 15

16. Give me a 5 if you’re ready for a break! 16

17. We’re Back…ready or not!!! 17

18. Three Additional Indicators for Calibration Indicator 2: Clearly defined behavior Indicator 9: Active learning, questioning, incorporation of interests, and involving all students Indicator 13: Monitoring for progress 18

19. 19 Indicator 2

20. Indicator 2— Teacher clearly defines expected behavior. Ensures that students are engaged and clear as to the expectations of the classroom with few reminders given. Monitors the behavior of students during whole- class, small group and seat work activities and during transitions between instructional activities. Stops inappropriate behavior promptly and consistently with an appropriate voice level / word choice. 20

21. Clearly Defining Expected Behavior: What Do you Expect… teachers to be doing/saying if they are performing at an “effective” level? students to be doing/saying if teachers are performing at an “effective” level? 21

22. 22 Indicator 9

23. Indicator 9— Engagement of learners: active learning/ involvement/ questioning/ incorporation of interests. Engages most students in active learning experiences 80 percent of the class time. Uses questioning techniques throughout the lesson, scaffolding to at least the mid-level of Bloom's taxonomy. Provides wait time for some student response and does random checking to ensure the involvement of all learners. Engages students by incorporating their general skills and interests into the lesson. 23

24. Engagement of Learners What Do you Expect… teachers to be doing/saying if they are performing at an “effective” level? students to be doing/saying if teachers are performing at an “effective” level? 24

25. 25 Indicator 13

26. Indicator 13— Teacher checks to determine if students are progressing toward stated objectives. When appropriate, moves to all areas of the room while students are working on guided practice to promote and reinforce positive student behaviors. Uses different types of student response techniques, both individual / group. Uses student response techniques to increase active engagement. Student’s understanding is evaluated by feedback. Uses wait time of 3-5 seconds (more for more complex questions) after voicing the question. Provides opportunity for students to formulate more thoughtful responses and allows time for the student to consider supporting evidence. 26

27. Checks/Monitors for Progress What Do you Expect… teachers to be doing/saying if they are performing at an “effective” level… students to be doing/saying if teachers are performing at an “effective” level… 27

28. The Three-Step Process of Scoring Step One : Gather evidence Step Two : Sort evidence into indicators, as appropriate Step Three : Determine rating 28

29. 29

30. Calibration Activity Two 30

31. During the video – Step One, Part A: Take notes, documenting the evidence you observed relevant to indicators 2, 9 & 13. – We will pause briefly at the 5 min. marker to give you add’l time to write your evidence down on notepaper After the video – Step One, Part B: Complete your notes and review them for the most significant evidence relevant to indicators 2, 9 & 13. – Transfer the key evidence onto sticky notes—one per piece of evidence. 31 Practice Video 1… just indicators 2, 9 & 13 Step One Independent Work

32. Additional Notes About the Video Your “universe of evidence” is the video alone. – “consistently implements” = consistently within the video – Disregard rubric requirements that require knowledge outside of video (stricken in the rubric) Video format is similar to what will be used on the calibration test. – One screen on the board; one panoramic screen on the class. 32

33. 33 Everyone Ready?

34. Calibration Activity #2… Brief Practice Video …just 2, 9 & 13 CCOSA--Tulsa Model Video for Calibration Activity No. 2 (7th grade ELA) -- view first 12 minutes only 34

35. During the video – Step One, Part A: Take notes, documenting the evidence you observed relevant to indicators 2, 9 & 13. – We will pause briefly at the 5 min. marker to give you add’l time to write your evidence down on notepaper After the video – Step One, Part B: Complete your notes and review them for the most significant evidence relevant to indicators 2, 9 & 13. – Transfer the key evidence onto sticky notes—one per piece of evidence. 35 Practice Video 1… just indicators 2, 9 & 13 Step One Independent Work

36. Step Two, Part A: Using the worksheets as placemats, sort the evidence (on sticky notes) into the appropriate indicator category If evidence applies to more than one indicator, you may copy the evidence onto multiple sticky notes Step Two, Part B: Share worksheets with Table Team Consolidate the table’s findings evidence onto one placemat per indicator. 36 Practice Video 1… just indicators 2, 9 & 13 Step Two Independent WorkTable Work

37. Pause and Share Evidence with the Room What evidence did you have for Indicator 2? What evidence did you have for Indicator 9? 37

38. Pause and Share Evidence with the Room What evidence did you have for Indicator 13? 38

39. Practice Video 1… just indicators 2, 9 & 13 Step Three Step Three Using the placemats and the rubric, determine the appropriate rating for each indicator and write the “best fit” rating on the placemat in the designated spot. 39 Table Work

40. Pause and Share Ratings with the Room What rating did you have for Indicator 2? Target: 2 What rating did you have for Indicator 13? Target: 2 What rating did you have for Indicator 9? Target: 2 40

41. 41

42. Calibration Activity #3 Practice Test video! 42

43. Practice Video…eight indicators Rating only Indicators 1, 2, 6, 7, 9, 11, 12 & 13 Viewing just the first 20 minutes of an Observation Engine video (secondary Math). – We will NOT stop at the video while we view it. We will: – Independently categorize evidence, and – Independently rate the indicators; then – Share our findings with table teams and report. 43

44. 44 Everyone Ready to Cue Music?

45. Link to Practice Test Video (Secondary Math) CCOSA--Tulsa Model Video Practice Test (Secondary Math) -- view first 20 minutes only 45

46. Practice Test -- Secondary 46 Ind. 1: 3 Ind. 2: 3 Ind. 6: 3 Ind. 7: 2 Ind. 9: 2 Ind. 11: 3 Ind. 12: 4 Ind. 13: 2

47. Target Scores… 47

48. Are We Ready? 48

49. Calibration Test Basics Required by the state, but it’s TPS-developed Online platform called Observation Engine. The testing time will last 90 minutes in all. Scoring only the 20 minutes of 3 videos – Indicators 1, 2, 6, 7, 9, 11, 12, & 13 – Customized by elementary/secondary (your choice) Either test provides K-12 calibration certification – After each video, you will have 10 minutes to score before we proceed with the next video. 49

50. Calibration Test Basics, continued Target scores determined by expert panel Your “grade” is determined by comparing your scores against the target scores. Goal is 80% on target The tests will be graded electronically using the scantrons on which you enter your scores. Pass/Fail data will be available in about 2 weeks, if not earlier. 50

51. Context of Calibration Test Your “universe of evidence” is the video alone. – “consistently implements” = consistently within the video – Disregard rubric requirements that require knowledge outside of video (stricken in the rubric) Tips – Use the truncated rubric form for each video, focusing on each required indicator separately. – Don’t stress! You will do fine! – Re-takes are available (with new videos) – Don’t let your professional preferences bias you.

52. Are we Ready? …. Questions? 52

53. For Tulsa Model or Calibration Questions… Jana Burk – burkja@tulsaschools.org Gene S. Kleindienst - kleinge@tulsaschools.org Katy Ackley – ackleka@tulsaschools.org Jennifer Gripado – gripaje@tulsaschools.org