1. SEDL January 10–11, 2013 UTOP Training Presented by Mary Walker

2.  Who am I?  Who are you?  Pick a partner and listen carefully — you will introduce your partner to the group at the end of this activity!  Introduce yourself by providing:  Your name and current position  Experience and background that led to your current position  One significant formal or informal educational experience that changed or impacted your thinking about teaching and learning WHO ARE WE? 2

3. MEASURING EFFECTIVE TEACHING: 3-MINUTE PAPER  What does effective teaching look like?  How can we measure effective teaching?  How can we help teachers reflect on their teaching and improve it? 3

4. SOME KEY FEATURES OF EFFECTIVE TEACHING [EMBEDDED IN THE UTEACH MODEL]  Organized, well-managed, on-task classroom  Attention to issues of diversity and access  Incorporating inquiry/investigative learning  Using technology for teaching and learning  Fluid and accurate communication of content  Fostering student-student collaboration  Formative assessment of student progress  Applications and interdisciplinary connections  Critical practices of self-reflection  Facilitating classroom discussion and “student talk” Research in Education; NSES, NRC, NCTM Standards 4

5.  RTOP [Reformed Teaching Practices]  http://physicsed.buffalostate.edu/AZTEC/RTOP/RTO P_full/about_RTOP.html http://physicsed.buffalostate.edu/AZTEC/RTOP/RTO P_full/about_RTOP.html  COP [Classroom Observation Protocol]  www.horizon-research.com/instruments/lsc/cop.pdf www.horizon-research.com/instruments/lsc/cop.pdf  http://www.horizon- research.com/instruments/hri_instrument.php?inst_i d=14 http://www.horizon- research.com/instruments/hri_instrument.php?inst_i d=14 OBSERVATIONAL PROTOCOL DEVELOPMENT FOR RESEARCH 5

6.  Focus on mathematics and science content and concepts that are significant and developmentally appropriate  Observers MUST have knowledge of STEM standards and course expectations  Values and evaluates what students are doing, not just what the teacher is doing  Provides feedback STEM teachers want and need in order to grow professionally WHAT MAKES UTOP UNIQUE? 6

7. DESCRIPTION OF UTOP  Full version has 27 indicators (teaching and student behaviors) in 4 domains  Classroom Environment  Lesson Structure  Implementation  Mathematics/Science Content  1-5 scale [DK/NA options]  Section Synthesis Ratings — “the human average” 7

8. RatingIndicator 1.1 The classroom environment encouraged students to generate ideas, questions, conjectures, and/or propositions that reflected engagement or exploration with important mathematics and science concepts. DescriptionDescription Rubric Specific Rating ExamplesRubricSpecific Rating Examples Evidence: 1.2 Interactions reflected collegial working relationships among students. (e.g. students worked together productively and talked with each other about the lesson). *It's possible that this indicator was not applicable to the observed lesson. You may rate NA in this case. DescriptionDescription Rubric Specific Rating ExamplesRubricSpecific Rating Examples Evidence: 1.3 Based on conversations, interactions with the teacher, and/or work samples, students were intellectually engaged with important ideas relevant to the focus of the lesson. DescriptionDescription Rubric Specific Rating ExamplesRubricSpecific Rating Examples Evidence: 1.4 The majority of students were on task throughout the class. DescriptionDescription Rubric Specific Rating ExamplesRubricSpecific Rating Examples Evidence: 1.5 The teacher’s classroom management strategies enhanced the classroom environment. DescriptionDescription Rubric Specific Rating ExamplesRubricSpecific Rating Examples Evidence: 1.6 The classroom is organized appropriately such that students can work in groups easily, get to lab materials as needed, teacher can move to each student of student group, etc. DescriptionDescription Rubric Specific Rating ExamplesRubricSpecific Rating Examples Evidence: 1.7 The classroom environment established by the teacher reflected attention to issues of access, equity, and diversity for students (e.g. cooperative learning, language-appropriate strategies and materials, attentiveness to student needs). DescriptionDescription Rubric Specific Rating ExamplesRubricSpecific Rating Examples This indicator should be rated a 1 if there is group work during the lesson, but the group work is highly unproductive. This could include behavior where the majority of the groups are socializing, off-task, arguing, or ignoring each other, as well as regular instances of students copying and/or certain group members doing all of the work. This indicator should be rated a 2 if … This indicator assesses the degree to which students have learned to be collegial, respectful, cooperative, and interactive when working in groups. Evidence of collegial working relationships among students includes collaborative discussions about topics relevant to the lesson and successful distributing of roles and responsibilities within each group… Rating of 3 Example: The students were put into debate groups for this class period - one group would debate another group, while the rest of the student groups were in the audience. The groups worked together smoothly - the students were able to pick who was doing what part of the debate, coordinate their arguments, and split the time slots when necessary. The audience also would occasionally compare their notes during breaks… UTOP AND ONLINE MANUAL 8

9. PILOT STUDY  Developed and tested UTOP in some of our graduates’ classrooms — 2007 to 2009  Conducted 83 observations of:  UTeach Graduates (N=21)  Non-UTeach Graduates (N=15)  Novice teachers (most 0–3 years exp)  Math, science, and computer science classes; middle and high school 9

10. PILOT STUDY  After starting out at similar levels, UTeachers gain higher UTOP scores over time [without seeing their own data!]  Teaching experience significant predictor of UTOP scores for UTeach group (p <.05)  Noyce Scholars rated significantly higher on UTOP than other groups (p <.01)  Key Question: Is the UTOP a valid and reliable instrument that measures important components of effective teaching? 10

11. MET/NMSI STUDY  UTOP study conducted in partnership with the Gates Foundation’s Measures of Effective Teaching project and the National Math and Science Initiative  Opportunity to examine reliability, consistency, factor structure  Look to see if we can connect teaching behaviors on UTOP to teacher value-added gains 11

12. THE MET PROJECT YEAR ONE  3000 teachers from 7 school districts, 7 states  Various subjects (mathematics, English, science) and grade levels  Multiple measures of effectiveness (observations, value- added, student surveys, teacher exams)  Multiple video lessons of each teacher  Multiple classroom observation instruments  Charlotte Danielson’s FFT  CLASS protocol (Pianta, et al.)  MQI Rubric (Ball, et al.)  UTOP (UTeach group) 12

13. MET/NMSI STUDY  99 raters (math and science master teachers with LTF/AP) scored 994 video lessons of 250 teachers using UTOP  All lessons grades 4–8 mathematics  One-third of videos double-scored 13

14.  Most of the 4–8 math video lessons from this national sample did not score highly on the UTOP  Many middle school math teachers observed teaching problematic content — mostly formulaic/key word type approaches.  Raters identified problematic content issues in around one half of all lessons RESULTS 14

15.  Surface-level engagement often seen, but little emphasis on conceptual understanding  “Orderly but unambitious” RESULTS 15

16.  If 1 observer comes in 1 time per year to observe a teacher with the UTOP…  33% of variance in scores due to teacher characteristics  66% due to rater bias and the characteristics of the lesson  If 4 observers each come in 1 time per year to observe a teacher with the UTOP…  66% of variance in scores due to teacher characteristics  Similar for 2 observations per year with two different observers present at each observation  This reliability is close to or higher than other MET instruments RELIABILITY 16

17. MET/NMSI STUDY: VALUE-ADDED CORRELATIONS  Are the teaching behaviors measured on the UTOP associated with higher student learning gains on standardized assessments and tests of conceptual understanding? 17

18. VALUE-ADDED CORRELATIONS This graph is copied from the released 2012 MET Report 18

19.  Multiple observations with STEM- knowledgeable, trained observers is necessary for reliability  Correlations with student test score gains and teacher UTOP scores are weakly positive at best  Observations, student perception survey data, and student learning measure different things but all are needed to get a complete picture of what happens in the classroom TEACHING EFFECTIVENESS STUDIES: LESSONS LEARNED 19

20.  Measure of Effective Teaching (2012). http://www.metproject.org/downloads/MET_Gathering_Feedback _Research_Paper.pdf http://www.metproject.org/downloads/MET_Gathering_Feedback _Research_Paper.pdf  Walkington, C., Walker, M., & Marder, M. (2011, July). Developing tools to evaluate the practice of Noyce Scholars: The UTeach Classroom Observation Protocol. Presentation at the NSF Robert Noyce Teacher Scholarship Program Conference. Washington, DC.  Walkington, C., et al. (2011). Development of the UTeach Observation Protocol: A classroom observation instrument to evaluate mathematics and science teachers from the Uteach preparation program. http://www.cwalkington.com/UTOP_Paper_2011.pdfhttp://www.cwalkington.com/UTOP_Paper_2011.pdf UTOP STUDIES 20

21.  Can a classroom observation tool [UTOP] change the way we teach and students learn when used by a professional learning community of teachers/administrators/university facilitators?  What other support structures and resources are needed to get the most out of UTOP observations — PLCs? Coaching and mentoring by facilitators or colleagues? On-demand PD? SO, WHY UTOP? PUTTING LESSONS LEARNED INTO PRACTICE 21

22.  Learn to use UTOP by using the tool  http://uteach.utexas.edu/UTOP http://uteach.utexas.edu/UTOP  UTOP Video Version  UTOP Manuals — Full and Video  UTOP Full Version  All videos from http://www.timssvideo.com/http://www.timssvideo.com/  8 th -grade mathematics lesson – US3 Exponents  8 th -grade science lesson – AU4 Energy Transfer  For extra practice: Go to the TIMSS website, view JP2, score with a friend, and post to discussion forum [email with details to follow] Overview: UTOP training agenda 22

23.  Rate each indicator with a 1–5, typed into the box in the Word document  Type 1–5 sentences of supporting evidence into the “Evidence” box to justify each numerical rating. HOW TO USE THE UTOP 23

24.  Supporting Evidence should be:  Specific — based on specific quotes and interactions from video  No opinions! You must justify your rating based on evidence.  Somewhat brief — try to average 3–4 sentences  Your evidence is how you prove to us that you actually watched the video  Bad Supporting Evidence:  Is brief  Is vague  Is opinionated  Is not really related to the indicator’s intent  Gives the teacher directive feedback (“You should have…”)  Is too specific to your knowledge/background WHAT IS EVIDENCE FOR RATINGS? 24

25.  View Video 1, taking field notes  Discuss with partner or group how to score Video 1 on each indicator AND come to consensus on the Synthesis Rating  Whole-group discussion and comparison with “expert” ratings  http://uteach.utexas.edu/UTOP/ http://uteach.utexas.edu/UTOP/ VIDEO 1: ENERGY 25

26.  View Video 2, taking field notes  Discuss “big ideas” and impressions with partner or group  End Day One VIDEO 2: EXPONENTS 26

27.  Review what you wrote earlier in the day about measuring effective teaching. Has anything changed? Why or why not?  Based on the day’s training, describe how you would use an instrument like the UTOP to help a teacher learn to grow professionally. EXIT TICKET/HOMEWORK 27

28.  Discuss answers to EXIT TICKET questions:  Review what you wrote about effective teaching. Has anything changed? Why or why not?  Based on the training so far, describe how you would use an instrument like the UTOP to help a teacher learn to grow professionally.  Discuss your responses with your group.  Choose a spokesperson to summarize for reporting out. WARM-UP 28

29.  Whole-group discussion of Video 2 and report out  Synthesis Rating for each domain  Highest indicator rating  Lowest indicator rating  View “teaser” of Video 3, JP2, Changing Shape without Changing Area  Begin field notes  Complete with a colleague(s) at a later time  Email completed UTOPs to me for posting [Details to follow] DAY TWO: UTOP TRAINING (CONTINUED) 29

30.  Can a classroom observation tool [UTOP] change the way we teach and students learn when used by a professional learning community of teachers/administrators/university facilitators?  What other support structures and resources are needed to get the most out of UTOP observations — PLCs? Coaching and mentoring by facilitators or colleagues? On-demand PD? SO, WHY UTOP? PUTTING LESSONS LEARNED INTO PRACTICE 30

31.  What are we doing with the UTOP?  Video annotation of user manual  Develop web-based training and recalibration modules  Develop targeted professional development modules  Manor Study  Preservice AT UTOP evaluations WRAP UP 31

32.  Profiles of two schools  Profiles show areas of strength and weaknesses  Teacher’s individual profile is shared in private conference  Can compare to own school profile  Teachers choose what areas to work on  Facilitators assist in forming PLC’s focused on similar areas  Facilitators can pair appropriate mentors  Facilitators can provide “on-demand” PD DATA REVEAL EXAMPLE 32

33.  What are you going to do with the UTOP? YOUR TURN 33

34. mwalker@austin.utexas.edu melissa.dodson@sedl.org CONTACT INFORMATION 34