MET Results & FfT Evolution NASDTEC June 10, 2013 Charlotte Danielson Mark Atkinson Change the title and subtitle to fit your presentation and audience

Why Assess Teacher Effectiveness? Ensure teacher quality Promote professional learning We’re not doing either one very well

Teacher Evaluation System Design High Rigor Low ←--------------------------------------- Level of Stakes -------------------→High Low Rigor System Design Given what I have said thus far, we can think of two continua related to evaluation systems: one related to the level of stakes, (in the form of licensing, employment, or compensation) and the other concerning the rigor of the system (the clarity of the criteria, the design of the items to be assessed, the training of the assessors, etc.) If one maps one continuum on the other, the result is a graph with four quadrants like this one. (Show the graph.) In the quadrant where both the stakes and the rigor are low (for example in most mentoring programs) there are no negative consequences of the low rigor. That is, the mentoring program may not be as good as it might be, but no one is harmed. Those systems with both high stakes and high rigor (for example, where the assessors go through extensive training and must pass a proficiency test - as in Praxis III and National Board) the result is a system with high levels of credibility and defensibility. The difficulty arises, I think, where the system has high stakes but low rigor (and therefore low defensibility and credibility.) In those situations there is opportunity for harm, and mischief, and abuse. Those are the ones that really worry me. I also wonder whether the infrastructure required to establish, and maintain, a system of high rigor, is worth the benefits. It will be interesting to see the situations in which it turns out to be worth it.

Teacher Evaluation System Design High Rigor Structured Mentoring Programs, e.g. New Teacher Center Low ←--------------------------------------- National Board Certification Praxis III Level of Stakes -------------------→High Informal Mentoring Programs Traditional Evaluation Systems Low Rigor DANGER!!

Getting it “Right” …What Does This Mean? Technically defensible - clear definition of practice - validated instrument - trained and certified evaluators - psychometrically valid Professionally defensible - “We’re not going to fire our way to Finland” - systems that promote learning

Accountability is Not Enough Number of Teachers Of course, if the goal is the improvement of teaching, one must define “teaching” in such a way that the definition captures our collective vision of how classrooms should be. Hence the fft. Teacher Effectiveness

Defining Effective Teaching Two basic approaches: Teacher practices, that is, what teachers do, how well they do the work of teaching Results, that is, what teachers accomplish, typically how well their students learn

Two Major Research Studies

Correlation Between Observation Ratings and VAM (CCSR) Results: Ratings explained a significant portion of variation in VAM in reading and math Relationship stronger in reading than in math Teachers with high observation ratings had high VAMs (and vice-versa)

Key Findings From the CCSR Study on Professional Conversation Principals and teachers thought the conferences they had about instruction were: More reflective than those they had using the CPS checklist Based on a shared language about instructional practice and improvement Evidence-based, which reduced subjectivity . However, the quality of the conversations could be improved because they were: Dominated by principal talk Driven by low-level questions Principals identified the need for additional training in this area

The Measures of Effective Teaching project Teachscape video capture, on- line training, and scoring tools 23,000 classroom videos from 3,000 teachers across 6 districts On-line training and certification tests for 5 teaching frameworks Framework for Teaching CLASS MQI (Math) PLATO (ELA) QST (Science) 1,000+ raters trained on-line Over 50K+ scored videos New York City Pittsburgh Denver Charlotte-Mecklenburg Memphis Dallas Hillsborough County Teachscape’s panoramic video capture, sharing, and scoring tools have been used to capture and score more than 23,000 lessons in over 3,000 classrooms across six states. Each lesson was scored multiple times against multiple frameworks using Teachscape’s software. These 23,000 lessons were scored over 80,000 times using Teachscape software to render valid, reliable evaluations.

Getting Evaluation Systems Right

MET vs. Widget Effect

Accuracy Is Critical For Relevant Feedback WHY? On a 4 point scale 88.3% of teachers were a 2 or a 3! - 7.5% at level 1 - 4.2 % at level 4 50% of the MET teachers scored within 0.4 points of one another; Teachers at the 25th and 75th percentile scored less than one-quarter of a point above or below the average teacher;

Weighting the Measures

Outcomes of Various “Weights”

Increasing Reliability With Observations

Efforts to Ensure Accuracy in MET Training & Certification Daily calibration Significant double scoring (15% - 20%) Scoring conferences with master raters Scoring supervisors Validity videos

Ranking Systems vs. Qualitative Feedback

Teachscape Focus™: An Ecosystem of Support Online, video-based training and assessment designed to focus and align educators on a common definition of teaching based on the Framework for Teaching For Observers Observer Training Scoring Practice Proficiency Assessment Calibration For Teachers Foundational training on the FfT Training modules on observable components of FfT Available in 2011 and 2013 Framework Editions* Teachscape Focus is a suite of tools that work together to ensure fair, consistent and reliable evaluations and a transparent evaluation process that builds trust between teachers and administrators Observers must: Have a good understanding of the domains and components of the Framework for Teaching Be able to distinguish the different performance levels for each component Be able to objectively identify evidence of teaching practice Connect evidence to a specific component of the Framework for Teaching Score classroom practice accurately on an established rubric based on the overall evidence Be able to identify their bias and personal preferences And for teachers: Teachers need to be conversant in the language of the FfT for self-assessment and to engage in constructive dialogue with their peers, coaches, and evaluators Build trust and transparency in teacher evaluation; training for teachers on the FFT that mirrors training for evaluators

Lesson Learned From 35,000 Observers Test anxiety isn’t just for kids With time, training & multiple tests, they pass at a 94% rate Biggest error observers make is to misplace evidence relative to the components of the instrument Most observers struggle with writing scoring rationales Almost all struggle with leading post-observation professional conversations – video helps Choices of video of distinguished teaching can become politically charged Training alone may not be sufficient to eliminate the positive bias that results from pre-existing professional relationships Evaluation systems that numerically rank teachers undermine the opportunity for feedback & learning

Evolution of the FfT

Teacher Evaluation Meets the Common Core

The Integration of the CCSS into The Framework for Teaching Domain 4: Professional Responsibilities Reflecting on Teaching Maintaining Accurate Records Communicating with Families Participating in a Professional Community Growing and Developing Professionally Showing Professionalism Domain 1: Planning and Preparation Demonstrating Knowledge of Content and Pedagogy Demonstrating Knowledge of Students Setting Instructional Outcomes Demonstrating Knowledge of Resources Designing Coherent Instruction Designing Student Assessments Domain 2: The Classroom Environment Creating an Environment of Respect and Rapport Establishing a Culture for Learning Managing Classroom Procedures Managing Student Behavior Organizing Physical Space Domain 3: Instruction Communicating With Students Using Questioning and Discussion Techniques Engaging Students in Learning Using Assessment in Instruction Demonstrating Flexibility and Responsiveness

What Questions Are We Trying to Answer? When classroom observers observe teaching of related lessons using both the CCSS Evidence Guides and the 2013 Edition of the Framework For Teaching do they detect relevant information with the Evidence Guides that is not discerned with the FfT? How does the information detected by the Evidence Guides (but not the FfT) help observers to guide/coach/support teachers to teach kids to mastery of the CCSS? How can the process for conducting observations with the FfT be amended to capture the relevant information unique to the Evidence Guides, while simultaneously be streamlined to be more efficient?

Discussion