1. MET Results & FfT Evolution
NASDTEC June 10, 2013
Charlotte Danielson
Mark Atkinson
Change the title and subtitle to fit your presentation and audience

2. Why Assess Teacher Effectiveness?
Ensure teacher quality
Promote professional learning
We’re not doing either one very well

3. Teacher Evaluation System Design
High Rigor
Low ←
Level of Stakes →High
Low Rigor
System Design
Given what I have said thus far, we can think of two continua related to evaluation systems: one related to the level of stakes, (in the form of licensing, employment, or compensation) and the other concerning the rigor of the system (the clarity of the criteria, the design of the items to be assessed, the training of the assessors, etc.) If one maps one continuum on the other, the result is a graph with four quadrants like this one. (Show the graph.)
In the quadrant where both the stakes and the rigor are low (for example in most mentoring programs) there are no negative consequences of the low rigor. That is, the mentoring program may not be as good as it might be, but no one is harmed. Those systems with both high stakes and high rigor (for example, where the assessors go through extensive training and must pass a proficiency test - as in Praxis III and National Board) the result is a system with high levels of credibility and defensibility.
The difficulty arises, I think, where the system has high stakes but low rigor (and therefore low defensibility and credibility.) In those situations there is opportunity for harm, and mischief, and abuse. Those are the ones that really worry me. I also wonder whether the infrastructure required to establish, and maintain, a system of high rigor, is worth the benefits. It will be interesting to see the situations in which it turns out to be worth it.

4. Teacher Evaluation System Design
High Rigor
Structured Mentoring Programs, e.g. New Teacher Center
Low ←
National Board Certification
Praxis III
Level of Stakes →High
Informal Mentoring Programs
Traditional Evaluation Systems
Low Rigor
DANGER!!

5. Getting it “Right” …What Does This Mean?
Technically defensible - clear definition of practice - validated instrument - trained and certified evaluators - psychometrically valid
Professionally defensible - “We’re not going to fire our way to Finland” - systems that promote learning

6. Accountability is Not Enough
Number of Teachers
Of course, if the goal is the improvement of teaching, one must define “teaching” in such a way that the definition captures our collective vision of how classrooms should be. Hence the fft.
Teacher Effectiveness

7. Defining Effective Teaching
Two basic approaches:
Teacher practices, that is, what teachers do, how well they do the work of teaching
Results, that is, what teachers accomplish, typically how well their students learn

8. Two Major Research Studies

9. Correlation Between Observation Ratings and VAM (CCSR)
Results:
Ratings explained a significant portion of variation in VAM in reading and math
Relationship stronger in reading than in math
Teachers with high observation ratings had high VAMs (and vice-versa)

10. Key Findings From the CCSR Study on Professional Conversation
Principals and teachers thought the conferences they had about instruction were:
More reflective than those they had using the CPS checklist
Based on a shared language about instructional practice and improvement
Evidence-based, which reduced subjectivity .
However, the quality of the conversations could be improved because they were:
Dominated by principal talk
Driven by low-level questions
Principals identified the need for additional training in this area

11. The Measures of Effective Teaching project
Teachscape video capture, on- line training, and scoring tools
23,000 classroom videos from 3,000 teachers across 6 districts
On-line training and certification tests for 5 teaching frameworks
Framework for Teaching
CLASS
MQI (Math)
PLATO (ELA)
QST (Science)
1,000+ raters trained on-line
Over 50K+ scored videos
New York
City
Pittsburgh
Denver
Charlotte-Mecklenburg
Memphis
Dallas
Hillsborough
County
Teachscape’s panoramic video capture, sharing, and scoring tools have been used to capture and score more than 23,000 lessons in over 3,000 classrooms across six states. Each lesson was scored multiple times against multiple frameworks using Teachscape’s software. These 23,000 lessons were scored over 80,000 times using Teachscape software to render valid, reliable evaluations.

12. Getting Evaluation Systems Right

13. MET vs. Widget Effect

14. Accuracy Is Critical For Relevant Feedback
WHY?
On a 4 point scale 88.3% of teachers were a 2 or a 3!
- 7.5% at level 1
- 4.2 % at level 4
50% of the MET teachers scored within 0.4 points of one another;
Teachers at the 25th and 75th percentile scored less than one-quarter of a point above or below the average teacher;

15. Weighting the Measures

16. Outcomes of Various “Weights”

17. Increasing Reliability With Observations

18. Efforts to Ensure Accuracy in MET
Training & Certification
Daily calibration
Significant double scoring (15% - 20%)
Scoring conferences with master raters
Scoring supervisors
Validity videos

19. Ranking Systems vs. Qualitative Feedback

20. Teachscape Focus™: An Ecosystem of Support
Online, video-based training and assessment designed to focus and align educators on a common definition of teaching based on the Framework for Teaching
For Observers
Observer Training
Scoring Practice
Proficiency Assessment
Calibration
For Teachers
Foundational training on the FfT
Training modules on observable components of FfT
Available in 2011 and 2013 Framework Editions*
Teachscape Focus is a suite of tools that work together to ensure fair, consistent and reliable evaluations and a transparent evaluation process that builds trust between teachers and administrators
Observers must:
Have a good understanding of the domains and components of the Framework for Teaching
Be able to distinguish the different performance levels for each component
Be able to objectively identify evidence of teaching practice
Connect evidence to a specific component of the Framework for Teaching
Score classroom practice accurately on an established rubric based on the overall evidence
Be able to identify their bias and personal preferences
And for teachers:
Teachers need to be conversant in the language of the FfT for self-assessment and to engage in constructive dialogue with their peers, coaches, and evaluators
Build trust and transparency in teacher evaluation; training for teachers on the FFT that mirrors training for evaluators

21. Lesson Learned From 35,000 Observers
Test anxiety isn’t just for kids
With time, training & multiple tests, they pass at a 94% rate
Biggest error observers make is to misplace evidence relative to the components of the instrument
Most observers struggle with writing scoring rationales
Almost all struggle with leading post-observation professional conversations – video helps
Choices of video of distinguished teaching can become politically charged
Training alone may not be sufficient to eliminate the positive bias that results from pre-existing professional relationships
Evaluation systems that numerically rank teachers undermine the opportunity for feedback & learning

22. Evolution of the FfT

23. Teacher Evaluation Meets the Common Core

24. The Integration of the CCSS into The Framework for Teaching
Domain 4: Professional Responsibilities
Reflecting on Teaching
Maintaining Accurate Records
Communicating with Families
Participating in a Professional Community
Growing and Developing Professionally
Showing Professionalism
Domain 1: Planning and Preparation
Demonstrating Knowledge of Content and Pedagogy
Demonstrating Knowledge of Students
Setting Instructional Outcomes
Demonstrating Knowledge of Resources
Designing Coherent Instruction
Designing Student Assessments
Domain 2: The Classroom Environment
Creating an Environment of Respect and Rapport
Establishing a Culture for Learning
Managing Classroom Procedures
Managing Student Behavior
Organizing Physical Space
Domain 3: Instruction
Communicating With Students
Using Questioning and Discussion
Techniques
Engaging Students in Learning
Using Assessment in Instruction
Demonstrating Flexibility and Responsiveness

25. What Questions Are We Trying to Answer?
When classroom observers observe teaching of related lessons using both the CCSS Evidence Guides and the 2013 Edition of the Framework For Teaching do they detect relevant information with the Evidence Guides that is not discerned with the FfT?
How does the information detected by the Evidence Guides (but not the FfT) help observers to guide/coach/support teachers to teach kids to mastery of the CCSS?
How can the process for conducting observations with the FfT be amended to capture the relevant information unique to the Evidence Guides, while simultaneously be streamlined to be more efficient?

26. Discussion