1. Teacher Quality, Teacher Evaluation, and “Value-Added”
Sean P. Corcoran New York University Education Policy Breakfast April 27, 2012
Introduction and note this is from an economist’s perspective. Teacher quality has been a central focus of my research since I was a graduate student at the U of Maryland about 10 years ago.

2. How did we get here?
Research finds teachers are the most important school influence on student achievement
Teachers appear to vary widely in effectiveness, as measured by student gains on standardized tests
Teachers can have long-run measurable effects on life outcomes (e.g., Chetty et al., 2012)
First of all, how is it that education policy came to be so fixated on teacher quality?

3. How did we get here?
By many measures, teacher quality is inequitably distributed across students and schools
There is some evidence that teacher quality has declined over the long-run (Corcoran et al., 2004)

4. How did we get here?
If teachers are so important, what are we doing to ensure high-quality teachers can be found in every classroom, particularly for those students who need them the most?
The generally accepted answer among policymakers: not much, or at least current efforts are not working very well (e.g., see The Widget Effect)
The Widget Effect (New Teacher Project): “Our school systems treat all teachers as interchangeable parts, not professionals. Excellence goes unrecognized and poor performance goes unaddressed.” … Most would agree that evaluation systems currently in place in most schools provide minimal information about performance to the teacher or her principal.

5. Two key issues What is teacher quality and how do we measure it?
What policies are most effective in improving the level and distribution of teacher quality?

6. Professional Development
Potential Teachers
Self-selection
Recruitment
Preparation
Teacher training
Alternative pathways
Screening
Certification
Testing
Hiring / Placement
Retention
Tenure decisions
Involuntary exits
Turnover/attrition
Evaluation
Professional Development
There are many points in the “teacher quality pipeline” where policies can potentially make a difference for the quality of teachers.
The Teacher Quality Pipeline

7. What is teacher quality?
The easy (non-)answer: skills, practices, personal characteristics that positively impact desired student outcomes
Not a very helpful definition … but does make clear that it is ultimately outcomes that indicate quality
For example when we say someone is a good doctor, we’re referring to what they do, not where they were trained, or what score they received on a medical licensing examination

8. The “old view”
Research and policy emphasized qualifications and experience as presumed indicators of quality
Certification
Certification test scores
Educational attainment (e.g. MA)
Subject matter preparation
College selectivity
Own academic abilities (e.g. SAT)
In-service professional development

9. The “old view” – why?
Convenience – these measures are readily available and easily observable; a lack of data on outcomes themselves
Face validity – on their face, they seem sensible
Reward structure – traditional salary structure rewards these qualifications (e.g. MA, experience)

10. The “old view”
NCLB’s Highly Qualified Teacher provision: all teachers of core academic subjects must:
Have a BA or better in the subject matter taught
Have full state certification
Demonstrate subject matter knowledge
It turns out the “old view” is not that old!

11. Professional Development
Policies that set high professional standards and barriers to entry
Potential Teachers
Self-selection
Recruitment
Preparation
Teacher training
Alternative pathways
Screening
Certification
Testing
Hiring / Placement
Retention
Tenure decisions
Involuntary exits
Turnover/attrition
Evaluation
Professional Development
Professional Development
The Teacher Quality Pipeline

12. The “old view”
Surprisingly (or not) research has not found qualifications to be highly predictive of student outcomes (i.e. test scores), although some do better than others

13. The “new view”
“Teaching should be open to anyone with a pulse and a college degree—and teachers should be judged after they have started their jobs, not before”
Malcolm Gladwell, The New Yorker, 12/15/2008

14. The “new view”
“Success should be measured by results…That’s why any state that makes it unlawful to link student progress to teacher evaluation will have to change its ways.” President Barack Obama, July 24, 2009

15. The “new view” In other words, let outcomes be the arbiter of quality
Great in theory, but which outcomes do we measure, and how does one measure teachers’ contribution to them?
How does one incorporate this information into personnel policies in ways that have desired effects?

16. Professional Development
Policies that focus on measurement and incentives
Potential Teachers
Self-selection
Recruitment
Preparation
Teacher training
Alternative pathways
Screening
Certification
Testing
Hiring / Placement
Retention
Tenure decisions
Involuntary exits
Turnover/attrition
Evaluation
Professional Development
Evaluation
The Teacher Quality Pipeline

17. Measurement: outcomes
Outcomes: to date, whatever we have on hand
Typically, student growth on standardized tests in reading and math, grades 3-8 (though not for long)
Necessarily a subset of expected skills/outcomes
Necessarily a short-run outcome
Is our evaluation measure properly aligned with the goals we have for our educational system?
Here the tail has largely been wagging the dog

18. Measurement: value-added
The theoretical construct: a teacher’s unique impact on student learning
In practice, a statistic used to estimate this impact
Here the tail has largely been wagging the dog

19. Measurement: value-added
“Unique impact” implies causality – i.e. ruling other possible explanations for student learning
Several possible sources of error:
Systematic error (bias): attributing “value-added” to the teacher when it is really due to some other factor
Random error (noise): getting a “noisy signal” of the teacher’s contribution to learning
Test scores are a reflection of many in- and out-of-school factors past and present

22. Measurement: value-added
So how can we attribute causality to a teacher?
If teachers were randomly assigned, this would be easy: systematic differences would almost surely be due to the teacher

23. Measurement: value-added
In the absence of this, we can instead devise a statistical model to account for other factors that explain differences in achievement
Uses student data to generate a predicted score in ELA and Math for every student, given prior year’s achievement, student and class characteristics
Student’s actual score is compared with their predicted score
Actual > Predicted: (+) value-added
Actual < Predicted: (-) value-added
For a teacher, average these over all students. The extent to which a teacher’s student perform systematically better or worse than predicted is an estimate of her value-added
Assign a percentile rank for the teacher and a “margin of error”

24. Measurement: value-added
Value-added is then defined as student achievement relative to predicted—in other words, there will always be a distribution of value-added + -

25. Value-added: bias
How confident are we that value-added measures isolate the unique contribution of individual teachers?
Classroom vs. teacher effects (esp. after 1 year)
Teacher vs. school effects
Mobile students
Tracking (e.g. Rothstein falsification test)
Rothstein – in a “falsification test,” finds that 5th grade teachers have large “effects” on 4th grade test score gains
May be dynamic tracking – assignment to teachers in 5th grade dependent on 4th grade experience

26. Value-added: bias
Does attributing outcomes to individual teachers even make sense?
Middle and high school settings
Team teaching
Evidence that teacher peers matter
The higher the stakes places on value-added measures, the more these questions matter

27. Value-added: noise
Even if value-added measures are not biased, they are still noisy—i.e. they are estimates with a high “margin of error”
More years of test results helps, although this may be “too late” to provide actionable information

32. Implications for policy
The promise of personnel decisions driven by outcomes has led to sweeping reforms of
Performance evaluations
Tenure and promotion, dismissal
Compensation
Principal evaluation
Evaluation of teacher training programs

33. Implications for policy
Race to the Top led numerous states to propose 50% or more of performance evaluations to be the “teacher’s impact on student achievement”
E.g. CO, FL, TN, NJ
Indiana: “negative” value-added teachers may not receive an effective rating, and tenure requires 3 years of effective ratings in a row
NY’s APPR: a somewhat more balanced approach

34. Professional Development
Potential Teachers
Self-selection
Recruitment
Preparation
Teacher training
Alternative pathways
Screening
Certification
Testing
Hiring / Placement
Retention
Tenure decisions
Involuntary exits
Turnover/attrition
Evaluation
Professional Development
The Teacher Quality Pipeline

35. Implications for policy
What can we realistically expect from value-added based policies?
Not as much timely, actionable information as we might like – though perhaps useful as an early warning indicator
Crude differentiation of teachers at best, but more than current practice

36. Implications for policy
What are the risks and implications of a system based on high-stakes use of imprecise measures?
Mechanical applications are dangerous
Risk of improper attribution and “Type I errors”
Public reporting has minimal benefits and may do harm
Unnecessary diversion of resources
Unclear effects on entry into teaching profession

37. Implications for policy
Little is know about how value-added measures will be used in practice

38. References Excellent and (mostly) non-technical resources:
Corcoran (2010) report for Annenberg
Harris (2010) Value Added Measures in Education
Koretz (2008) in American Educator
Braun (2005) primer for ETS
“Merit Pay for Florida Teachers: Design and Implementation Issues” (RAND 2007)
Rivkin (2007) CALDER policy brief
Harris (2009) and Hill (2009) point/counterpoint in the Journal of Policy Analysis and Management