Teacher Quality, Teacher Evaluation, and “Value-Added” Sean P. Corcoran New York University Education Policy Breakfast April 27, 2012 Introduction and note this is from an economist’s perspective. Teacher quality has been a central focus of my research since I was a graduate student at the U of Maryland about 10 years ago.

How did we get here? Research finds teachers are the most important school influence on student achievement Teachers appear to vary widely in effectiveness, as measured by student gains on standardized tests Teachers can have long-run measurable effects on life outcomes (e.g., Chetty et al., 2012) First of all, how is it that education policy came to be so fixated on teacher quality?

How did we get here? By many measures, teacher quality is inequitably distributed across students and schools There is some evidence that teacher quality has declined over the long-run (Corcoran et al., 2004)

How did we get here? If teachers are so important, what are we doing to ensure high-quality teachers can be found in every classroom, particularly for those students who need them the most? The generally accepted answer among policymakers: not much, or at least current efforts are not working very well (e.g., see The Widget Effect) The Widget Effect (New Teacher Project): “Our school systems treat all teachers as interchangeable parts, not professionals. Excellence goes unrecognized and poor performance goes unaddressed.” … Most would agree that evaluation systems currently in place in most schools provide minimal information about performance to the teacher or her principal.

Two key issues What is teacher quality and how do we measure it? What policies are most effective in improving the level and distribution of teacher quality?

Professional Development Potential Teachers Self-selection Recruitment Preparation Teacher training Alternative pathways Screening Certification Testing Hiring / Placement Retention Tenure decisions Involuntary exits Turnover/attrition Evaluation Professional Development There are many points in the “teacher quality pipeline” where policies can potentially make a difference for the quality of teachers. The Teacher Quality Pipeline

What is teacher quality? The easy (non-)answer: skills, practices, personal characteristics that positively impact desired student outcomes Not a very helpful definition … but does make clear that it is ultimately outcomes that indicate quality For example when we say someone is a good doctor, we’re referring to what they do, not where they were trained, or what score they received on a medical licensing examination

The “old view” Research and policy emphasized qualifications and experience as presumed indicators of quality Certification Certification test scores Educational attainment (e.g. MA) Subject matter preparation College selectivity Own academic abilities (e.g. SAT) In-service professional development

The “old view” – why? Convenience – these measures are readily available and easily observable; a lack of data on outcomes themselves Face validity – on their face, they seem sensible Reward structure – traditional salary structure rewards these qualifications (e.g. MA, experience)

The “old view” NCLB’s Highly Qualified Teacher provision: all teachers of core academic subjects must: Have a BA or better in the subject matter taught Have full state certification Demonstrate subject matter knowledge It turns out the “old view” is not that old!

Professional Development Policies that set high professional standards and barriers to entry Potential Teachers Self-selection Recruitment Preparation Teacher training Alternative pathways Screening Certification Testing Hiring / Placement Retention Tenure decisions Involuntary exits Turnover/attrition Evaluation Professional Development Professional Development The Teacher Quality Pipeline

The “old view” Surprisingly (or not) research has not found qualifications to be highly predictive of student outcomes (i.e. test scores), although some do better than others

The “new view” “Teaching should be open to anyone with a pulse and a college degree—and teachers should be judged after they have started their jobs, not before” Malcolm Gladwell, The New Yorker, 12/15/2008

The “new view” “Success should be measured by results…That’s why any state that makes it unlawful to link student progress to teacher evaluation will have to change its ways.” President Barack Obama, July 24, 2009

The “new view” In other words, let outcomes be the arbiter of quality Great in theory, but which outcomes do we measure, and how does one measure teachers’ contribution to them? How does one incorporate this information into personnel policies in ways that have desired effects?

Professional Development Policies that focus on measurement and incentives Potential Teachers Self-selection Recruitment Preparation Teacher training Alternative pathways Screening Certification Testing Hiring / Placement Retention Tenure decisions Involuntary exits Turnover/attrition Evaluation Professional Development Evaluation The Teacher Quality Pipeline

Measurement: outcomes Outcomes: to date, whatever we have on hand Typically, student growth on standardized tests in reading and math, grades 3-8 (though not for long) Necessarily a subset of expected skills/outcomes Necessarily a short-run outcome Is our evaluation measure properly aligned with the goals we have for our educational system? Here the tail has largely been wagging the dog

Measurement: value-added The theoretical construct: a teacher’s unique impact on student learning In practice, a statistic used to estimate this impact Here the tail has largely been wagging the dog

Measurement: value-added “Unique impact” implies causality – i.e. ruling other possible explanations for student learning Several possible sources of error: Systematic error (bias): attributing “value-added” to the teacher when it is really due to some other factor Random error (noise): getting a “noisy signal” of the teacher’s contribution to learning Test scores are a reflection of many in- and out-of-school factors past and present

Measurement: value-added So how can we attribute causality to a teacher? If teachers were randomly assigned, this would be easy: systematic differences would almost surely be due to the teacher

Measurement: value-added In the absence of this, we can instead devise a statistical model to account for other factors that explain differences in achievement Uses student data to generate a predicted score in ELA and Math for every student, given prior year’s achievement, student and class characteristics Student’s actual score is compared with their predicted score Actual > Predicted: (+) value-added Actual < Predicted: (-) value-added For a teacher, average these over all students. The extent to which a teacher’s student perform systematically better or worse than predicted is an estimate of her value-added Assign a percentile rank for the teacher and a “margin of error”

Measurement: value-added Value-added is then defined as student achievement relative to predicted—in other words, there will always be a distribution of value-added + -

Value-added: bias How confident are we that value-added measures isolate the unique contribution of individual teachers? Classroom vs. teacher effects (esp. after 1 year) Teacher vs. school effects Mobile students Tracking (e.g. Rothstein falsification test) Rothstein – in a “falsification test,” finds that 5th grade teachers have large “effects” on 4th grade test score gains May be dynamic tracking – assignment to teachers in 5th grade dependent on 4th grade experience

Value-added: bias Does attributing outcomes to individual teachers even make sense? Middle and high school settings Team teaching Evidence that teacher peers matter The higher the stakes places on value-added measures, the more these questions matter

Value-added: noise Even if value-added measures are not biased, they are still noisy—i.e. they are estimates with a high “margin of error” More years of test results helps, although this may be “too late” to provide actionable information

Implications for policy The promise of personnel decisions driven by outcomes has led to sweeping reforms of Performance evaluations Tenure and promotion, dismissal Compensation Principal evaluation Evaluation of teacher training programs

Implications for policy Race to the Top led numerous states to propose 50% or more of performance evaluations to be the “teacher’s impact on student achievement” E.g. CO, FL, TN, NJ Indiana: “negative” value-added teachers may not receive an effective rating, and tenure requires 3 years of effective ratings in a row NY’s APPR: a somewhat more balanced approach

Professional Development Potential Teachers Self-selection Recruitment Preparation Teacher training Alternative pathways Screening Certification Testing Hiring / Placement Retention Tenure decisions Involuntary exits Turnover/attrition Evaluation Professional Development The Teacher Quality Pipeline

Implications for policy What can we realistically expect from value-added based policies? Not as much timely, actionable information as we might like – though perhaps useful as an early warning indicator Crude differentiation of teachers at best, but more than current practice

Implications for policy What are the risks and implications of a system based on high-stakes use of imprecise measures? Mechanical applications are dangerous Risk of improper attribution and “Type I errors” Public reporting has minimal benefits and may do harm Unnecessary diversion of resources Unclear effects on entry into teaching profession

Implications for policy Little is know about how value-added measures will be used in practice

References Excellent and (mostly) non-technical resources: Corcoran (2010) report for Annenberg http://www.annenberginstitute.org/products/Corcoran.php Harris (2010) Value Added Measures in Education Koretz (2008) in American Educator Braun (2005) primer for ETS “Merit Pay for Florida Teachers: Design and Implementation Issues” (RAND 2007) Rivkin (2007) CALDER policy brief Harris (2009) and Hill (2009) point/counterpoint in the Journal of Policy Analysis and Management