-- Preliminary, Do Not Quote Without Permission -- VALUE-ADDED MODELS AND THE MEASUREMENT OF TEACHER QUALITY Douglas HarrisTim R. Sass Dept. of Ed. LeadershipDept. of Economics and Policy StudiesFlorida State University Florida State IES Research Conference – June 2006

-- Preliminary, Do Not Quote Without Permission -- Evaluating Value-Added Methodology The recent availability of panel data has produced a flood of research studies using various “value-added” approaches The recent availability of panel data has produced a flood of research studies using various “value-added” approaches Research Questions Research Questions Are assumptions underlying the value-added approach valid? Are assumptions underlying the value-added approach valid? Are some methods more likely to produce reliable estimates than others? Are some methods more likely to produce reliable estimates than others? What data are most important to obtaining consistent estimates? What data are most important to obtaining consistent estimates?

-- Preliminary, Do Not Quote Without Permission -- Evaluating Value-Added Methodology Basic Model Types Basic Model Types Cumulative Model Cumulative Model Unrestricted Value-Added Model Unrestricted Value-Added Model Value-Added Models with Persistence Restrictions Value-Added Models with Persistence Restrictions Restricted Value-Added or “Gain-Score” Model Restricted Value-Added or “Gain-Score” Model Contemporaneous Model Contemporaneous Model Specification Issues for Value-Added Models Specification Issues for Value-Added Models Treatment of teacher heterogeneity Treatment of teacher heterogeneity Measures of classroom/school inputs Measures of classroom/school inputs Treatment of student heterogeneity Treatment of student heterogeneity Aggregation Aggregation

-- Preliminary, Do Not Quote Without Permission -- General Cumulative Model of Student Achievement

-- Preliminary, Do Not Quote Without Permission -- Basic Assumptions of Value-Added Models Cumulative achievement function does not vary with age and is additively separable. Cumulative achievement function does not vary with age and is additively separable. Family inputs are time invariant. Family inputs are time invariant. Parents do not compensate for poor school inputs or poor outcomes Parents do not compensate for poor school inputs or poor outcomes Todd and Wolpin (2005) reject exogeneity of parental inputs at 90 percent, but not at 95 percent confidence level Todd and Wolpin (2005) reject exogeneity of parental inputs at 90 percent, but not at 95 percent confidence level The marginal inputs of all school-based inputs, parental inputs, and the initial student endowment each decline geometrically (at potentially different rates) over time. The marginal inputs of all school-based inputs, parental inputs, and the initial student endowment each decline geometrically (at potentially different rates) over time. Lagged achievement serves as a sufficient statistic for prior inputs Lagged achievement serves as a sufficient statistic for prior inputs We find twice-lagged inputs do not provide additional information We find twice-lagged inputs do not provide additional information

-- Preliminary, Do Not Quote Without Permission -- Unrestricted Value-Added Model

-- Preliminary, Do Not Quote Without Permission -- Persistence Restrictions Restricted Value-Added or “Gain-Score” Model Restricted Value-Added or “Gain-Score” Model is assumed to equal 1 (no decay in effect of past inputs) is assumed to equal 1 (no decay in effect of past inputs) Alternatively, can interpret as an achievement growth model where growth is independent of past school inputs. Alternatively, can interpret as an achievement growth model where growth is independent of past school inputs. Contemporaneous Model Contemporaneous Model is assumed to equal 0 (complete decay in effect of past inputs). is assumed to equal 0 (complete decay in effect of past inputs).

-- Preliminary, Do Not Quote Without Permission -- Decomposition of School-based Inputs in Value-Added Model

-- Preliminary, Do Not Quote Without Permission -- Modeling Teacher Heterogeneity Substituting teacher time-invariant measured characteristics for teacher fixed effects

-- Preliminary, Do Not Quote Without Permission -- Classroom and School Inputs Exclusion of peer variables (P -ijmt ) Number of peers (class size) and peer characteristics (gender, race, mobility, age) If peer variables are correlated with student and teacher characteristics (X it and T kt ), omission will produce inconsistent estimates Exclusion of school fixed effects (  m ) Given that teachers do not frequently change schools, omission of school effects will mean that teacher fixed effects will capture both teacher effects and some of the school effect, leading to inconsistent estimates

-- Preliminary, Do Not Quote Without Permission -- Modeling Student Heterogeneity Substituting measured time-invariant student characteristics for student fixed effects Race/ethnicity, foreign/native born, language parent speak at home, free-lunch status As with teachers, if unmeasured time-invariant student characteristics are correlated with independent variables, will get inconsistent estimates

-- Preliminary, Do Not Quote Without Permission -- Modeling Student Heterogeneity Fixed vs. random student effects Fixed effects allow for a separate intercept parameter for each student (equal to the mean error for that student) whereas random effects assume that the student-specific intercepts are drawn from a known distribution (typically normal) Since random effects are part of the error structure, they must be orthogonal to the model variables (X it, P -ijmt, T kt ) in order to yield consistent estimates Given that fixed effects estimates are always consistent (whether or not unobserved student heterogeneity is correlated with other variables in the model), can test orthogonality assumption by applying a Hausman test Multilevel fixed effects models have been computationally burdensome

-- Preliminary, Do Not Quote Without Permission -- Aggregation Measuring characteristics of specific teachers vs. grade-level- within-school averages Since Texas data does not identify specific teacher, work by Rivkin, Hanushek and Kain (2005) relies on average characteristics of teachers within a grade Advantages/Disadvantages of aggregation Eliminates problems associated with non-random assignment of students to teachers within a school May reduce problem of measurement error since individual errors may cancel out at grade level May upwardly bias estimated impacts of school resources in the presence of omitted variables Tends to reduce precision of estimates

-- Preliminary, Do Not Quote Without Permission -- Data Florida’s K-20 Education Data Warehouse Florida’s K-20 Education Data Warehouse Census of all children attending public schools in Florida Census of all children attending public schools in Florida Student records linked over time Student records linked over time Covers 1995/1996 – 2003/2004 school years Covers 1995/1996 – 2003/2004 school years Includes student test scores and student demographic data, plus enrollment, attendance, disciplinary actions and participation in special education and limited English proficiency programs Includes student test scores and student demographic data, plus enrollment, attendance, disciplinary actions and participation in special education and limited English proficiency programs Includes all employee records including individual teacher characteristics and means of linking students and teachers to specific classrooms Includes all employee records including individual teacher characteristics and means of linking students and teachers to specific classrooms

-- Preliminary, Do Not Quote Without Permission -- Sample for Analysis Middle school students (grades 6-8) who took SSS- NRT (Stanford-9) math test in three consecutive years during 1999/2000 – 2003/2004 Middle school students (grades 6-8) who took SSS- NRT (Stanford-9) math test in three consecutive years during 1999/2000 – 2003/2004 Enrolled in a single math course in the Fall Enrolled in a single math course in the Fall Up to 4 years of achievement gains Up to 4 years of achievement gains 4 cohorts of students 4 cohorts of students Includes a variety of math courses, from remedial to advanced and gifted classes Includes a variety of math courses, from remedial to advanced and gifted classes Use random sample of 100 middle schools Use random sample of 100 middle schools Reduces computational burden of estimating fixed effects Reduces computational burden of estimating fixed effects Represents about 12% of middle schools in state Represents about 12% of middle schools in state

-- Preliminary, Do Not Quote Without Permission -- Value-Added Model Estimates With Varying Degrees of Persistence

-- Preliminary, Do Not Quote Without Permission -- Correlation of Estimated Teacher Effects From Models with Varying Degrees of Persistence

-- Preliminary, Do Not Quote Without Permission -- Restricted Value-Added Model Estimates With Differing Controls for Teacher Heterogeneity

-- Preliminary, Do Not Quote Without Permission -- Restricted Value-Added Model Estimates With Differing Classroom/School Controls

-- Preliminary, Do Not Quote Without Permission -- Correlation of Estimated Teacher Effects From Models with Differing Classroom/School Controls

-- Preliminary, Do Not Quote Without Permission -- Restricted Value-Added Model Estimates with Differing Controls for Student Heterogeneity

-- Preliminary, Do Not Quote Without Permission -- Correlation of Estimated Teacher Effects From Models With Differing Controls for Student Heterogeneity

-- Preliminary, Do Not Quote Without Permission -- Restricted Value-Added Model Estimates -- Teacher-Specific vs. Within-School Grade-Level Averages

-- Preliminary, Do Not Quote Without Permission -- Summary of Findings Model Selection Model Selection Restricted value-added model seems to be a good approximation of the full cumulative model Restricted value-added model seems to be a good approximation of the full cumulative model Specification Specification Use of student and teacher fixed effects (rather than covariates) important Use of student and teacher fixed effects (rather than covariates) important Random effects may yield inconsistent estimates Random effects may yield inconsistent estimates Important to include school fixed effects, but classroom peer variables relatively unimportant Important to include school fixed effects, but classroom peer variables relatively unimportant Aggregation to the grade level has some effect, though estimates not radically different from estimates with teacher- level data Aggregation to the grade level has some effect, though estimates not radically different from estimates with teacher- level data