Informing Policy: State Longitudinal Data Systems Jane Hannaway, Director The Urban Institute CALDER

State of U.S. Education ½ of minority students graduate from high school ½ of minority students graduate from high school 4 grade level gap between white and minority students by 12 th grade 4 grade level gap between white and minority students by 12 th grade 15% of minorities earn BAs w/in 9 years of 9 th grade 15% of minorities earn BAs w/in 9 years of 9 th grade

The WILL and the WAY The WILL The WILL –Left, Right, Center –Agreement on education crisis –Strange bedfellows The WAY The WAY –Few, but growing, guideposts

Finding the WAY with Evidence -A New Day- Who has the evidence? Who has the evidence? –States have the makings of the evidence Where are the makings? Where are the makings? –State administrative data systems Why do states have it? Why do states have it? –Important effect of NCLB Why important? Why important? –Address questions never before possible

Research Background: What We Know Teachers matter- single most important schooling contributor to student outcomes Teachers matter- single most important schooling contributor to student outcomes Wide variation in teacher effectiveness. Some teachers are simply much better than others Wide variation in teacher effectiveness. Some teachers are simply much better than others Standard measures of teacher quality not much related to effectiveness, but directly related to spending. Standard measures of teacher quality not much related to effectiveness, but directly related to spending.

Research Background: What We Don’t Know What is it about teachers that matters? What is it about teachers that matters?

3 Research Probes Teacher Maldistribution Teacher Maldistribution Teacher Selection Teacher Selection Teacher Mobility Teacher Mobility

Teacher Maldistribution 1 Comparison of VA of teachers in high/ low poverty schools Comparison of VA of teachers in high/ low poverty schools North Carolina and Florida North Carolina and Florida Findings Findings –Low poverty - higher va, but not much –High poverty – larger variation in school

Teacher Value-Added at Percentiles by School Poverty Levels (North Carolina-Math)

Teacher Value-Added at Percentiles by School Poverty Levels (Florida- Math)

Novice teachers are less effective than experienced teachers. Novice teachers are less effective than experienced teachers. Returns to experience taper off 3-5 years. Returns to experience taper off 3-5 years.

Distribution of Value-Added of Elementary Math Teachers in High Poverty Schools Solid line: Novice teachers Dash line: Teachers with 1-2 years of experience Dotted line: Teachers with 3-5 years of experience

Distribution of Value-Added of Elementary Math Teachers in Lower Poverty Schools Solid line: Novice teachers Dash line: Teachers with 1-2 years of experience Dotted line: Teachers with 3-5 years of experience

Teacher Maldistribution 2 New York City New York City –Phasing out of emergency certification –Introduction of alternative route teachers

LAST Exam Failure Rate of Elementary Teachers by Poverty Quartile,

LAST Exam Failure Rate of New Teachers by Poverty Quartile,

Predicted Effectiveness For Highest and Lowest Poverty Schools Narrows by 25%

Can change predicted effectiveness by selection up- front Mean VA by Quintile (poor schools) Passed Exam Not Certified Math SAT Verb al SAT college competitiveness MostSomeLessNot  Meaningful difference based only on attributes, though monitoring, development and selective retention also needed

Teacher Selection Teach for America Teach for America –North Carolina –Secondary school –Mainly math and science

TFA Findings – high school Student FE, Math subjects TFA v. all others TFA v. in- field non-TFA TFA v. traditional track TFA Experience 3-5 yrs 3-5 yrs yrs 6-10 yrs > 10 yrs > 10 yrs All TFA coefficients are significant at the.05 level.

Teacher Mobility Mobility highest at most challenging schools Mobility highest at most challenging schools The worst teachers are the first to leave The worst teachers are the first to leave General tendency to move to more affluent schools General tendency to move to more affluent schools

Topic of the Day: Performance Incentives Objective?? Objective?? –Recruitment/ selection –Retention/ deselection –Increase performance thru effort

Issues How good are the measures? How good are the measures? Individual vs school rewards? Individual vs school rewards? Teachers without test scores? Teachers without test scores?

VA Measures Problems Problems –Year to year variability –Measurement error –Sorting How serious? How serious? –Less serious for policy research –More serious for individual stakes

Predicting Performance Using first 2 yrs of performance – top to top/ bottom to bottom quintile Using first 2 yrs of performance – top to top/ bottom to bottom quintile –Goldhaber and Hansen (NC): 46%/ 44% –Koedel and Betts (SanDiego): 29%/ 35% –Sass (Florida): 22-32%/ 24-24%

Policy Implications Use VA freely for research Use VA freely for research Use VA carefully for individual teacher judgments Use VA carefully for individual teacher judgments –Important information –Corrorboration More years are better More years are better –Move tenure decision out!

Research Questions Are teachers in high poverty schools more/ less effective (value-added) than teachers in lower poverty schools? Are teachers in high poverty schools more/ less effective (value-added) than teachers in lower poverty schools? Do school factors affect differences in the value- added of high poverty and lower poverty teachers? Do school factors affect differences in the value- added of high poverty and lower poverty teachers? Do teacher qualifications affect differences in the value-added of high poverty and lower poverty teachers? Do teacher qualifications affect differences in the value-added of high poverty and lower poverty teachers?

Data Florida (2000/ /05) Florida (2000/ /05) –Elementary –Student achievement – FCAT-SSS Grades 3-10 Grades 3-10 –Teacher links Assignment, certification, experience, education Assignment, certification, experience, education North Carolina (2000/1-2004/5) North Carolina (2000/1-2004/5) –Elementary –Student achievement EOG – grades 3-8 EOG – grades 3-8 EOC – secondary subjects EOC – secondary subjects –Teacher linked through proctor and verification Assignment, certification, experience, education Assignment, certification, experience, education

Definitions High poverty elementary schools (>70% FRL students) High poverty elementary schools (>70% FRL students) Lower poverty elementary schools (<70% FRL students) Lower poverty elementary schools (<70% FRL students) Very low poverty schools (<30% FRL students). Very low poverty schools (<30% FRL students).

NC Student-Teacher Link EOC student-level records Aggregate to EOC test classrooms by school, year, subject, proctor id Instructional Classroom records including school, year, subject, a teacher id. Decision Rules Match if teacher and proctor id identical and fit statistic < 1.5.

Sample Restrictions Exclude charter schools Exclude charter schools Exclude schools that switch high poverty to lower poverty status Exclude schools that switch high poverty to lower poverty status Only classrooms w/ students Only classrooms w/ students Only self-contained elementary classrooms Only self-contained elementary classrooms

Analytic Sample 0-30% FRL 30-70% FRL % FRL Total Florida (Elementary School Level) 3, 084 6, 975 5,232 14, 052 North Carolina (Elementary School Level) 2,207 5, 945 2, 316 9,212 Note: We focus on elementary schools, grades 3-5 where poverty information is most reliable. We exclude teachers from charter schools and we exclude classrooms with 40 students in our samples.

Methodological Challenges Non-random sorting of teachers and students Non-random sorting of teachers and students Distinguishing teacher and school effects Distinguishing teacher and school effects Precision in Teacher Effects Estimates Precision in Teacher Effects Estimates Sources of Teacher Effectiveness Differentials Sources of Teacher Effectiveness Differentials

Descriptive Findings: Elementary Student Demographics

Descriptive Findings: Student Performance Florida North Carolina 0-30% FRL 0-30% FRL 30-70% FRL % FRL 0-30% FRL 30-70% FRL % FRL Level Scores: Math0.49**0.25** **0.15**-0.32 Reading0.50**0.26** **0.14**-0.34 Gain Scores: Math-0.02**-0.01** * Reading * **0.01**0.00 * Differences between the given estimate and the corresponding estimates for schools with % FRL students significant at ≤ 5% and ** differences significant at ≤ 1%.

Descriptive Findings: Teacher Experience

Descriptive Findings: Teacher Qualifications

Distribution of Value-Added of Elementary Reading Teachers in Lower Poverty Schools Solid line: Novice teachers Dash line: Teachers with 1-2 years of experience Dotted line: Teachers with 3-5 years of experience

Distribution of Value-Added of Elementary Reading Teachers in High Poverty Schools Solid line: Novice teachers Dash line: Teachers with 1-2 years of experience Dotted line: Teachers with 3-5 years of experience

Florida Florida North Carolina North Carolina Difference Difference Math: FE Estimates -+ Student Covariate Estimates -+ Reading: FE Estimates ++ Student Covariate Estimates ++ Differences in Estimates of Teacher Value-Added

Magnitude of Differences in Value Added Estimates

Differences in Standard Deviations of Value-Added Florida Florida North Carolina North Carolina Difference Difference Math: FE Estimates -- Student Covariate Estimates -- Reading: FE Estimates -- Student Covariate Estimates --

Differences between Lower- and High-Poverty by Percentile of Teacher Value Added

Teacher Value-Added at Percentiles by School Poverty Levels (North Carolina- Reading)

Teacher Value-Added at Percentiles by School Poverty Levels (Florida- Reading)

Correlation of Teacher Qualifications and Value- Added

Sources of Difference in Teacher Value- Added Between High-Poverty and Lower-Poverty Elementary Schools

Sensitivity Analysis School Effect School Effect Empirical Bayes Adjustment Empirical Bayes Adjustment

Conclusions Teachers in high poverty schools, on average, are less effective than teachers in lower poverty schools. Teachers in high poverty schools, on average, are less effective than teachers in lower poverty schools. –Changing schools (high poverty/lower poverty) does not affect teacher effectiveness There is greater teacher variation within high poverty schools than within lower poverty schools. There is greater teacher variation within high poverty schools than within lower poverty schools.

Conclusions (con’t) Differences in teachers in High Poverty and Lower Poverty schools: Differences in teachers in High Poverty and Lower Poverty schools: –only weakly related to teacher qualifications –more strongly related to marginal effect of qualifications (experience) –not explained by school poverty level

Study Limitations Issues with value-added measures Issues with value-added measures –separating current teacher contributions from other current contributions E.g., current family circumstances E.g., current family circumstances - dynamic sorting sorting on time variant characteristics sorting on time variant characteristics –Instability of measures E.g., measurement error, motivation E.g., measurement error, motivation