Understanding Student Achievement: The Value of Administrative Data Eric Hanushek Stanford University

Big Issues in School Policy Debates  Relating analysis to policy interests  Confidence in causation  Generalizability

Analytical designs  Random assignment experiments  Natural experiments  “Data solutions”  Trade-offs Credibility Expense Questions that can be addressed

UTD Texas Schools Project  Multiple cohorts followed  Annual achievement in grades 3-8 (TAAS math and reading)  Each cohort > 200,000 students in over 3,000 schools  Augmented with district data

Examples of Topics  Teacher quality variations  Charter schools  Not discussed School choice and mobility Special education Teacher mobility Racial composition Peer achievement

Existing Evidence on Teacher Quality  Substantial variation in teacher quality  Observable characteristics of teachers explain little of the variation  Salary and other factors affect teacher transition probabilities  No evidence on transitions and teacher quality

Questions Addressed  What is variation in teacher quality? Measurable characteristics?  Do urban schools lose their best teachers? Quality by transitions  Do districts hire the best teachers?

Basic model

Measurement Error and Calculation of Variance of Teacher Quality  Observe teachers in two years:  Correlation across years:

Estimated Variance in Teacher Quality Lonestar District Within district Within school and year unadjusted demographic controls unadjusted demographic controls Teacher-year variation Adjacent year correlation Teacher quality variance / (s.d.) (0.32) (0.27) (0.22) (0.22)

Conclusions on Teacher Quality  Very large differences among teachers Differences within schools much larger than between schools  Conventional measures not good index of quality (master’s degree, certification test)  Observable characteristics First year of experience Teacher-student race match  Common assumptions about market for teachers not correct Best do not leave Districts with advantages do not use them

Popularity of charter schools  3,000 charter schools  40 states plus DC since 1991  1 percent of total students  10 percent of size of private school market  7+ percent rate of closure

Evaluation issues  Most analysis of entry and participation  No reliable information on performance  Difficulty of selection issue  Very political

Evaluation approaches  Model selection process [Heckman (1979)]  Instrument for attendance [Neal(1997)]  Intake randomization [Howell and Peterson (2002)]

Difficulties with traditional approaches  Difficult to find factors affecting attendance but not achievement  Cannot handle treatment heterogeneity

Empirical framework  Mean differences in individual value-added Identify charter school from individual entry-exit Consider time varying effects associated with charter school movements  Heterogeneity across schools  Consumer responsiveness to quality

Charter enrollment th grade0.2 %0.8% 7 th grade0.2%0.9%

Participation rates by race/ethnicity Blacks0.8%2.2% Hispanics0.1%0.6% Whites0.0%0.4% Low income0.3%0.8%

Charters by vintage (analytical) Total one

Charters by vintage (analytical) Total one two

Charters by vintage (analytical) Total one two Three Four Five

Charter school effect Charter-0.17 Age Age Age Age Age 5 or more0.02

Demographically Adjusted School Quality

Do parents make good decisions?  Parents cannot see value added  Considerable mobility/exiting  Models: Exit=f(quality, age, year, race, grade)

Parental Choice (linear probability of exit) Student characteristics Student + peer characteristics Student + peer characteristics + peer achievement School quality School quality x charter

Parental Choice (linear probability of exit) Student characteristics Student + peer characteristics Student + peer characteristics + peer achievement School quality School quality x charter high income low income

Conclusions on Charter Schools  Difficult start-up period  Mean performance regular ≈ charter after two years  Heterogeneity in both markets  Parents react to quality in charter market Low income reaction one half upper income

Administrative data  Pros Broader generalizability Understanding heterogeneity Perhaps less costly  Cons Requires structure (e.g., linearity, time pattern of achievement) Regulatory problems (confidentiality) Data quality issues

Papers on Teacher Quality and Charter Schools  or Hanushek, Eric A., John F. Kain, Daniel M. O'Brien, and Steve G. Rivkin "The market for teacher quality." National Bureau of Economic Research, Working Paper No , (February). Hanushek, Eric A., John F. Kain, Steve G. Rivkin, and Gregory F. Branch "Charter school quality and parental decision making with school choice." National Bureau of Economic Research, (March).