The Determinants of Student Achievement: Different Estimates for Different Measures Tim Sass Department of Economics Florida State University CALDER Conference October 4, 2007

Different Measures  Types of Tests Criterion Reference Tests  Test whether student has learned elements in state established instructional standards  State specific Nationally Normed Tests  Tests whether student has learned a set of concepts and skills that may or may not correspond to any particular state’s curriculum benchmarks  Allows interstate comparisons

Different Measures  Scaling Non-Vertically Aligned Scale Scores  Scale potentially different at each grade level  Can’t compare learning gains  Criterion reference tests are typically not vertically aligned Vertical or Developmental Scales  A single equal-interval scale that spans all grade levels  A one-unit change means the same at all levels within and between grades  Some norm-referenced exams are of this type  Stanford Achievement Test

Non-Vertically Aligned Scores Grade 10 Grade 9 Grade 8 Grade 7 Grade 6 Grade 5 Grade 4 Grade 3 Trigonometry Single-Digit Addition

Vertically Scaled Scores Grade 10 Grade 9 Grade 8 Grade 7 Grade 6 Grade 5 Grade 4 Grade 3 If done right, vertically scaled exam ideal for analyzing learning gains since one-point change has same meaning everywhere on the scale. Trigonometry Simple Addition

Different Measures  Scale Scores Normalized by Grade and Year Frequently used by researchers to compare a student’s performance on criterion referenced tests over time  Compares a student’s performance relative to the performance of other students taking the same grade-level exam in the same year  Unit of measure is the standard deviation  If performance distribution changes from grade to grade, normalized scores may not be comparable Also sometimes used to try to equate performance on different exams when a state changes their test midstream

Normalized score sets mean to zero and rescales score Normalized Scores Grade 5 0

Different Results  Analysis of the Effectiveness of NBPTS Certified Teachers Harris and Sass, “The Effects of NBPTS-Certified Teachers on Student Achievement” (February, 2007)  Compares the effectiveness of NBPTS-certified teachers (NBCTs) with the effectiveness of non-NBCTs in Florida  In many cases, results vary whether use scores from Florida’s criterion reference test, the FCAT-Sunshine State Standards exam (FCAT-SSS), or the Stanford Achievement Test, a norm-referenced test (FCAT-NRT)

Value-Added Estimates of Reading Achievement Selected Explanatory Variables FCAT-SSS Developmental Scale FCAT-NRT Developmental Scale FCAT-SSS Normalized by Grade & Year FCAT-NRT Normalized by Grade & Year NBPTS Certified First-Year Teacher Years of Teaching Experience Years of Teaching Experience Years of Teaching Experience Advanced Degree Class Size Note: all coefficients expressed in standard deviation units omitted experience category is teachers with 10+ years of experience coefficients in green are statistically significant at the 95% confidence level

Different Results  More variation in estimated effects across exams than in different scalings of same exam  Estimated effects of variables representing small proportions of teachers most variable NBPTS Certification Advanced Degrees  Why are there differences across exams? Differences in material covered Differential ceiling effects

Grade 10 Grade 9 Grade 8 Grade 7 Grade 6 Grade 5 Grade 4 Grade 3 Trigonometry Simple Addition Vertically Scaled Scores With Ceiling

Conclusions  Not much difference between developmental scale scores and non-vertically aligned scores that are normalized by grade and year  Different tests can yield different results Low-incidence variables seem to be most sensitive to test instrument Not clear whether differences due to material tested or differential ceiling effects