(C) SAS Institute Inc All Rights Reserved. Measurement of Academic Growth of Individual Students Toward Variable and Meaningful Academic Standards S. Paul Wright, William L. Sanders, June C. Rivers November 2005

(C) SAS Institute Inc All Rights Reserved. Measurement of Academic Growth of Individual Students Toward Variable and Meaningful Academic Standards 1. Introduction 2. Some Longitudinal (Growth) Models 2.1 Nested model 2.2 Cross-classified model 3. EVAAS Projection Methodology 3.1 The methodology 3.2 A connection to “growth models” 3.3 Simulations 3.4 EVAAS Projection Advantages 4. Using Projections to Enhance NCLB and Other Important Educational Objectives 5. Discussion/Conclusions

(C) SAS Institute Inc All Rights Reserved. Nested Model j = school i = student within a school, t = time X ij = student-level covariate W j = school-level covariate Level-1: Y tij = π 0ij + π 1ij ·t + ε tij. Level-2: π 0ij = β 00j + β 01j X ij + r 0ij, π 1ij = β 10j + β 11j X ij + r 1ij. Level-3: β 00j = γ γ 001 W j + u 00j, β 01j = γ γ 011 W j + u 01j, β 10j = γ γ 101 W j + u 10j, β 11j = γ γ 111 W j + u 11j.

(C) SAS Institute Inc All Rights Reserved. A Simplified Combined Nested Model: Y tij = ( γ γ 001 W j + γ 010 X ij + u 00j + r 0ij ) + ( γ γ 101 W j + γ 110 X ij + u 10j + r 1ij )·t + ε tij.

(C) SAS Institute Inc All Rights Reserved. Simple Cross-Classified Model (no covariates) Level-1: Y tij = π 0ij + π 1ij ·t + ε tij. Level-2: π 0ij = θ 0 + r 0i + u 0j, π 1ij = θ 1 + r 1i. Combined:Y tij = (θ 0 + r 0i + u 0j ) + (θ 1 + r 1i )·t + ε tij.

(C) SAS Institute Inc All Rights Reserved. Cumulative Cross-Classified Model (with covariates) Level-1: Y tij = π 0ij + π 1ij ·t + ε tij. Level-2: π 0ij = θ 0 + β 0 X i + r 0i + ∑ j ∑ h≤t D hij ( γ 0 W j + u 0j ), π 1ij = θ 1 + r 1i. Combined: Y tij = [θ 0 + β 0 X i + r 0i + ∑ j ∑ h≤t D hij ( γ 0 W j + u 0j )] + [θ 1 + r 1i ]·t + ε tij. D hij =1 if student “i” was in school “j” at time “h”, D hij =0 otherwise.

(C) SAS Institute Inc All Rights Reserved. EVAAS Projection Methodology Projected_Score i = M Y + b 1 (X 1i − M 1 ) + b 2 (X 2i − M 2 ) +... M Y, M 1, etc. are estimated mean scores for the response variable (Y) and for the predictor variables (Xs). “Non-Standard” Features (vs. “regression”) Not every student has the same set of predictors Means represent “average schooling experience” Hierarchical data structure

(C) SAS Institute Inc All Rights Reserved. EVAAS Projection Methodology Projected_Score i = M Y + b 1 (X 1i − M 1 ) + b 2 (X 2i − M 2 ) +... Regression slopes: b = C XX −1 C XY C XX = Cov(X), pooled-within-schools C XY = Cov(X, Y), pooled-within-schools

(C) SAS Institute Inc All Rights Reserved. A Connection to “Growth Models” A simplified nested linear growth model: Level-1: Y ti = π 0i + π 1i ·t + ε ti. Level-2: π 0i = β 00 + r 0i, π 1i = β 10 + r 1i. Combined: Y ti = (β 00 + β 10 ·t) + (r 0i + r 1i ·t + ε ti ) = μ t + δ ti. C i = var( δ i ) = Z i T Z i T + I σ 2 where σ 2 = var(ε ti ), assumed same for all “t” and “i”; T = var({r 0i, r 1i }), assumed same for all “i”; Z i has: a column of “1”s (intercept), a column of “t”s (slope).

(C) SAS Institute Inc All Rights Reserved. A Connection to “Growth Models” Linear Growth versus EVAAS: Linear Growth EVAAS Data: Vertically linked Unrestricted Means: Linear Unrestricted Covariances: Structured Unstructured

(C) SAS Institute Inc All Rights Reserved. Simulations Data generation: Model: Y ti = (β 00 + r 0i ) + (β 10 + r 1i )·t + ε ti = μ t + δ ti, i = 1, …, 2500; t = 0, 1, 2, 3; β 00 = 400; β 10 = 100; μ t = {400, 500, 600, 700}; σ 2 = var(ε ti ) = 5 2 = 25; τ 00 = var(r 0i ) = 15 2 = 225; τ 11 = var(r 1i ) = 5 2 = 25; τ 01 = cov(r 0i, r 1i ) = 0.

(C) SAS Institute Inc All Rights Reserved. Simulations Parameter estimation and projections: EVAAS: Two samples 1. For parameter estimation 2. For projections to t = 3 Linear Growth Model: Second sample only Parameter estimation using t = 0, 1, 2 Projections to t = 3

(C) SAS Institute Inc All Rights Reserved. Simulations Results: Mean prediction error (bias): MPE = Σ i [projected(Y 3i ) − Y 3i ] / Mean squared prediction error: MSPE = Σ i [projected(Y 3i ) − Y 3i ] 2 / 2500.

(C) SAS Institute Inc All Rights Reserved. Simulation Results Original data: μ t = {400, 500, 600, 700}. EVAAS: MPE = −0.11; MSPE = Growth: MPE = −0.19; MSPE = Variation #1: μ t = {400, 500, 600, 700}, τ 11 = 0. EVAAS: MPE = +0.14; MSPE = Growth: MPE = −0.19; MSPE = Variation #2: μ t = {400, 505, 605, 700}, τ 11 = 0. EVAAS: MPE = +0.14; MSPE = Growth: MPE = +8.15; MSPE = Variation #3: μ t = {400, 510, 610, 700}, τ 11 = 0. EVAAS: MPE = +0.14; MSPE = Growth: MPE = ; MSPE =

(C) SAS Institute Inc All Rights Reserved. EVAAS Projection Advantages Test scores need not be vertically linked No assumption about shape of growth curve Missing values are easily handled Massive data sets are readily accommodated

(C) SAS Institute Inc All Rights Reserved. EVAAS Projection Applications NCLB Safe Harbor Individual student counseling

(C) SAS Institute Inc All Rights Reserved. Issues Growth/projection versus Value-added models Modeling problems Fractured student records Varying test scales Changing testing regimes