The Impact of Including Predictors and Using Various Hierarchical Linear Models on Evaluating School Effectiveness in Mathematics Nicole Traxel & Cindy Walker University of Wisconsin - Milwaukee April 14, 2009 The Milwaukee Mathematics Partnership (MMP) is supported by the National Science Foundation under Grant No

Introduction Value added models Value added models Fair and accurate way to assess the effectiveness of schools Fair and accurate way to assess the effectiveness of schools Determine how much value a school adds to student learning by examining student progress over time Determine how much value a school adds to student learning by examining student progress over time Operational definition: Effectiveness=growth Operational definition: Effectiveness=growth Hierarchical linear models can be used to implement a value added accountability system Hierarchical linear models can be used to implement a value added accountability system Hierarchical because students nested within schools Hierarchical because students nested within schools Can determine how much of growth can be attributed to the student and to the school Can determine how much of growth can be attributed to the student and to the school

Types of Hierarchical Linear Models Several different hierarchical linear models can be used to assess school effectiveness, so which is best? Several different hierarchical linear models can be used to assess school effectiveness, so which is best? 2-level hierarchical model predicts final achievement from initial achievement. Can include student level and school level predictors of achievement, not growth 2-level hierarchical model predicts final achievement from initial achievement. Can include student level and school level predictors of achievement, not growth 2-level growth model predicts change in test scores from one year to the next. Can include student level and school level predictors of growth, not achievement 2-level growth model predicts change in test scores from one year to the next. Can include student level and school level predictors of growth, not achievement 3-level individual growth model predicts achievement and change over time. Can include student level and school level predictors of growth and achievement 3-level individual growth model predicts achievement and change over time. Can include student level and school level predictors of growth and achievement

Research Questions Do effectiveness rankings differ depending on which type of model is used? Do effectiveness rankings differ depending on which type of model is used? Does predictor significance remain constant across model types? Does predictor significance remain constant across model types? Does including predictors change effectiveness rankings of school? Does including predictors change effectiveness rankings of school?

Sample & Measures 7,232 students from 128 school from a large urban school district in the Midwest 7,232 students from 128 school from a large urban school district in the Midwest 3 rd to 4 th grade 3 rd to 4 th grade 87% minority, 79% receive free/reduced lunch 87% minority, 79% receive free/reduced lunch Mathematics scores on a state mandated standardized test Mathematics scores on a state mandated standardized test Math Focus score for each school – “There is a strong focus on increasing student achievement in mathematics at my school.” Math Focus score for each school – “There is a strong focus on increasing student achievement in mathematics at my school.” Gain in Math Focus calculated by subtracting Math Focus score from 1 st year from Math Focus score from 2 nd year Gain in Math Focus calculated by subtracting Math Focus score from 1 st year from Math Focus score from 2 nd year

The Models That Were Fit 2-Level 2-Level Gain 3-Level Initial Score None Student Race Student SES Student Race & SES Student & School Race Student & School SES Student & School Race & SES Note: Initial Score was included as a student level covariate in all 2-level and 3-level models, but not in the 2-level gain models.

Comparisons Predictor significance across models Predictor significance across models Effectiveness rankings across predictors being included within each model type Effectiveness rankings across predictors being included within each model type Effectiveness rankings across model types for models including only initial score or no predictors Effectiveness rankings across model types for models including only initial score or no predictors Effectiveness rankings across model types for models including only initial score or no predictors validated using gain in Math Focus score Effectiveness rankings across model types for models including only initial score or no predictors validated using gain in Math Focus score

Predictor Significance Student-level SES and Student-level Race were significant predictors of the average achievement of students within schools but not of the average growth of students within schools Student-level SES and Student-level Race were significant predictors of the average achievement of students within schools but not of the average growth of students within schools School-level Race and School-level SES were significant predictors of the average achievement among schools but not of the average growth among schools School-level Race and School-level SES were significant predictors of the average achievement among schools but not of the average growth among schools But not when both were included, due to collinearity But not when both were included, due to collinearity

Predictors Do Not Change Effectiveness 2-Level Model2-Level Gain Model3-Level Model Student Level Race Student Level SES Student Level Race & SES Student & School Level Race Student & School Level SES Student & School Level Race & SES Spearman Correlations between the null model and models including predictors for each model type.

Correlations Among Model Types (Null Models) 2-level and 2-level gain models 2-level and 2-level gain models r =.090 r = level and 3-level models 2-level and 3-level models r =.101 r = level gain and 3-level models 2-level gain and 3-level models r =.993 r =.993 Therefore, rankings from 2-level gain and 3-level models are very similar to one another Therefore, rankings from 2-level gain and 3-level models are very similar to one another

Validating Effectiveness Rankings Pearson correlations between gain in math focus and effectiveness estimates from null models of each model type. Pearson correlations between gain in math focus and effectiveness estimates from null models of each model type. 2-level: r =.034, p = level: r =.034, p = level gain: r =.200, p = level gain: r =.200, p = level: r =.177, p = level: r =.177, p =.089

Conclusions, Part One Including predictors, even if they are significant, does not change effectiveness estimates Including predictors, even if they are significant, does not change effectiveness estimates Effectiveness estimates from null 2-level model were different from effectiveness estimates from null 2-level gain and 3-level models Effectiveness estimates from null 2-level model were different from effectiveness estimates from null 2-level gain and 3-level models Effectiveness estimates from 2-level gain and 3- level models were very similar Effectiveness estimates from 2-level gain and 3- level models were very similar

Conclusions, Part Two Correlation between 2-level gain and 3-level models and gain in math focus had higher magnitudes than correlation between 2-level model and gain in math focus Correlation between 2-level gain and 3-level models and gain in math focus had higher magnitudes than correlation between 2-level model and gain in math focus Effectiveness estimates from 2-level gain and 3- level models are more valid than those from 2- level model Effectiveness estimates from 2-level gain and 3- level models are more valid than those from 2- level model

So which model type should I use? 3-level model has several advantages over 2-level gain model 3-level model has several advantages over 2-level gain model Includes ALL available data-all participants with at least one observation are included Includes ALL available data-all participants with at least one observation are included Can include many years of data Can include many years of data Provides more information (growth and achievement estimates) Provides more information (growth and achievement estimates)