CAASPP Results 2015 to 2016 Santa Clara Assessment and Accountability Network May 26, 2017 Eric E, Zilbert Administrator, Psychometrics, Evaluation and Data Office Assessment Development and Administration Division California Department of Education

Presentation Overview Background and Introduction Properties of the Smarter Balanced Scale Discussion of Change Scores Scale Score Distributions by Grade, 2016 Change Score Distributions by Grade 2015-2016 Normative Growth by Scale Score in 2015 Changes in Relative Standing 2015-2016

Change is the Only Constant 2016 – Second Administration of the Smarter Balanced Assessments in English language arts and mathematics Features of the Assessments: Computer Adaptive Component Performance Task Component Continuous Reporting Scale 4 Achievement Levels set by Grade Level Each grade level has a pre-specified lowest obtainable scale score (LOSS) and highest obtainable scale score (HOSS) for each grade. The results of the 2016 assessments provide the first opportunity to examine score changes for students on the scales.

About Change Scores The simple difference between last year’s score and this year’s score needs to be interpreted in light of the following: The variability of change scores is significantly greater than that for scores in a single year. The error is essentially compounded. Score changes are impacted by regression to the mean for very high scoring and very low scoring students. The meaning of a change score is somewhat different for different parts of the scale and for different grades. Change scores are best interpreted in terms of where the student started in the previous year.

Summary Statistics for Smarter Balanced Assessments 2015

Change in Mean Smarter Balanced Scale Scores 2015-2016

Change in Mean Smarter Balanced Scale Scores 2015-2016

Properties of the Smarter Balanced Scales

Properties of the Smarter Balanced Scales

ELA Scale Score Distributions by Grade

Math Scale Score Distributions by Grade

2015-16 ELA Scale Score Changes by Grade

2015-16 ELA Scale Score Changes

2015-16 Math Scale Score Changes

2015-16 Math Scale Score Changes

Change and Prior Performance Calculated average performance by scale score in the previous year. Estimated decile average change by picking students within 5 scale score points +/- of the percentile reported Calculated at the 10th, 25th, 50th, 75th, and 90th percentiles of 2015 performance. Demonstrate the likelihood of growth in future years with no more than 81% accuracy (test reliability=.9).

Statewide Average Change in Scale Score by Scale Score in 2015: ELA

Normative Growth by Scale Score in 2015 ELA

Statewide Average Change in Scale Score by Scale Score in 2015: Mathematics

Normative Growth by Scale Score in 2015 Mathematics

What to do? Focus on ranges of growth: little or no growth, expected growth, more than expected growth. Set accountability targets based on progress of groups of students, not on the growth of individuals Investigating residual gain, conditional growth percentile, and quantile regression approaches to evaluating growth Analysis of covariance can be used to compare scores for groups of students for program evaluation purposes

Simple Procedure for ANCOVA for Two Student Groups in Excel Yi = individual student score in year 2 Xi = individual student score in year 1 Xavg= average in year 1 R = reliability of the assessment = .9 Calculate Perform a simple t test for the two groups using the adjusted year two scores. This method minimizes the effects of non-random assignment to treatments and regression to the mean. Xadj = R(Xi – Xavg) = .9(Xi – Xavg) Yadj = Yi-Xadj

For Further Information Eric E. Zilbert, Ph.D. Administrator Psychometrics, Evaluation, and Data Unit Assessment Development and Administration Division California Department of Education 916-445-4902 ezilbert@cde.ca.gov