1. CAASPP Results 2015 to 2016
Santa Clara Assessment and Accountability Network May 26, Eric E, Zilbert Administrator, Psychometrics, Evaluation and Data Office Assessment Development and Administration Division California Department of Education

2. Presentation Overview
Background and Introduction
Properties of the Smarter Balanced Scale
Discussion of Change Scores
Scale Score Distributions by Grade, 2016
Change Score Distributions by Grade
Normative Growth by Scale Score in 2015
Changes in Relative Standing

3. Change is the Only Constant
2016 – Second Administration of the Smarter Balanced Assessments in English language arts and mathematics
Features of the Assessments:
Computer Adaptive Component
Performance Task Component
Continuous Reporting Scale
4 Achievement Levels set by Grade Level
Each grade level has a pre-specified lowest obtainable scale score (LOSS) and highest obtainable scale score (HOSS) for each grade.
The results of the 2016 assessments provide the first opportunity to examine score changes for students on the scales.

4. About Change Scores
The simple difference between last year’s score and this year’s score needs to be interpreted in light of the following:
The variability of change scores is significantly greater than that for scores in a single year. The error is essentially compounded.
Score changes are impacted by regression to the mean for very high scoring and very low scoring students.
The meaning of a change score is somewhat different for different parts of the scale and for different grades.
Change scores are best interpreted in terms of where the student started in the previous year.

5. Summary Statistics for Smarter Balanced Assessments 2015

6. Change in Mean Smarter Balanced Scale Scores 2015-2016

7. Change in Mean Smarter Balanced Scale Scores 2015-2016

8. Properties of the Smarter Balanced Scales

9. Properties of the Smarter Balanced Scales

10. ELA Scale Score Distributions by Grade

11. Math Scale Score Distributions by Grade

12. 2015-16 ELA Scale Score Changes by Grade

13. 2015-16 ELA Scale Score Changes

14. 2015-16 Math Scale Score Changes

15. 2015-16 Math Scale Score Changes

16. Change and Prior Performance
Calculated average performance by scale score in the previous year.
Estimated decile average change by picking students within 5 scale score points +/- of the percentile reported
Calculated at the 10th, 25th, 50th, 75th, and 90th percentiles of 2015 performance.
Demonstrate the likelihood of growth in future years with no more than 81% accuracy (test reliability=.9).

17. Statewide Average Change in Scale Score by Scale Score in 2015: ELA

18. Normative Growth by Scale Score in 2015 ELA

19. Statewide Average Change in Scale Score by Scale Score in 2015: Mathematics

20. Normative Growth by Scale Score in 2015 Mathematics

21. What to do?
Focus on ranges of growth: little or no growth, expected growth, more than expected growth.
Set accountability targets based on progress of groups of students, not on the growth of individuals
Investigating residual gain, conditional growth percentile, and quantile regression approaches to evaluating growth
Analysis of covariance can be used to compare scores for groups of students for program evaluation purposes

22. Simple Procedure for ANCOVA for Two Student Groups in Excel
Yi = individual student score in year 2 Xi = individual student score in year 1 Xavg= average in year 1 R = reliability of the assessment = .9 Calculate Perform a simple t test for the two groups using the adjusted year two scores. This method minimizes the effects of non-random assignment to treatments and regression to the mean.
Xadj = R(Xi – Xavg) = .9(Xi – Xavg)
Yadj = Yi-Xadj

23. For Further Information
Eric E. Zilbert, Ph.D.
Administrator
Psychometrics, Evaluation, and Data Unit
Assessment Development and Administration Division
California Department of Education