Why Measure Performance with Scale Scores Why Measure Performance with Scale Scores? Dominic Zarecki, Senior Data Analyst Date:   Dec-01-2016 Time:  4:30-5:00 PM Room: Carr 

Agenda “Distance from Met” is: Nuanced Publicly available Fair to all grades Fair to all schools

The Foundation: “Distance from Met” Improved from 24 below to 22 above Met Declined from 44 below to 78 below Met Example student score report created by CDE, found here: http://www.cde.ca.gov/ta/tg/ca/documents/caaspp16scorerpt.pdf Distance from Met measures how many scale score points a student is above or below the Level 3 (i.e. Met) cut point. This allows us to standardize scores across grades in a way that is more nuanced than the four SBAC levels. On the far right, we see that Sophie is 78 points Met in Math. While this is Level 1, it is right near the top of Level 1. (If you look really closely, you’ll notice that the standard error indicates that Sophia could very well score in Level 2 if she took the same test again. Change in Distance from Met = to what extent is a student outpacing the academic treadmill? The example on the right is particularly helpful. We see that Sophia’s scale score improved slightly, from 2392 to 2407. But the academic treadmill moved up even more; if Sophia had made average growth, she would have grown by even more scale score points. We can see this clearly by looking at her change in distance from Met: she dropped from 44 below to 78 below. Distance from Met also allows us to see improvement within levels. This is especially helpful with level 1, where a student can make a lot of progress but still not yet be at level 2.

How do we calculate “Distance from Met” for schools? A three-step process tells us how far the average student in a school is from the Met cut point: Calculate the Gap: school average minus Met cut point for each grade and subject Weight by student: Gap multiplied by the percent of students in that grade and subject Create one number: Add the values from all grades, then average the ELA and Math scores For each subject, we calculate the weighted average of DFM scores across all grades in a school. We weight by the number of students with valid test scores. This produces the same average DFM for each subject that we would obtain with student-level data. We then take the average of the ELA and Math DFM scores to obtain one overall DFM score for each school. The average ranked school in 2015-16 had a DFM of 23 points below Met. The standard deviation was 46 points, and DFM ranged from 215 points below Met to 151 points above Met. To make State Ranks, we divide schools into deciles based on their DFM. For the ranges, see: http://www.cde.ca.gov/ta/tg/ca/sbscalerange.asp The research file is available here: http://caaspp.cde.ca.gov/sb2016/ResearchFileList

Surprisingly even across grades! ELA DFM Math DFM Overall DFM 3 -18 -11 -15 4 -19 -25 -22 5 -6 -43 6 -12 -28 7 -10 -42 -26 8 -8 -45 11 17 -60 Given that the scale is over 600 points wide, a difference of up to 13 points across grades is extremely small (2% of range). The primary assumption this makes is that DFM is an equally difficult standard across grades. The table below shows the average DFM of students statewide by subject and overall. We see that while DFM for each subject varies significantly by grade, overall DFM is extremely uniform across grade levels. With the exception of third grade, average DFM scores only range from -22 to -28. This means that DFM measures schools fairly regardless of the grades they serve.

DFM and % Met are 0.97 correlated… 39% Met/Exceeded (57th percentile) -26 DFM (59th percentile) Distribution of 2015 CAASPP Scores State Mean Met Cut Point -33 DFM

… but can diverge dramatically 38% Met/Exceeded (47th percentile) -17 DFM (60th percentile) Distribution of 2016 CAASPP Scores State Mean Met Cut Point -31 DFM

Thank you Dominic Zarecki, Senior Data Analyst (dzarecki@fortuneschool.us)