Value Added in CPS

What is value added? A measure of the contribution of schooling to student performance Uses statistical techniques to isolate the impact of schooling from other factors Focuses on how much students improve from one year to the next

Demographic adjustments Value added makes adjustments for demographics of schools and classrooms Adjustments determined by relationships between growth, student characteristics Adjustments measure partial differences in growth across groups district-wide

Some schools with low percent meet/exceed are high value-added schools in which students grow

Value added in many domains Annual state assessments Focus on year-to-year student improvement Short-term assessments Focus on short-term student improvement Potentially faster turnaround High school assessments Explore/PLAN/ACT, for example Focus on improvement in high school

Value added in CPS Based on ISAT for grades 3 through 8 Analyzes students’ ISAT scores, demographics, and schools attended Schools and classrooms where students improve more (relative to similar students) identified as high value added Extra ISAT points gained by students at a school/classroom on average relative to observably similar students across district

Alternative understanding Average student gain on ISAT relative to district average, with adjustments for: Shape of the test scale (Prior ISAT score) Grade level Gender, mobility, free/reduced-price lunch, race/ethnicity, disability, language proficiency, homelessness, other-subject pretest Enrollment in multiple schools or classrooms

Regression model (in English) Post-on-Pre Link x Posttest = Pretest School and Classroom Effects Student Characteristics Unobserved Factors + + + Value Added

Student characteristics Gender Race/ethnicity Free or reduced-price lunch Language proficiency (by Access score) Disability (by disability type) Mobility Homelessness Other-subject pretest

Why include student characteristics? One goal of value-added analysis is to be as fair as possible We want to remove the effect of factors that were not caused by the school during the specific period we are evaluating

What do we want to evaluate? Related to the school Not related to the school Examples Curriculum Classroom teacher School culture Math pull-out program at school Structure of lessons in school Safety at the school Examples Student motivation English Language Learner Status At home support Household financial resources Learning disability Prior knowledge Value added reflects the impact of these factors These factors need to be measured and isolated

Controlling for other factors Students bring different resources to the classroom. These factors can affect growth, so we want to remove the effects of these non-school factors. Not related to the school Examples Student motivation English Language Learner Status At home support Household financial resources Learning disability Prior knowledge These factors need to be measured and isolated

Controlling for other factors In order to include a characteristic in the model, we must have data on that characteristic for all students. Some characteristics are harder to measure and collect than others. The data that we do have available can tell us something about the effect of data we would like to have.

Controlling for other factors What we want Household financial resources What we have Free or reduced-price lunch Related data For example, we can use free or reduced-price lunch as a substitute for our ideal data about household finances in our calculations. Some possible talking points on the question: Less likely to have external help (tutor service, etc.) Reduced access to resources at home (books, computer, etc.) Reduced parental availability for homework / study help (multiple jobs?) Doug Harris’s book points (summer learning loss, etc.) From Doug Harris Book: “Should value-added models take into account student race, income, and other student factors in estimating value-added? accountability. In one respect, the issue boils down to whether taking This is one of the more controversial questions in using value-added for (see Figure 1.1). But it turns out that these factors are less closely income are after all closely related to student attainment on test scores into account prior achievement is enough. Race, ethnicity, and related to achievement growth. are associated with achievement in every grade, then the association The reason for this should be intuitive: if student demographics scored 60 points in one year and 80 points in the next year and grade, simple, concrete example to highlight this: Suppose that a student should largely “cancel out” when subtracting the two. Here is a make sure the child does her homework each night (in all years). This explaining these scores is that fact that the students’ parents do not for growth of 20 points. Now, suppose that part of the reason (instead of 60 and 8) if the student had done her homework. Notice would have been. So, the student would have scored 65 and 85 might reduce the student’s score by 5 points in each year from what it on scores should cancel out in this way. as the influence on student scores is constant over time, the influence that the growth is exactly the same in both cases—20 points. So long exists in 1st grade, it grows by about 30 percent between 1st and 5th rates. One study finds that while most of the gap by family income But minority and low-income students do still learn at slower years when students are not in school. This is most likely because school, but to the “summer learning loss,” the period between school grade.1 Almost all of this, in turn is due, not to what happens in the beginning of the school year, the summer learning loss is learning growth. Because standardized tests are not administered at the same factors creating the starting gate inequalities also affect therefore problematic. An accurate measure of value-added must take learning loss is substantially outside the control of schools and embedded within the student growth measures. Further, the summer into account all of the factors outside their control. that some see it as reducing expectations for these students. There The concern with accounting for race and income, however, is are legitimate differences of opinion on this point, but let me clarify schools can get by giving less effort to raise achievement for this could mean that accounting for race and income means that the two different meanings of “lower expectations.” On the one hand, point of value-added is to create an even playing field and one that and income do not lower expectations in this sense. In fact, the whole disadvantaged students. Value-added measures that account for race provides incentives for schools to help all students. schools serving disadvantaged students will have the same measured Alternatively, some interpret “lower expectations” to mean that then it is a legitimate point. Value-added models that adjust less student learning. If this is what is meant by lower expectations, performance as schools serving advantaged students while generating performance. Again, this just reflects the fact that schools should not of disadvantaged students to reach the same level of school predictions based on student race and income do require less learning into that category. home environments and other factors affecting students clearly fall be held accountable for factors outside their control and students’ different groups, value-added measures can be designed to place as concern is about how much emphasis schools place on achievement of There is some potential middle ground on this issue. If the in growth for a low-income student counts twice as much as for a example, we could design the accountability system so that 10 points much or as little weight on disadvantaged students as we wish. For between student race and income and student achievement growth they also end up estimating the statistical relationship (correlation) high-income student. Also, when statisticians estimate value-added, (see Oakville example above). Districts could use the measured role of disadvantage is associated with lower growth, then this might to overcome racial and income achievement gaps.2 If socio-economic student disadvantage as an indicator of how well the district is doing expectations and even greater effort for those students. rather than lowering expectations, it can be a driver of higher motivate districts to try even harder to address the issue. That is, The same goes for disadvantaged students. To the degree that reduce the problem of driving out teachers of low-attainment schools. It is also worth recalling that value-added measures can help those students at a disadvantage. growth, failing to account for those factors will place the teachers of student race and income are associated with their achievement to the Cardinal Rule: Hold people accountable for what they can when predicting each school’s achievement. The reason comes back In my view, it is best to account for student race and income does not mean that schools can get by with giving less effort to these students will receive higher performance ratings with less growth, it control. While this means that schools serving disadvantaged statistical relationship between student disadvantage and student growth of racial minorities and low-income students, and use the students. If, in addition, we give disproportionate weight to the expectations—and outcomes—for these students rather than lower accounting for student disadvantage would seems to raise achievement growth as a motivation to address the gap, then rationale behind it.” them. I cannot prove that, unfortunately, but there is a strong

Adjustments are based on real data Why is it important that VARC uses student test scores to calculate adjustment factors? We do not have a preconceived notion of which student subgroups will grow faster than others We want to be as fair as possible when evaluating school performance Student subgroups perform differently on each subject area from year-to-year We want our adjustments to apply specifically to the situation we are evaluating

Multiple regression Measures effects of each variable on posttest controlling for all other variables Effect of pretest on posttest controls for student characteristics, schools Effects of student characteristics on posttest control for pretest, schools Effects of schools (value added) on posttest control for pretest, student characteristics All effects measured simultaneously

Dosage Accounts for students changing schools and classrooms Students enrolled in a school or classroom for a fraction of a year get a fractional “dose” of the school’s or classroom’s effect Apportions student growth among schools and classrooms enrolled in the same year

Pretest measurement error Pretest measures student attainment in previous year with measurement error Models that ignore this will bias in favor of high-attainment schools and classrooms Measurement error is accounted for in VA model to correctly account for pretest Using approaches in Fuller (1987) Ensures against bias

Models that correct for measurement error avoid biasing in favor of schools and classrooms with high initial scores

Value added model All of these features ensure that value added reflects the results of schooling on student achievement Value added uses the data available to measure the impact of schools and classrooms as accurately, fairly, and realistically as possible

Work in progress Classroom-level value added Measures student growth within classrooms Differential effects value added Measures growth among students of a particular group (ELL, disability, etc.) in a school or classroom Value added from other assessments Scantron (short-term) Explore/PLAN/ACT (high school)