Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council Dr. Robert H. Meyer Research Professor and Director Value-Added Research Center University of Wisconsin-Madison February 25, 2011

Attainment and Gain Attainment – a “point in time” measure of student proficiency compares the measured proficiency rate with a predefined proficiency goal. Gain – measures average gain in student scores from one year to the next

Attainment versus Gain Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

Growth: Starting Point Matters Reading results of a cohort of students at two schools School 2006 Grade 4 Scale Score Avg. 2007 Grade 5 Average Scale Score Gain A 455 465 10 B 425* 455* 30 Grade 4 Proficient Cutoff 438 Grade 5 Proficient Cutoff 463 *Scale Score Average is below Proficient Example assumes beginning of year testing

Value-Added A kind of growth model that measures the contribution of schooling to student performance on the standardized tests Uses statistical techniques to separate the impact of schooling from other factors that may influence growth Focuses on how much students improve on the tests from one year to the next as measured in scale score points

Value-Added Model Definition A value-added model (VAM) is a quasi-experimental statistical model that yields estimates of the contribution of schools, classrooms, teachers, or other educational units to student achievement, controlling for non-school sources of student achievement growth, including prior student achievement and student and family characteristics. A VAM produces estimates of productivity under the counterfactual assumption that all schools serve the same group of students. This facilitates apples-to-apples school comparisons rather than apples-to-oranges comparisons. The objective is to facilitate valid and fair comparisons of productivity with respect to student outcomes, given that schools may serve very different student populations.

A More Transparent (and Useful) Definition of VA Value-added productivity is the difference between actual student achievement and predicted student achievement. Or, value-added productivity is the difference between actual student achievement and the average achievement of a comparable group of students (where comparability is defined by a set of characteristics such a prior achievement, poverty and ELL status).

In English x = + + + Post-on-Pre Link Posttest Pretest Student Characteristics School Effects Unobserved Factors + + + Value Added

VARC Philosophy Development and implementation of a value-added system should be structured as a continuous improvement process that allows for full participation of stakeholders Model Co-Build; Complete customization Analysis Reporting Value–added is one tool in a toolbox with multiple indicators

VARC Value-Added Partners Design of Wisconsin State Value-Added System (1989) Minneapolis (1992) Milwaukee (1996) Madison (2008) Wisconsin Value-Added System (2009) Milwaukee Area Public and Private Schools (2009) Racine (2009) Chicago (2006) Department of Education: Teacher Incentive Fund (TIF) (2006 and 2010) New York City (2009) Minnesota, North Dakota & South Dakota: Teacher Education Institutions and Districts (2009) Illinois (2010) Hillsborough County , FL (2010) Broward County, FL (2010) Atlanta (2010) Los Angeles (2010) Tulsa (2010)

Districts and States working with VARC Minneapolis Milwaukee Madison Racine Chicago New York City Los Angeles Tulsa Atlanta Hillsborough County Broward County

Measuring knowledge Many factors influence what a student learns and how their knowledge is measured A variety of measures, including (but not limited to) assessments, tell us what a student knows at a point in time. What are some ways we measure knowledge?

Measuring knowledge Large scale assessments Daily teacher assessments Local assessments used by the district Daily teacher assessments Observations MAP WKCE Diagnostic Test End-of-course Exam Daily Journal Unit Project After-school Activities Hands-on Project

The Simple Logic of Value-Added Analysis School Value-Added Report School specific data Grade level value-added Comparison Value-Added Reports Compare a school to other schools in the district, CESA, or state Also allows for grade level comparisons Tabular Data available for School Report and Comparison Reports

-How to read post-on-pre graph -Hallmark of growth: 7th grade made strongly predicts 8th grade math (as evident in the strong positive association) -Noise in test scores

Attainment and Value-Added

How complex should a value-added model be? Rule: "Simpler is better, unless it is wrong.“ Implies need for “quality of indicator/ quality of model” diagnostics.

Model Features Demographics Posttest on pretest link Measurement error Student mobility: dose model Classroom vs. teacher: unit vs. agent Differential effects Selection bias mitigation: longitudinal data Test property analysis

MAP vs. ISAT MAP dates: September, January, May MAP: uses Rasch equating ISAT: 3PL MAP: slightly higher reliability - ~0.96 in math, ~0.94 in reading ISAT math ~0.93, reading ~0.9 Cut scores on MAP are determined by equipercentile equating to ISAT

Minimal correlation between initial status and value-added

Grade-Level Statewide Results Subject State Grade N Mean Score SD of Score Reliability of Value-Added Math MN 3 59460 200.0 13.9 0.901 4 58346 210.8 14.6 0.916 5 57053 219.9 16.2 0.907 6 52400 226.7 16.5 0.873 7 47985 232.1 17.4 0.883 8 44227 236.4 17.9 0.823 9 26512 238.8 18.2 0.826

Grade-Level Statewide Results Subject State Grade N Mean Score SD of Score Reliability of Value-Added Math WI 3 43289 199.9 13.2 0.820 4 44140 209.3 13.7 0.842 5 43822 217.3 14.8 0.849 6 47004 222.7 15.2 0.836 7 44549 228.4 16.0 0.837 8 43246 233.1 16.8 0.865 9 26427 234.0 17.7 0.862

Grade-Level Statewide Results Subject State Grade N Mean Score SD of Score Reliability of Value-Added Reading WI 3 43139 194.8 15.1 0.736 4 43671 202.9 14.4 0.780 5 43668 209.7 13.8 0.737 6 46233 214.0 14.2 0.719 7 44616 218.3 14.0 0.792 8 43251 221.7 14.1 0.826 9 28066 223.3 0.843

MPS and MMSD Value-Added compared to Wisconsin 6th to 7th Grade (Nov 2006 – Nov 2007) Mathematics – State VA Model School Effects MPS School Effects MMSD School Effects School/District VA Productivity Parameters in WKCE Scale Score Units (Relative to State)

for more information about VARC and value-added Visit the VARC Website http://varc.wceruw.org/ for more information about VARC and value-added