1. Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council
Dr. Robert H. Meyer
Research Professor and Director
Value-Added Research Center
University of Wisconsin-Madison
February 25, 2011

2. Attainment and Gain
Attainment – a “point in time” measure of student proficiency
compares the measured proficiency rate with a predefined proficiency goal.
Gain – measures average gain in student scores from one year to the next

3. Attainment versus Gain
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8

4. Growth: Starting Point Matters
Reading results of a cohort of students at two schools
School
2006 Grade 4
Scale Score Avg.
2007 Grade 5
Average
Scale Score Gain A
455
465 10 B
425*
455* 30
Grade 4 Proficient Cutoff 438
Grade 5 Proficient Cutoff 463
*Scale Score Average is below Proficient
Example assumes beginning of year testing

5. Value-Added
A kind of growth model that measures the contribution of schooling to student performance on the standardized tests
Uses statistical techniques to separate the impact of schooling from other factors that may influence growth
Focuses on how much students improve on the tests from one year to the next as measured in scale score points

6. Value-Added Model Definition
A value-added model (VAM) is a quasi-experimental statistical model that yields estimates of the contribution of schools, classrooms, teachers, or other educational units to student achievement, controlling for non-school sources of student achievement growth, including prior student achievement and student and family characteristics.
A VAM produces estimates of productivity under the counterfactual assumption that all schools serve the same group of students. This facilitates apples-to-apples school comparisons rather than apples-to-oranges comparisons.
The objective is to facilitate valid and fair comparisons of productivity with respect to student outcomes, given that schools may serve very different student populations.

7. A More Transparent (and Useful) Definition of VA
Value-added productivity is the difference between actual student achievement and predicted student achievement.
Or, value-added productivity is the difference between actual student achievement and the average achievement of a comparable group of students (where comparability is defined by a set of characteristics such a prior achievement, poverty and ELL status).

8. In English x = + + + Post-on-Pre Link Posttest Pretest Student
Characteristics
School Effects
Unobserved
Factors + + +
Value
Added

9. VARC Philosophy
Development and implementation of a value-added system should be structured as a continuous improvement process that allows for full participation of stakeholders
Model Co-Build; Complete customization
Analysis
Reporting
Value–added is one tool in a toolbox with multiple indicators

10. VARC Value-Added Partners
Design of Wisconsin State Value-Added System (1989)
Minneapolis (1992)
Milwaukee (1996)
Madison (2008)
Wisconsin Value-Added System (2009)
Milwaukee Area Public and Private Schools (2009)
Racine (2009)
Chicago (2006)
Department of Education: Teacher Incentive Fund (TIF) (2006 and 2010)
New York City (2009)
Minnesota, North Dakota & South Dakota: Teacher Education Institutions and Districts (2009)
Illinois (2010)
Hillsborough County , FL (2010)
Broward County, FL (2010)
Atlanta (2010)
Los Angeles (2010)
Tulsa (2010)

11. Districts and States working with VARC
Minneapolis
Milwaukee
Madison
Racine
Chicago
New York City
Los Angeles
Tulsa
Atlanta
Hillsborough
County
Broward
County

12. Measuring knowledge
Many factors influence what a student learns and how their knowledge is measured
A variety of measures, including (but not limited to) assessments, tell us what a student knows at a point in time.
What are some ways we measure knowledge?

13. Measuring knowledge Large scale assessments Daily teacher assessments
Local assessments used by the district
Daily teacher assessments
Observations
MAP
WKCE
Diagnostic Test
End-of-course Exam
Daily Journal
Unit Project
After-school Activities
Hands-on Project

14. The Simple Logic of Value-Added Analysis
School Value-Added Report
School specific data
Grade level value-added
Comparison Value-Added Reports
Compare a school to other schools in the district, CESA, or state
Also allows for grade level comparisons
Tabular Data available for School Report and Comparison Reports

15. -How to read post-on-pre graph
-Hallmark of growth: 7th grade made strongly predicts 8th grade math (as evident in the strong positive association)
-Noise in test scores

17. Attainment and Value-Added

18. How complex should a value-added model be?
Rule: "Simpler is better, unless it is wrong.“
Implies need for “quality of indicator/ quality of model” diagnostics.

19. Model Features Demographics Posttest on pretest link Measurement error
Student mobility: dose model
Classroom vs. teacher: unit vs. agent
Differential effects
Selection bias mitigation: longitudinal data
Test property analysis

20. MAP vs. ISAT MAP dates: September, January, May
MAP: uses Rasch equating
ISAT: 3PL
MAP: slightly higher reliability - ~0.96 in math, ~0.94 in reading
ISAT math ~0.93, reading ~0.9
Cut scores on MAP are determined by equipercentile equating to ISAT

35. Minimal correlation between initial status and value-added

39. Grade-Level Statewide Results
Subject
State
Grade N
Mean Score
SD of Score
Reliability of Value-Added
Math MN 3
59460
200.0
13.9
0.901 4
58346
210.8
14.6
0.916 5
57053
219.9
16.2
0.907 6
52400
226.7
16.5
0.873 7
47985
232.1
17.4
0.883 8
44227
236.4
17.9
0.823 9
26512
238.8
18.2
0.826

40. Grade-Level Statewide Results
Subject
State
Grade N
Mean Score
SD of Score
Reliability of Value-Added
Math WI 3
43289
199.9
13.2
0.820 4
44140
209.3
13.7
0.842 5
43822
217.3
14.8
0.849 6
47004
222.7
15.2
0.836 7
44549
228.4
16.0
0.837 8
43246
233.1
16.8
0.865 9
26427
234.0
17.7
0.862

41. Grade-Level Statewide Results
Subject
State
Grade N
Mean Score
SD of Score
Reliability of Value-Added
Reading WI 3
43139
194.8
15.1
0.736 4
43671
202.9
14.4
0.780 5
43668
209.7
13.8
0.737 6
46233
214.0
14.2
0.719 7
44616
218.3
14.0
0.792 8
43251
221.7
14.1
0.826 9
28066
223.3
0.843

42. MPS and MMSD Value-Added compared to Wisconsin
6th to 7th Grade (Nov 2006 – Nov 2007) Mathematics – State VA Model School Effects
MPS
School Effects
MMSD
School Effects
School/District VA Productivity Parameters in WKCE Scale Score Units
(Relative to State)

43. for more information about VARC and value-added
Visit the VARC Website
for more information about VARC and value-added