1. AACC Mini Conference
June 8-9, 2011
Developing and Selecting Student Growth Measures for Use in Teacher Evaluation Systems
Joan Herman and Pete Goldschmidt
CCSSO National Conference on Student Assessment
Orlando, Florida
June 21, 2011

2. Source of Presentation
Developing and Selecting Assessments of Student Growth for Use in Teacher Evaluation Systems
Joan L. Herman, Margaret Heritage, and Pete Goldschmidt

3. Focus of Attention
Sophisticated statistical models proposed to estimate the relative value individual teachers add to students’ performance
Little attention paid to the quality of the student assessments models use to estimate student growth

4. Quality of Measures Matters
Generating student growth scores requires at least two assessments of student learning
Carefully designed and validated to provide trustworthy evidence for evaluating teacher effectiveness
Guidance to states and districts as they develop and/or select student growth measures

5. Purpose of Guidance Recognize the pressure on states and districts
Provide an understanding of what developing/selecting quality assessment entails
Support short-term and long-term planning for continuous improvement of measures

6. BASIC ARGUMENT
JUSTIFYING USE IN
TEACHER EVALUATION

7. Validity
Validity is overarching concept that defines quality in educational measurement
Concerns the extent to which an assessment measures what it is intended to measure and provides sound evidence for specific decision- making purposes.
Validation involves evaluating or justifying a specific interpretation(s) or use(s) of the scores

8. Validity Framework
Establishes the basic argument that justifies the use of student growth measures as part of teacher evaluation
Lays out the essential claims within the argument that need to be justified
Suggests sources of evidence for substantiating the claims
Uses accumulated evidence to evaluate and improve score validity

9. ARGUMENTS AND PROPOSITIONS

10. Propositions to Justify Use of Measures for Evaluating Teacher Effectiveness

11. CLAIMS AND EVIDENCE

12. Propositions and Claims to the Validity Evaluation
Proposition 1: The standards clearly define learning expectations
for the subject area and each grade level.
Design Claims:
Clarity
Feasibility
Explicit progressions
Evidence:
Expert reviews

13. Propositions and Claims to the Validity Evaluation
Proposition 2: The assessment instruments have been designed
to yield scores that can accurately and fairly reflect student
achievement of standards.
Design Claims:
Alignment with standards
(specs and items)
Fair and Accessible
Replicable procedures
Evidence:
Expert reviews of alignment
Sensitivity reviews
Measurement review of
administration and scoring
procedures

14. Propositions and Claims to the Validity Evaluation
Proposition 2b: The assessment instruments have been designed
to yield scores that accurately and fairly reflect student growth
over the course of the year.
Design Claims:
Sample the range of where
students may start and end
the school year
Designed to be sensitive to
instruction
Evidence:
Expert review

15. Propositions and Claims to the Validity Evaluation
Proposition 3: There is evidence that the assessment scores accurately and fairly measure the learning expectations
Design Claims:
Psychometric analyses confirm
the assessment’s blueprint
Scores are sufficiently precise
and reliable
Scores are fair/unbiased
Evidence:
Psychometric analyses
Content analysis
Bias analyses

16. Propositions and Claims to the Validity Evaluation
Proposition 4: There is evidence that student growth scores accurately and fairly measure student progress over the course of the year
Design Claims:
Score scale reflects the full
distribution of where students
may start and end the year
Growth scores are sufficiently
precise and reliable for all students
Growth scores are fair/relatively
free of bias
Evidence:
Psychometric modeling and
fit statistics
Sensitivity/bias analyses

17. Propositions and Claims to the Validity Evaluation
Proposition 5: There is evidence that assessment scores represent teachers’ contribution to student growth
Design Claims:
Scores are instructionally sensitive
Scores representing teacher
contribution are sufficiently
precise and reliable
Scores representing teachers
contributions are relatively free
of bias
Evidence:
Research studies on
instructional sensitivity
Assumption checking
Advanced statistical modeling

18. ACCUMULATED EVIDENCE

19. Propositions and Claims to the Validity Evaluation
Validity is a matter of degree
Appraisal of all claims and evidence
Strengths and weaknesses
Assessment and validity evidence can always be improved

20. GETTING STARTED

21. Where to start?
In the beginning- will not meet all or even many of the criteria
Start by being clear on learning expectations
Ensure assessments developed or selected are aligned
Collect evidence for propositions during operational administrations
Develop long-term agenda to improve assessments

22. Finally…..
A single assessment cannot adequately capture the multi-faceted domain of teacher effectiveness
Multiple measures are essential