AACC Mini Conference June 8-9, 2011 Developing and Selecting Student Growth Measures for Use in Teacher Evaluation Systems Joan Herman and Pete Goldschmidt CCSSO National Conference on Student Assessment Orlando, Florida June 21, 2011

Source of Presentation Developing and Selecting Assessments of Student Growth for Use in Teacher Evaluation Systems Joan L. Herman, Margaret Heritage, and Pete Goldschmidt

Focus of Attention Sophisticated statistical models proposed to estimate the relative value individual teachers add to students’ performance Little attention paid to the quality of the student assessments models use to estimate student growth

Quality of Measures Matters Generating student growth scores requires at least two assessments of student learning Carefully designed and validated to provide trustworthy evidence for evaluating teacher effectiveness Guidance to states and districts as they develop and/or select student growth measures

Purpose of Guidance Recognize the pressure on states and districts Provide an understanding of what developing/selecting quality assessment entails Support short-term and long-term planning for continuous improvement of measures

BASIC ARGUMENT JUSTIFYING USE IN TEACHER EVALUATION

Validity Validity is overarching concept that defines quality in educational measurement Concerns the extent to which an assessment measures what it is intended to measure and provides sound evidence for specific decision- making purposes. Validation involves evaluating or justifying a specific interpretation(s) or use(s) of the scores

Validity Framework Establishes the basic argument that justifies the use of student growth measures as part of teacher evaluation Lays out the essential claims within the argument that need to be justified Suggests sources of evidence for substantiating the claims Uses accumulated evidence to evaluate and improve score validity

ARGUMENTS AND PROPOSITIONS

Propositions to Justify Use of Measures for Evaluating Teacher Effectiveness

CLAIMS AND EVIDENCE

Propositions and Claims to the Validity Evaluation Proposition 1: The standards clearly define learning expectations for the subject area and each grade level. Design Claims: Clarity Feasibility Explicit progressions Evidence: Expert reviews

Propositions and Claims to the Validity Evaluation Proposition 2: The assessment instruments have been designed to yield scores that can accurately and fairly reflect student achievement of standards. Design Claims: Alignment with standards (specs and items) Fair and Accessible Replicable procedures Evidence: Expert reviews of alignment Sensitivity reviews Measurement review of administration and scoring procedures

Propositions and Claims to the Validity Evaluation Proposition 2b: The assessment instruments have been designed to yield scores that accurately and fairly reflect student growth over the course of the year. Design Claims: Sample the range of where students may start and end the school year Designed to be sensitive to instruction Evidence: Expert review

Propositions and Claims to the Validity Evaluation Proposition 3: There is evidence that the assessment scores accurately and fairly measure the learning expectations Design Claims: Psychometric analyses confirm the assessment’s blueprint Scores are sufficiently precise and reliable Scores are fair/unbiased Evidence: Psychometric analyses Content analysis Bias analyses

Propositions and Claims to the Validity Evaluation Proposition 4: There is evidence that student growth scores accurately and fairly measure student progress over the course of the year Design Claims: Score scale reflects the full distribution of where students may start and end the year Growth scores are sufficiently precise and reliable for all students Growth scores are fair/relatively free of bias Evidence: Psychometric modeling and fit statistics Sensitivity/bias analyses

Propositions and Claims to the Validity Evaluation Proposition 5: There is evidence that assessment scores represent teachers’ contribution to student growth Design Claims: Scores are instructionally sensitive Scores representing teacher contribution are sufficiently precise and reliable Scores representing teachers contributions are relatively free of bias Evidence: Research studies on instructional sensitivity Assumption checking Advanced statistical modeling

ACCUMULATED EVIDENCE

Propositions and Claims to the Validity Evaluation Validity is a matter of degree Appraisal of all claims and evidence Strengths and weaknesses Assessment and validity evidence can always be improved

GETTING STARTED

Where to start? In the beginning- will not meet all or even many of the criteria Start by being clear on learning expectations Ensure assessments developed or selected are aligned Collect evidence for propositions during operational administrations Develop long-term agenda to improve assessments

Finally….. A single assessment cannot adequately capture the multi-faceted domain of teacher effectiveness Multiple measures are essential