From Evidence to Action: A Seamless Process in Formative Assessment? Margaret Heritage Jinok Kim Terry Vendlinski American Educational Research Association Annual Meeting New York, NY - March 24-28, 2008

Evidence to Action: A Seamless Process? A Generalizability study of measures of teacher knowledge Teachers can draw appropriate inferences from evidence Difficulties in deciding next instructional steps

Formative Assessment

An ongoing process to gather evidence and provide feedback about learning Use the feedback to identify the gap between current learning and desired goals (Sadler, 1989)

Teacher Knowledge Measures

Measures of Teacher Knowledge for Teaching Mathematics Drawn from POWERSOURCE Conceptualize teacher knowledge as central to and embedded in the everyday practice of teaching Key principles underlying mastery of algebra I Distributive property Rational number equivalence Solving equations

Student Assessment

Teacher Knowledge Measures 1. What is the key principle that these assessments address? Why do students need to understand this principle for algebra I? 2. What inferences would you draw from this student’s responses? What does this student know? What does this student not know? 3. If this student were in your class, based on your responses to questions 2 and 3, what would you do next in your instruction? 4. If you were this child’s teacher, what written feedback would you give to this student?

Scoring Rubric: Next Instructional Steps for Teaching the Distributive Property ScoreCriteria 4  Explain the distributive property as repeated addition  Explain factoring as distribution in reverse  Model the use of the distributive property with whole numbers  Model generalizing to other numbers and variables 3Either  Explain the distributive property as repeated addition Or  Explain factoring as distribution in reverse  Model the use of the distributive property with at least whole number 2  Explain procedures for how to use the distributive property, equating procedures with the order of operations 1  No explanation of the distributive property

Generalizability (G) Study

G-Theory Framework Object of Measurement: Teachers’ pedagogical knowledge in mathematics Aims: To assess the generalizability of a task score in a single condition to an average score in a countless combination of conditions Determine the source of error if the task score is not generalizable

Initial Assumptions: 3 Potential Sources of Measurement Error Variations in raters’ scoring of a teacher’s responses A teacher’s knowledge of principles may vary A teacher’s score may vary depending on the task

Sample of Participating Teachers 118 6th grade teachers from a variety of districts in Los Angeles County 90% credentialed for K-6, 60% for 7-8, 34% grade 9, 20% for % general credential 14% mathematics credential

Analysis o xr x p x to xr x p x t object of measurement (teacher) raterprincipletype of task

Variability Findings: Main Effects Main effect of raters = 0.2% The impact of raters is negligible Main effect of type of task = 25% Teachers’ scores may not be generalizable across tasks

Variability Findings: Interactions Interactions of object of measurement (teacher), principle, and task = 39.4% Some principle-task combinations are more difficult for some teachers than for other teachers Interactions of principle and task = 8.2% Some combinations of principle and task more difficult

VariableNMeanStd DevMinMax Principle: distributive property Task: identifying key principle Task: evaluating student understanding Task: planning next instruction  Principle: solving equations Task: identifying key principle Task: evaluating student understanding Task: planning next instruction  Principle: rational number equivalence Task: identifying key principle Task: evaluating student understanding Task: planning next instruction 

Improving the Translation of Evidence to Action

Know Which Way to Go?

Learning from Teaching

And Finally … Challenge of developing valid and reliable measures of teacher knowledge Using assessment information to plan subsequent instruction tends to be the most difficult task Evidence may provide the basis for action but cannot itself “form” the action