The Design Phase: Using Evidence-Centered Assessment Design Monty Python argument

Workshop Flow The construct of MKT –Gain familiarity with the construct of MKT –Examine available MKT instruments in the field Assessment Design –Gain familiarity with the Evidence-Centered Design approach –Begin to design a framework for your own assessment Assessment Development –Begin to create your own assessment items in line with your framework Assessment Validation –Learn basic tools for how to refine and validate an assessment Plan next steps for using assessments

The PADI Project – padi.sri.com

What is Evidence-Centered Design? Approach developed by ETS Uses “evidentiary reasoning” to design the underlying principles of an assessment Answers these questions: –What complex of KSAs should be assessed? –What types of evidence would we need to show a test-taker has these KSAs? –What kinds of assessment items would allow us to gather this evidence? Provides a set of tools for structuring a systematic approach to doing this

Phases of ECD 1)Domain Analysis Big-picture narrative of the knowledge domain 2)Domain Modeling Organization of the information in the domain in terms of: The aspect of proficiency of the test-taker in the domain The kinds of things the test-taker might do to provide evidence of their proficiency The kinds of situations that might make it possible to provide such evidence 3)Conceptual Assessment Framework Blueprint for the actual assessment that takes all of this into account 4)The Assessment

Phases of ECD for Assessing MKT 1)Domain Analysis of MKT Big-picture narrative of the knowledge domain MKT Interpreting unconventional forms or representations Choosing problems and examples that can illustrate key curricular ideas Differentiating between colloquial and mathematical uses of language Linking precise aspects of representations Understanding implications of models and representations Evaluating mathematical statements

Phases of ECD for Assessing MKT 2)Domain Modeling Organization of the information in the domain in terms of: The aspect of proficiency of the test-taker in the domain The kinds of things the test-taker might do to provide evidence of their proficiency The kinds of situations that might make it possible to provide such evidence Create a “Design Pattern”

Phases of ECD for Assessing MKT 3)Conceptual Assessment Framework Blueprint for the actual assessment that takes all of this into account Teacher Model

Types of Relationships to Curriculum/PD Assessment Development Curriculum/PD Development Assessment Development Curriculum/PD Development Assessment Development Curriculum/PD Development

Adapting Off-the-Shelf Instruments Start with this process for your own assessment needs Evaluate the fit of the instrument for your needs Important: Do not simply mix and match!

Attributes of a Design Pattern Framing infoTitle, Summary, Rationale What the test-taker should know Focal Knowledge, Skills, and Abilities Evidence we can collect to show they know it Potential observations Potential work products Potential rubrics Kinds of situations that can evoke this evidence Characteristic features Variable features Other infoExemplar items, Online resources, References, Misc

Examples of Design Patterns MKT for SimCalc Co-construct as a group Do your own

Foundations of proportionality Proportionality includes: linearity, rate, function, slope in graphs, interpreting tables with an underlying rate Find the missing number Consider the function

NCTM Curriculum Focal Points Grade 7: Developing an understanding of and applying proportionality, including similarity. Students extend their work with ratios to develop an understanding of proportionality that they apply to solve single and multistep problems in numerous contexts. They use ratio and proportionality to solve a wide variety of percent problems, including problems involving discounts, interest, taxes, tips, and percent increase or decrease. They also solve problems about similar objects (including figures) by using scale factors that relate corresponding lengths of the objects or by using the fact that relationships of lengths within an object are preserved in similar objects. Students graph proportional relationships and identify the unit rate as the slope of the related line. They distinguish proportional relationships (y/x = k, or y = kx) from other relationships, including inverse proportionality (xy = k, or y = k/x). Grade 8: Analyzing and representing linear functions and solving linear equations and systems of linear equations. Students use linear functions, linear equations, and systems of linear equations to represent, analyze, and solve a variety of problems. They recognize a proportion (y/x = k, or y = kx) as a special case of a linear equation of the form y = mx + b, understanding that the constant of proportionality (k) is the slope and the resulting graph is a line through the origin. Students understand that the slope (m) of a line is a constant rate of change, so if the input, or x-coordinate, changes by a specific amount, a, the output, or y-coordinate, changes by the amount ma. Students translate among verbal, tabular, graphical, and algebraic representations of functions (recognizing that tabular and graphical representations are usually only partial representations), and they describe how such aspects of a function as slope and y-intercept appear in different representations. Students solve systems of two linear equations in two variables and relate the systems to pairs of lines that intersect, are parallel, or are the same line, in the plane. Students use linear equations, systems of linear equations, linear functions, and their understanding of the slope of a line to analyze situations and solve problems. Students graph proportional relationships and identify the unit rate as the slope of the related line. They distinguish proportional relationships (y/x = k, or y = kx) from other relationships, including inverse proportionality (xy = k, or y = k/x). They recognize a proportion as a special case of a linear equation of the form y = mx + b, understanding that the constant of proportionality (k) is the slope and the resulting graph is a line through the origin. Students translate among verbal, tabular, graphical, and algebraic representations of functions.

KSAs Common Content Knowledge Proportionality concepts Specialized Content Knowledge Interpreting unconventional forms or representations Choosing problems and examples that can illustrate key curricular ideas Differentiating between colloquial and mathematical uses of language Linking precise aspects of representations Understanding implications of models and representations Evaluating mathematical statements

Design pattern example#2, Co- constructedCo- constructed

Workshop Flow The construct of MKT –Gain familiarity with the construct of MKT –Examine available MKT instruments in the field Assessment Design –Gain familiarity with the Evidence-Centered Design approach –Begin to design a framework for your own assessment Assessment Development –Begin to create your own assessment items in line with your framework Assessment Validation –Learn basic tools for how to refine and validate an assessment Plan next steps for using assessments

Activity #2 Create a Design Pattern Find Activity #2 in your binder Pick a design pattern topic Work on your own, with a partner, or with a small group Complete the Design Pattern form Feedback / Review Process Discussion to follow –Show-and-tell of 2 or 3 Design Patterns –Your insights, questions, challenges

Some Useful References (all available on the web) Baxter, G. P., & Mislevy, R. J. (2005). The Case for an Integrated Design Framework for Assessing Science Inquiry. PADI Technical Report 5. Menlo Park, CA: SRI International. Embretson, S. E. (Ed) (1985). Test Design: Developments in psychology and psychometrics. New York: Academic Press, Inc. Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to Evidence-Centered Design. CRESST Technical Paper Series. Los Angeles, CA: CRESST. Mislevy, R. J, Hamel, L. et al. (2003). Design Patterns for Assessing Science Inquiry. PADI Technical Report 1. Menlo Park, CA: SRI International. PADI Website:

Feedback What worked? What didn’t work? What do you still want to learn?

Welcome back!

Q1. Definition of MKT “By mathematical knowledge for teaching, we mean the mathematical knowledge used to carry out the work of teaching mathematics. Examples of this “work of teaching” include explaining terms and concepts to students, interpreting students’ statements and solutions, judging and correcting textbook treatments of particular topics, using representations accurately in the classroom, and effects of teachers’ mathematical knowledge on student achievement providing students with examples of mathematical concepts, algorithms, or proofs” (Hill, Rowan, & Ball, 2005). “We look across mathematical knowledge needed for or used in teaching, including “pure” content knowledge as taught in secondary, undergraduate, or graduate mathematics courses; pedagogical content knowledge and curricular knowledge (also described by Shulman) both possibly taught in mathematics methods courses; and what is more elusive, knowledge that, while also mathematical, is not typically taught in undergraduate mathematics courses and is not be entirely pedagogical. Mathematical knowledge for teaching, keeping the emphasis on mathematics and acknowledging that teachers may know and use mathematics that is different from what is required for other professions” (Ferrini-Mundy, et. al., 2008)

Question 2 Teacher’s attitudes about being evaluated

Q3. Adapting Off-the-Shelf Instruments Start with this process for your own assessment needs Evaluate the fit of the instrument for your needs Important: Do not simply mix and match!