Robert J. Mislevy & Min Liu University of Maryland Geneva Haertel SRI International Robert J. Mislevy & Min Liu University of Maryland Geneva Haertel SRI International The Critical Role of Design Patterns in Large-Scale Assessment DR K-12 grant # , “Application of Evidence-Centered Design to State Large-Scale Science Assessment.” This material is based upon work supported by the National Science Foundation under Grant No. DRL Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. DR K-12 grant # , “Application of Evidence-Centered Design to State Large-Scale Science Assessment.” This material is based upon work supported by the National Science Foundation under Grant No. DRL Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Three design challenges  Hard-to-assess standards in the domain of interest  e.g., inquiry science skills  Efficient and valid design and development of complex tasks  e.g., scenarios, simulations  Accessibility of tasks for diverse learners  varying perceptual and expressive capabilities  valid use of assistive technology, modifications, alternative assessments

 Design Patterns in Architecture  Design Patterns in Software Engineering  Design Patterns in Literature Design Patterns

Motivation for Assessment Design Patterns  In-between structure, to connect...  Thinking about science learning & inquiry  Technical elements of measurement & delivery  Narrative, not technical, contents  Some Design Patterns from PADI  Model-Based Reasoning  Model Formation; Evaluation; Model Revision; Use  Observational & Experimental Investigations  Systems Thinking

Motivation for Assessment Design Patterns  They lay out a design space for developers  Choices, connections, examples  Things to be aware of (e.g., research on Universal Design for Learning)  Can improve both Efficiency + Validity  Attributes reflect assessment argument structure

Assessment Arguments  What complex of knowledge, skills, or other attributes should be assessed?  What behaviors or performances should reveal those constructs?  What tasks or situations should elicit those behaviors? Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), Student Model Evidence Model Task Model Mislevy, R.J., & Haertel, G. (2006). Implications for evidence-centered design for educational assessment. Educational Measurement: Issues and Practice, 25, 6-20.

The Structure of Assessment Design Patterns ATTRIBUTEDESCRIPTION RationaleHow/why this DP addresses evidence about focal KSAs. Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. Student Model Evidence Model Task Model

ATTRIBUTEDESCRIPTION RationaleHow/why this DP addresses evidence about focal KSAs. Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. The Rationale provides background into the nature of the Focal KSAs, and the kinds of things that people do in what kinds of situations that evidence it. E.g., overview, research links, examples. Student Model Evidence Model Task Model How Design Patterns Support Thinking about the Assessment Argument

Student Model Evidence Model Task Model ATTRIBUTEDESCRIPTION RationaleHow/why this DP addresses evidence about focal KSAs. Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. How Design Patterns Support Thinking about the Assessment Argument The design pattern is organized around Focal KSAs. They will be involved in the Student Model, although there may be other KSAs that are included in the target of inference (e.g., Model Revision— but what models, what context?). Associated with Characteristic Features of Tasks.

ATTRIBUTEDESCRIPTION RationaleHow/why this DP addresses evidence about focal KSAs. Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. How Design Patterns Support Thinking about the Assessment Argument Additional KSAs play multiple roles. You need to think about which ones you really DO want to include as targets of inference (validity) and which ones you really DON’T (invalidity). Student Model Evidence Model Task Model

ATTRIBUTEDESCRIPTION RationaleHow/why this DP addresses evidence about focal KSAs. Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. How Design Patterns Support Thinking about the Assessment Argument The Additional KSAs you DO want to include as targets of inference are part of the claim. E.g., knowing Mendel’s laws as well as being able to formulate a model in an investigation. Connected with Variable Features of Tasks. Student Model Evidence Model Task Model

ATTRIBUTEDESCRIPTION RationaleHow/why this DP addresses evidence about focal KSAs. Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. How Design Patterns Support Thinking about the Assessment Argument The Additional KSAs you DON’T want to include as targets of inference introduce construct-irrelevant reasons for poor performance. (Especially important for assessing special populations – UDL & accommodations.) Connected with Variable Features of Tasks & Work Products. Student Model Evidence Model Task Model

ATTRIBUTEDESCRIPTION RationaleHow/why this DP addresses evidence about focal KSAs. Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. How Design Patterns Support Thinking about the Assessment Argument The Characteristic Features of Tasks help you think about critical features of the tasks situation you need to get evidence about the Focal KSAs. Student Model Evidence Model Task Model

ATTRIBUTEDESCRIPTION RationaleHow/why this DP addresses evidence about focal KSAs. Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. How Design Patterns Support Thinking about the Assessment Argument Student Model Evidence Model Task Model Variable Features of Tasks also help you think about data concerning the situation – but now to influence difficulty … or to bring in or reduce demand for Additional KSAs to avoid alternative explanations.

ATTRIBUTEDESCRIPTION RationaleHow/why this DP addresses evidence about focal KSAs. Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. How Design Patterns Support Thinking about the Assessment Argument Some Variable Features of Tasks help you match features of tasks and background / knowledge / characteristics of students: Interests, familiarity, previous instruction. Student Model Evidence Model Task Model

ATTRIBUTEDESCRIPTION RationaleHow/why this DP addresses evidence about focal KSAs. Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. How Design Patterns Support Thinking about the Assessment Argument Potential Work Products help you think about what you want to capture from a performance – product, process, constructed model, written explanation, etc. Can also call attention to demand for Additional KSAs, & avoid alternative explanations for poor performance Student Model Evidence Model Task Model

ATTRIBUTEDESCRIPTION RationaleHow/why this DP addresses evidence about focal KSAs. Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. How Design Patterns Support Thinking about the Assessment Argument Potential Observations are possibilities for the qualities of Work Products – i.e., the data concerning the performance. Student Model Evidence Model Task Model

ATTRIBUTEDESCRIPTION RationaleHow/why this DP addresses evidence about focal KSAs. Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. How Design Patterns Support Thinking about the Assessment Argument And Potential Rubrics are algorithms/rubrics/rules for evaluating Work Products to interpret evidence from the student’s performance. Student Model Evidence Model Task Model

Current Catalog of Design Patterns  ECD/PADI related projects have produced over 100 Design Patterns  Domains include science inquiry, science content, mathematics, economics, model-based reasoning  Span grades  Organized around themes, models, and processes, not surface features or formats of tasks  Support the design of scenario-based, multiple choice, and performance tasks

Current Catalog of Design Patterns Subject Areas Education Standards Unifying Themes/ Inquiry Big Ideas within Disciplines Learning Progressions Language Proficiency Total Science Mathematics Economics Language Arts Management/ Business Second Language Acquisition Grand Total

For more information…  PADI: Principled Assessment Design for Inquiry   Links to NSF & IES follow-on projects  Lots of Tech Reports, interactive online examples  Bob Mislevy home page   Links to papers on ECD  Cisco applications

Now for the Good Stuff …  Examples of design patterns with content  Different projects  Different grain sizes  Different users  How they are being used to tackle pervasive challenges of large-scale assessment.  How they evolved to suit needs of users  Same essential structure, but  Representations, language, emphases, and affordances tuned to users and needs