Robert J. Mislevy University of Maryland Geneva Haertel & Britte Haugan Cheng SRI International Robert J. Mislevy University of Maryland Geneva Haertel & Britte Haugan Cheng SRI International Design Patterns: Supporting Task Design by Scaffolding the Assessment Argument DR K-12 grant # , “Application of Evidence-Centered Design to State Large-Scale Science Assessment.” NSF Discovery Research K-12 PI meeting, November 10, Washington D.C. This material is based upon work supported by the National Science Foundation under Grant No. DRL Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. DR K-12 grant # , “Application of Evidence-Centered Design to State Large-Scale Science Assessment.” NSF Discovery Research K-12 PI meeting, November 10, Washington D.C. This material is based upon work supported by the National Science Foundation under Grant No. DRL Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Overview  Design patterns  Background  Evidence-Centered Design  Main idea  Layers  Assessment Arguments  Attributes of Design Patterns  How they inform task design

Design Patterns  Design Patterns in Architecture  Design Patterns in Software Engineering  Polti’s Thirty-Six Dramatic Situations

Messick’s Guiding Questions  What complex of knowledge, skills, or other attributes should be assessed?  What behaviors or performances should reveal those constructs?  What tasks or situations should elicit those behaviors? Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2),

Evidence-Centered Assessment Design  Organizing formally around Messick quote  Principled framework for designing, producing, and delivering assessments  Conceptual model, object model, design tools  Connections among design, inference, and processes to create and deliver assessments.  Particularly useful for new / complex assessments.  Useful to think in terms of layers

 From Mislevy & Riconscente, in press Assessment Delivery Students interact with tasks, performances evaluated, feedback created. Four- process delivery architecture. Assessment Implementation Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? How do we represent key aspects of the domain in terms of assessment argument. Conceptualization. Design structures: Student, evidence, and task models. Generativity. Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability. Layers in the assessment enterprise

 From Mislevy & Riconscente, in press Assessment Delivery Students interact with tasks, performances evaluated, feedback created. Four- process delivery architecture. Assessment Implementation Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? How do we represent key aspects of the domain in terms of assessment argument. Conceptualization. Design structures: Student, evidence, and task models. Generativity. Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.

 From Mislevy & Riconscente, in press Assessment Delivery Students interact with tasks, performances evaluated, feedback created. Four- process delivery architecture. Assessment Implementation Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? How do we represent key aspects of the domain in terms of assessment argument. Conceptualization. Design structures: Student, evidence, and task models. Generativity. Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability. Assessment argument structures Design Patterns

 From Mislevy & Riconscente, in press Assessment Delivery Students interact with tasks, performances evaluated, feedback created. Four- process delivery architecture. Assessment Implementation Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? How do we represent key aspects of the domain in terms of assessment argument. Conceptualization. Design structures: Student, evidence, and task models. Generativity. Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability. Psychometric models Automated scoring Task templates Object models Simulation environments

 From Mislevy & Riconscente, in press Assessment Delivery Students interact with tasks, performances evaluated, feedback created. Four- process delivery architecture. Assessment Implementation Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? How do we represent key aspects of the domain in terms of assessment argument. Conceptualization. Design structures: Student, evidence, and task models. Generativity. Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability. Authoring interfaces Simulation environments Re-usable platforms & elements

 From Mislevy & Riconscente, in press Assessment Delivery Students interact with tasks, performances evaluated, feedback created. Four- process delivery architecture. Assessment Implementation Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? How do we represent key aspects of the domain in terms of assessment argument. Conceptualization. Design structures: Student, evidence, and task models. Generativity. Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability. Interoperable elements IMS/QTI, SCORM Feedback / instruction / reporting

Toulmin’s Argument Structure Claim Backing unless since Warrant Alternative explanation so Data

Assessment Argument Structure Claim about student Warrant for assessment argument since Alternative explanations unless Data concerning performance so

Assessment Argument Structure Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning performance Data concerning situation

Assessment Argument Structure Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning situation Data concerning performance Student acting in assessment situation Warrant for scoring since Warrant for task design

Assessment Argument Structure Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning situation Data concerning performance Other information concerning student vis a vis assessment situation Student acting in assessment situation since Warrant for scoring since Warrant for task design e.g., near or far transfer, familiarity with tools, assessment format, representational forms, evaluation standards, task content & context. Not in measurement models, but crucial to inference.

PADI Design Patterns  Structured around assessment arguments  Substance based on recurring principles, ways of thinking, inquiry, etc.  E.g., NSES on inquiry, unifying themes  Science ed. & cog psych research

Some PADI Design Patterns  Model-Based Reasoning  Model Formation; Evaluation; Revision; Use  Model-Based Inquiry  Design under Constraints  Generate Scientific Explanations  Troubleshooting (with Cisco)  Assessing Epistemic Frames (in progress; with David Williamson Shaffer)

ATTRIBUTEDESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. RationaleHow/why this DP addresses evidence about focal KSAs Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. The Structure of Assessment Design Patterns

ATTRIBUTEDESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. RationaleHow/why this DP addresses evidence about focal KSAs Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning situation Data concerning performance Other information concerning student vis a vis assessment situation Student acting in assessment situation since Warrant for scoring since Warrant for task design How Design Patterns Support Thinking about the Assessment Argument

ATTRIBUTEDESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. RationaleHow/why this DP addresses evidence about focal KSAs Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning situation Data concerning performance Other information concerning student vis a vis assessment situation Student acting in assessment situation since Warrant for scoring since Warrant for task design How Design Patterns Support Thinking about the Assessment Argument The design pattern is organized around Focal KSAs. They will be involved in the Claim, although there may be other KSAs that are included in the target of inference (e.g., Model Formation—but what models, what context?). Associated with Characteristic Features of Tasks.

ATTRIBUTEDESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. RationaleHow/why this DP addresses evidence about focal KSAs Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning situation Data concerning performance Other information concerning student vis a vis assessment situation Student acting in assessment situation since Warrant for scoring since Warrant for task design How Design Patterns Support Thinking about the Assessment Argument The Rationale provides background into the nature of the Focal KSAs, and the kinds of things that people do in what kinds of situations that evidence it. It contributes to the Warrant in the assessment argument.

ATTRIBUTEDESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. RationaleHow/why this DP addresses evidence about focal KSAs Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning situation Data concerning performance Other information concerning student vis a vis assessment situation Student acting in assessment situation since Warrant for scoring since Warrant for task design How Design Patterns Support Thinking about the Assessment Argument Additional KSAs play multiple roles. You need to think about which ones you really DO want to include as targets of inference (validity) and which ones you really DON’T (invalidity).

ATTRIBUTEDESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. RationaleHow/why this DP addresses evidence about focal KSAs Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning situation Data concerning performance Other information concerning student vis a vis assessment situation Student acting in assessment situation since Warrant for scoring since Warrant for task design How Design Patterns Support Thinking about the Assessment Argument The Additional KSAs you DO want to include as targets of inference are part of the claim. E.g., knowing Mendel’s laws as well as being able to formulate a model in an investigation. Connected with Variable Features of Tasks.

ATTRIBUTEDESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. RationaleHow/why this DP addresses evidence about focal KSAs Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning situation Data concerning performance Other information concerning student vis a vis assessment situation Student acting in assessment situation since Warrant for scoring since Warrant for task design How Design Patterns Support Thinking about the Assessment Argument The Additional KSAs you DON’T want to include as targets of inference introduce alternative explanations for poor performance. (Especially important for assessing special populations – UDL & acommodations.) Connected with Variable Features of Tasks & Work Products.

ATTRIBUTEDESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. RationaleHow/why this DP addresses evidence about focal KSAs Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning situation Data concerning performance Other information concerning student vis a vis assessment situation Student acting in assessment situation since Warrant for scoring since Warrant for task design How Design Patterns Support Thinking about the Assessment Argument The Characteristic Features of Tasks help you think about critical data concerning the situation –what you need to get evidence about the Focal KSAs.

ATTRIBUTEDESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. RationaleHow/why this DP addresses evidence about focal KSAs Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning situation Data concerning performance Other information concerning student vis a vis assessment situation Student acting in assessment situation since Warrant for scoring since Warrant for task design How Design Patterns Support Thinking about the Assessment Argument Variable Features of Tasks also help you think about data concerning the situation – but now to influence difficulty … or to bring in or reduce demand for Additional KSAs to avoid alternative explanations.

ATTRIBUTEDESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. RationaleHow/why this DP addresses evidence about focal KSAs Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning situation Data concerning performance Other information concerning student vis a vis assessment situation Student acting in assessment situation since Warrant for scoring since Warrant for task design How Design Patterns Support Thinking about the Assessment Argument Some Variable Features of Tasks help you match features of tasks and background / knowledge / characteristics of students: Interests, familiarity, previous instruction.

ATTRIBUTEDESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. RationaleHow/why this DP addresses evidence about focal KSAs Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning situation Data concerning performance Other information concerning student vis a vis assessment situation Student acting in assessment situation since Warrant for scoring since Warrant for task design How Design Patterns Support Thinking about the Assessment Argument Potential Work Products help you think about what you want to capture from a performance – product, process, constructed model, written explanation, etc. Can also call attention to demand for Additional KSAs, & avoid alternative explanations (e.g., Stella)

ATTRIBUTEDESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. RationaleHow/why this DP addresses evidence about focal KSAs Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning situation Data concerning performance Other information concerning student vis a vis assessment situation Student acting in assessment situation since Warrant for scoring since Warrant for task design How Design Patterns Support Thinking about the Assessment Argument Potential Observations are possibilities for the qualities of Work Products – i.e., the data concerning the performance.

ATTRIBUTEDESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. RationaleHow/why this DP addresses evidence about focal KSAs Additional KSAs Other knowledge/skills/abilities that may be required by tasks. Characteristic features of tasks Aspects of assessment situations that are needed to evoke evidence about the focal KSAs. Variable features of tasks Aspects of assessment situations that can be varied to shift difficulty or focus. Potential work products What students actually say, do, or make, to produce evidence. Potential observations Aspects of work products we might identify and evaluate, as evidence about students’ KSAs. Potential rubrics Ways of evaluating work products to produce values of observations. Claim about student Warrant for assessment argument Alternative explanations since so unless Data concerning situation Data concerning performance Other information concerning student vis a vis assessment situation Student acting in assessment situation since Warrant for scoring since Warrant for task design How Design Patterns Support Thinking about the Assessment Argument And Potential Rubrics are algorithms/rubrics/rules for evaluating Work Products to get the data concerning the performance.

For more information…  PADI: Principled Assessment Design for Inquiry   NSF project, collaboration with SRI et al.  Links to follow-on projects  Bob Mislevy home page   Links to papers on ECD  Cisco applications

Now for the Good Stuff …  Examples of design patterns with content  Different projects  Different grain sizes  Different users  How they evolved to suit needs of users  Same essential structure  Representations, language, emphases, and affordances tuned to users and needs  How they are being used

Use of Design Patterns in STEM Research and Development Projects Britte Haugan Cheng and Geneva Haertel DRK-12 PI Meeting, November 2009

Current Catalog of Design Patterns  ECD/PADI related projects have produced over 100 Design Patterns  Domains include: science inquiry, science content, mathematics, economics, model-based reasoning  Design Patterns span grades  Organized around themes, models, and processes, not surface features or formats of tasks  Support the design of scenario-based, multiple choice, and performance tasks  The following examples show how projects have used and customized Design Patterns in ways that suit their needs and users

Example 1 DRK-12 Project An Application of ECD to a State, Large-scale Science Assessment  Challenge in Minnesota Comprehensive Assessment of science:  How to design scenario-based tasks, technology-enhanced interactions, grounded in standards both EFFICIENTLY and VALIDLY.  Design Patterns support storyboard writing and task authoring  Designers are committee of MN teachers, supported by Pearson  Project focuses on a small number of Design Patterns for “hard-to- assess” science content/inquiry  Based on Minnesota state science standards and benchmarks and the NSES inquiry standards  Design Patterns are Web-based and interactive

Design Pattern Observational Investigation  Relates science content/processes to components of assessment argument  Higher-level, cross-cutting themes, ways of thinking, ways of using science, rather than many finer-grained standards  Related to relevant standards and benchmarks  Interactive Features:  Examples and details  Activate pedagogical content knowledge  Presents exemplar assessment tasks  Provides selected knowledge representations  Links among associated assessment argument components

Design Pattern Observational Investigation

Design Pattern Observational Investigation (cont.)

Interactive Feature: Details

Interactive Feature: Linking assessment argument components

Design Pattern Highlights Observational Investigation  Relates science content/processes to components of assessment argument  Higher-level, cross-cutting themes, ways of thinking, ways of using science, rather than many fine-grained standards  Interactive Features:  Examples and details  Activates pedagogical content knowledge  Presents exemplar assessment tasks  Provides selected knowledge representations  Relates relevant standards and benchmarks  Links among associated assessment argument components

Design Pattern Reasoning about Complex Systems  Relates science content/processes to components of assessment argument  Across scientific domains and standards  Convergence among the design of instruction, assessment and technology  Interactive Features:  Explicit support for designing tasks around multi-year learning progression

Design Pattern Reasoning about Complex Systems

Interactive Feature: Details

Interactive Feature: Linking assessment argument components

Design Pattern Highlights Reasoning about Complex Systems  Relates science content/processes to components of assessment argument  Across scientific domains and standards  Convergence among the design of instruction, assessment and technology  Interactive Feature:  Explicit support for designing tasks around multi-year learning progression

Example 2 Principled Assessment Designs in Inquiry Model-Based Reasoning Suite  Relates science content/processes to components of assessment argument  A suite of seven related Design Patterns support curriculum-based assessment design  Theoretically and empirically motivated by Stewart and Hafner (1994), Research on Problem-Solving: Genetics. In D. L. Gable (Ed.), Handbook of research on science teaching and learning. New York: MacMillan Publishing.  Aspects of model-based reasoning including model formation, model use, model revision, and coordination among aspects of model-based reasoning  Multivariate student model: scientific reasoning and science content  Interactive Feature:  Support the design of both:  Independent tasks associated with an aspect of model-based reasoning  Steps in a larger investigation comprised of several aspects including model conceptualization, model use and model evaluation

Design Pattern Model Formation

Design Pattern Model Formation (cont.)

Interactive Feature: Links among Design Patterns

Design Pattern Highlights Model-based Reasoning Suite  Relates science content/processes to components of assessment argument  Facilitate the integration of model-based reasoning skills into any science content area  Serve as basis of a learning progression  Interactive Features:  Support the design of both independent tasks associated with an aspect of model-based reasoning and steps in a larger investigation that is comprised of several aspects including conceptualization of a model to its use and evaluation  Explicit supports (links among Design Patterns) for designing both investigations and focused tasks

Example 3 Principled Science Assessment Designs for Students with Disabilities Designing and Conducting Scientific Investigations Using Appropriate Methodology  Relates science content/processes to components of assessment argument  Guide refinement of science assessment tasks across multiple states by identifying and reducing sources of construct-irrelevant variance  Integrate six categories of Universal Design for Learning (UDL) into the assessment design process:  Perceptual, linguistic, cognitive, motoric, executive, affective  Interactive Feature:  Highlight relationships among Additional KSAs, Variable Task Features and Potential Work Products to reduce construct-irrelevant variance in a systematic manner

Design Pattern Designing and Conducting a Scientific Investigation Using Appropriate Methodology

Design Pattern Designing and Conducting a Scientific Investigation Using Appropriate Methodology (cont.)

Interactive Feature: Linking Additional KSAs and Potential Work Products

Design Pattern Highlights Designing and Conducting a Scientific Investigation Using Appropriate Methodology  Relates science content/processes to components of assessment argument  Integrate UDL in assessment design process rather than applying accommodations to an existing task  Supports the selection of task features that reduce construct-irrelevant variance and enhance the performance of all test takers  Particular attention to knowledge representation and executive processing demands  Further customization of Design Patterns to develop assessment tasks for students with particular disabilities  Interactive Feature:  Relate the perceptual and expressive capabilities required to complete an assessment task to that task’s features (Additional KSAs, Variable Task Features and Potential Work Products)

Example 4 Alternate Assessments in Mathematics Describe, extend, and make generalizations about geometric and numeric patterns  Relates math content/processes to components of assessment argument  Standards-based Design Patterns co-designed across three states to guide the development of statewide assessment tasks for students with significant cognitive disabilities  Integration of six UDL categories into the design process  Interactive Feature:  For logistical reasons, Word document used to create Design Patterns  Attributes visualized in accordance with the assessment argument resulting in increased efficiency and improved quality of argument  New arrangement now under development for use in online system

Design Pattern Describe, extend, and make generalizations about geometric and numeric patterns

Design Pattern Describe, extend, and make generalizations about geometric and numeric patterns (cont.)

Interactive Feature: Horizontal View Aligning Focal KSAs, Potential Observations and Potential Work Products

Interactive Feature: Horizontal View Aligning Additional KSAs and Variable Task Features

Design Pattern Highlights Describe, extend, and make generalizations about geometric and numeric patterns  Relates math content/processes to components of assessment argument  Deconstruction of NCTM expectations to identify KSAs that are less difficult or tasks that assess related cognitive background knowledge  Supports the principled alignment of task difficulty and scope with challenges to accessibility  Interactive Feature:  Use of multiple views of the Design Pattern to support understanding of the relationship of components of the assessment argument  Increased efficiency of design and validity of assessment argument

Summary  Design Patterns are organized around assessments and key ideas in science and math, as opposed to surface features of assessment tasks.  Support designing tasks that move in ways NSES and NCTM advocate in ways that build on research and experience  Design Patterns support task design for different purposes and different formats (e.g., learning, summative, classroom, large-scale, hands-on, P&P, simulations).  Especially important for newer forms of assessment  Technology-based, scenario based tasks in Minnesota  Scenario-based learning & assessment (Foothill-DeAnza project)  Simulation-based tasks (network troubleshooting, with Cisco)  Games-based assessment (just starting, with MacArthur project)

Summary  Design Patterns are eclectic—they are not associated with any particular underlying theory of learning or cognition; all psychological perspectives can be represented  Document design decisions  Represent hierarchical relationships among Focal KSAs, sequential steps required for the completion of complex tasks, or superordinate, subordinate, and coordinate relations among concepts  Re-usable; a family of assessment tasks can be produced from a single Design Pattern  Enhance the integration of UDL with the evidence-centered design process  Technology makes evident the relationships among Design Pattern attributes and their role in the assessment argument