University of Maryland Slide 1 July 6, 2005 Presented at Invited Symposium K3, “Assessment Engineering: An Emerging Discipline” at the annual meeting of the National Council on Measurement in Education, Chicago, IL, April 10-12, 2007 PADI is supported by the Interagency Educational Research Initiative (IERI) under grants REC (PADI Planning Grant)and REC (PADI Implementation Grant).Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. Implications of Evidence-Centered Design for Educational Testing: Lessons from the PADI Project Robert J. Mislevy University of Maryland Geneva D. Haertel SRI International April 12, 2007

University of Maryland Slide 2 July 6, 2005 Outline l Remarks on assessment engineering l Evidence-centered assessment design  Assessment arguments  Layers l Implications  Explicitness, representations, reusability, generativity l PADI l Too many notes?

University of Maryland Slide 3 July 6, 2005 Assessment Engineering l Is assessment design art or science? l Assessment engineering submits there are recurring structures and relationships we can exploit – in terms of language, data structures, representational forms, and processes – to enhance efficiency and validity in assessment/task design.

University of Maryland Slide 4 July 6, 2005 Assessment Engineering l Idea not new or exclusive; some examples:  Guttman, Suppes, Hively et al., Obsbourne, Bormuth ~ 60s.  Roid & Haladyna (1982): Technology for Test-Item Writing  Embretson (1985): Test design: Developments in psychology and psychometrics.  Embretson, Wilson, Baker, Luecht, Gorin, Tatsuoka, Bejar, Shavelson, Irvine & Kyllonen… l Evidence-centered design (ECD)  Mislevy, Steinberg, & Almond (2003)  PADI project

University of Maryland Slide 5 July 6, 2005 Assessment Arguments What complex of knowledge, skills, or other attributes should be assessed? What behaviors or performances should reveal those constructs? What tasks or situations should elicit those behaviors? (Messick, 1994)

From Mislevy & Riconscente, in press Assessment Delivery How do students and tasks actually interact? How do we report examinee performance? How do students and tasks actually interact? How do we report examinee performance? Assessment Implementation Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? How do we represent key aspects of the domain in terms of assessment argument. Design structures: Student, evidence, and task models How do we choose and present tasks, and gather and analyze responses? Layers in the assessment enterprise

From Mislevy & Riconscente, in press Assessment Delivery How do students and tasks actually interact? How do we report examinee performance? How do students and tasks actually interact? How do we report examinee performance? Assessment Implementation Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? How do we represent key aspects of the domain in terms of assessment argument. Design structures: Student, evidence, and task models How do we choose and present tasks, and gather and analyze responses? Cog Psych, expertise studies, domain research Cog Psych, expertise studies, domain research

From Mislevy & Riconscente, in press Assessment Delivery How do students and tasks actually interact? How do we report examinee performance? How do students and tasks actually interact? How do we report examinee performance? Assessment Implementation Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? How do we represent key aspects of the domain in terms of assessment argument. Design structures: Student, evidence, and task models How do we choose and present tasks, and gather and analyze responses? Specific implementations. Surface elements. Specific implementations. Surface elements.

From Mislevy & Riconscente, in press Assessment Delivery How do students and tasks actually interact? How do we report examinee performance? How do students and tasks actually interact? How do we report examinee performance? Assessment Implementation Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? How do we represent key aspects of the domain in terms of assessment argument. Design structures: Student, evidence, and task models How do we choose and present tasks, and gather and analyze responses? Assessment Argument

From Mislevy & Riconscente, in press Assessment Delivery How do students and tasks actually interact? How do we report examinee performance? How do students and tasks actually interact? How do we report examinee performance? Assessment Implementation Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? How do we represent key aspects of the domain in terms of assessment argument. Design structures: Student, evidence, and task models How do we choose and present tasks, and gather and analyze responses? Explicit connection to domain & purpose Generative structures for recurring kinds of proficiencies (e.g., inquiry cycles, troubleshooting) across projects PADI Design patterns

From Mislevy & Riconscente, in press Assessment Delivery How do students and tasks actually interact? How do we report examinee performance? How do students and tasks actually interact? How do we report examinee performance? Assessment Implementation Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? How do we represent key aspects of the domain in terms of assessment argument. Design structures: Student, evidence, and task models How do we choose and present tasks, and gather and analyze responses? Generative Design Schemas

From Mislevy & Riconscente, in press Assessment Delivery How do students and tasks actually interact? How do we report examinee performance? How do students and tasks actually interact? How do we report examinee performance? Assessment Implementation Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? What is important about this domain? What work and situations are central in this domain? What KRs are central to this domain? How do we represent key aspects of the domain in terms of assessment argument. Design structures: Student, evidence, and task models How do we choose and present tasks, and gather and analyze responses? Explicit connection of argument to machinery Generative structures for recurring task situations (e.g., item shells) Re-usable / interoperable data structures PADI templates

University of Maryland Slide 13 July 6, 2005 Supported by NSF, via IERI program Focus on scaling up ECD ideas specifically focusing on science inquiry tasks Partners: »SRI International (Geneva, co-PI) »University of Maryland (Mislevy, co-PI) »University of California at Berkeley (BEAR – Wilson – & FOSS –Long ) »University of Michigan (BioKIDS – Songer) PADI: Principled Assessment Design for Inquiry

University of Maryland Slide 14 July 6, 2005 PADI Conceptual frameworks & representations »Domain Modeling: Design patterns for inquiry »CAF: Task-design (templates) Object model for task design »Data structure Software tools to aid the design process »Design system »Wizards

University of Maryland Slide 15 July 6, 2005 PADI Libraries of exemplars … »Design patterns for assessing inquiry »Task templates and pointers to tasks “Scoring engine” (MRCMLM) Worked-through applications »FOSS, BioKids; GLOBE, FCI, Mystery Powders, Mystery Boxes Evaluation studies – FOSS & BioKids

University of Maryland Slide 16 July 6, 2005 PADI For more information: padi.sri.com/ Includes lots of tech reports.

University of Maryland Slide 17 July 6, 2005 Too many notes? As Emperor Joseph II said to Mozart. Power in general/abstract representations, »Reveals fundamental similaries under assessments that look different on the surface. »Right perspective for design of systems, but not necessarily for work within systems. Some people need to work from first principles, but… Scaling up for wide range of users requires exemplars, support tools, tuned interfaces.