Download presentation
Presentation is loading. Please wait.
Published byDorcas Wright Modified over 8 years ago
1
On Layers and Objects in Assessment Design Robert Mislevy, University of Maryland Michelle Riconscente, University of Maryland Robert Mislevy, University of Maryland Michelle Riconscente, University of Maryland
2
2 Evidence-Centered Design Conceptual design framework Broad applicability Based on Evidentiary reasoning principles Demands of assessment production & delivery Assures Validity of assessment Coordination among collaborators (Mislevy, Almond, & Lukas, 2003)
3
3 Defining Assessment Reasoning particular things Reasoning from the particular things students do, say, or make… to inferences to inferences about their knowledge, skills & abilities.
4
4 Some Fundamentals Evidentiary Reasoning Assessment as Argument
5
5 Evidentiary Reasoning C R B unless since on account of W A so supports Toulmin (1958) D
6
6 Example I C R B unless on account of W A so supports D since Toulmin (1958)
7
7 Example II John caused the car accident and Marg was seriously injured. John’s BAC was.13. Marg was unconscious and bleeding. John’s speech was slurred. Marg had the right of way. W1: Since an intoxicated driver is generally presumed to be at fault in an accident…. W2: Since the impact of a 2000lb. auto moving at 50mph on a human will generally cause serious injury… B1: State Code: drunk driving at.10 BAC and common law doctrine of negligence per se. B2: Emergency medical records at Wishard Hospital indicate this type of collision will result in serious condition. Marg was also intoxicated… Toulmin (1958) C R B unless on account of W A so supports D since
8
8 Assessment as Argument C D since W Inferences Observations Situations Inferences Observations Observations needed to ground them Situations Situations that will evoke them reasoning Chain of reasoning connecting them since
9
9 Some Fundamentals Evidentiary Reasoning Assessment as Argument Knowledge Representations (KRs) Common Language
10
10 Knowledge Representations Directions from Campus to the White House http://www.mapquest.com/directions/main.adp?go=1&do=nw&un=m& 2tabval=address&cl=EN&ct=NA&1tabval=address&1y=US&1a=&1c=& 1s=&1z=&1ah=NXXzjLrv%252bgGeW7JeQ4fDrUNya%252b4DtKsjxh eMTbVThQJ3lE9WrNx7DAJWKfgZBsxDNU8KMBJ3MT8wsLFk85Pi6j LRqQHxBFYMlklq5k82hIODOViBiLYDxDz3sk%252bXxGlUcWB2Xe c5x21sAggO4xXFyg%253d%253d&2y=US&2a=1600+Pennsylvania+Av enue&2c=Washington&2s=DC&2z=&2ah=&idx=0&id=417d7541- 00047-01632-400c2551&aid=417d7541-00048-01632-400c2551 OR ?
11
11 Some Fundamentals Evidentiary Reasoning Assessment as Argument Knowledge Representations (KRs) Common Language Layers Leveraging Varied Expertise Common Structures
12
12 A Layered Approach Leveraging Varied Expertise e.g., Housing Developments Common Structures e.g., Architecture, Software Design Iterate through Assessment Argument narrative technical specifications InferencesObservationsSituations
13
13 Assessment Delivery ECD Layers Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What tasks are central to proficiency in this domain? What KRs are central to this domain? What is important about this domain? What tasks are central to proficiency in this domain? What KRs are central to this domain? Represent key aspects of the domain in terms of assessment argument. What tools and materials do we need to implement this kind of assessment? How do we move to implementation with actual examinees?
14
14 Valued work Task features Representational forms Performance outcomes Valued knowledge Knowledge structure and relationships Knowledge-task relationships Domain Analysis
15
15 Assessment Delivery ECD Layers Conceptual Assessment Framework Domain Modeling Domain Analysis
16
16 Domain Modeling Express the content of the domain analysis in terms of the assessment argument. Takes narrative form PADI KR: Design Patterns InferencesObservationsSituations Domain Modeling
17
17 Design Patterns Identify in narrative form: Knowledge, skills, and abilities (KSAs) Observations to support inference Features of task situations that elicit target KSAs Related content or inquiry standards Do not provide a concrete design or implementation of an assessment task
18
18
19
19 Assessment Delivery ECD Layers Conceptual Assessment Framework Domain Modeling Domain Analysis
20
20 Conceptual Assessment Framework Move from narrative to more technical specifications. Express assessment argument in terms closer to implementation ECD Models Conceptual Assessment Framework
21
21 STUDENT MODEL EVIDENCE MODEL TASK MODEL InferencesObservationsSituations From Argument to Models (Mislevy, Almond, & Lukas, 2003)
22
22 EVIDENCE MODEL How do we measure it? Quality Relevance STUDENT MODEL TASK MODEL From Argument to Models (Mislevy, Almond, & Lukas, 2003) What are we measuring? Where do we measure it?
23
23 The Passenger There are perfectly satisfactory answers to all your questions, but I don’t think you understand how little you would learn from them… Your questions are much more revealing about yourself than my answers would be about me. (Peploe, Wollen, & Antonioni, 1975)
24
24 HIDDEN SLIDE – READ JOKE While crossing the US-Mexican border on his bicycle, the man was stopped by a guard who pointed to two sacks the man had on his shoulders. "What's in the bags?", asked the guard. "Sand," said the cyclist. "Get them off - we'll take a look," said the guard. The Cyclist did as he was told, emptied the bags, and proving they contained nothing but sand, reloaded the bags, put them on his shoulders and continued across the border. Two weeks later, the same thing happened. Again the guard demanded to see the two bags, which again contained nothing but sand. This went on every week for six months, until one day the cyclist with the sand bags failed to appear. A few days later, the guard happened to meet the cyclist downtown. "Say friend, you sure had us crazy", said the guard. "We knew you were smuggling something across the border. I won't say a word - but what is it you were smuggling?" "Bicycles!"
25
25 Move from narrative to more technical specifications. Express assessment argument in terms closer to implementation ECD Models pre-blueprints Create pre-blueprints from which we will eventually generate blueprints and operational assessments. PADI KR: Task Templates Conceptual Assessment Framework
26
26 Task Templates Support the specification of technical details Link CAF components Serve as pre-blueprints: abstractions of multiple assessment tasks Become task specifications when all template components are specified
27
27
28
28
29
29 Summing Up ECD: assessment design as development of an assessment argument Coordinate work across expertise Use of layers Knowledge Representations Illustrated with PADI work In addition: Explication of reasoning behind assessment design decisions Identification of reusable elements and pieces of infrastructure
30
30 An ECD “Flight Check” Conceptual design framework Broad applicability Based on Evidentiary reasoning principles Demands of assessment production & delivery Assures Validity of assessment Coordination among collaborators (Mislevy, Almond, & Lukas, 2003)
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.