Show Me an Evidential Approach to Assessment Design Michael Rosenfeld F. Jay Breyer David M. Williamson Barbara Showers

A Brief Introduction to Evidence Centered Design (ECD) Michael Rosenfeld, Ph.D. Rosenfeld & Associates

What is Evidence Centered Design? A formal framework for assessment design Emphasizes assessment as an evidential argument Encourages models and representations that strengthen validity arguments Focuses the test development process to better target decisions of interest

The Basic Models of ECD Proficiency Model – What you want to measure Evidence Model – How to recognize & interpret observable evidence of unobservable proficiencies Task Models – How to elicit valid and reliable evidence

ECD Validity Chain Evidence Proficiency Tasks Task Model Task Model Task Model Task Model

What is Gained from ECD? Better understanding of the domain, for new constructs Explicit chain of evidence from assessment goals, to models of evidence, to assessment tasks Strong construct validity argument Rapid and efficient item generation from models automatic item generation Opportunities for innovation, task types other than multiple choice Philosophical shift from content-authoring to evidential argument

Typical Test Development Content Specification Item Writing Test Blueprint Analysis Cut Scores The Job Knowledge & Tasks Major Dimensions Job Analysis Incumbent Survey Content Linking Statistical Modeling

Typical Test Development and Evidence Centered Design Content Specification Item Writing Test Blueprint Analysis Cut Scores Typical Test DevelopmentEvidence Centered Design Proficiency Model Task Model Evidence Model Tasks

Evidence Centered Design Proficiency Model Task Model Evidence Model Tasks Claims (Criterion) Knowledge, Strategies & Skills Reporting Variables Content Linking Evidential Argument Job Analysis Cognitive Task Analysis Statistical Modeling Task Design

Still to Come Example of the Proficiency Modeling process (Jay) Illustration of Evidence Modeling and Task Modeling (David) Commentary from an applied perspective (Barbara)

Proficiency Models in the ECD Framework F. Jay Breyer, Ph.D.

Proficiency Model Combination of Formalization of assessment goals, population and purpose Formal claims to be made on the basis of the assessment He/she is competent to operate an automobile Proficiencies of interest Knowledge of rules, procedures, and laws Ability to implement driving regulations in practice Skill in automobile operation

What is a Proficiency? Proficiencies are knowledge, skills and abilities that provide the basis for making claims about people

Building a Proficiency Model: Assessment Goals Assessment Goals Protect the public health, safety & welfare Examinee population All eligible (age, prior training requirements, etc.) citizens self-selecting into the program Test users/stakeholders State regulatory bodies State resident drivers Other citizens exposed to driving Result Usage/Decisions State issuing the license (to issue or deny) Means of official identification

Building a Proficiency Model: Assessment Claims Formal Claims are developed for assessment goals Claims can be course-grained and general Can safely operate a motor vehicle These general claims can be be supported by a variety of more specific sub-claims in a hierarchy Operates a vehicle in conformance with posted signs Stops at a posted stop sign Obeys posted speed limits The final hierarchy of claims drives the evidential argument and the score reporting needs of the test

Building a Proficiency Model: Proficiencies Proficiency variables are the basis of making claims Proficiency variables, like claims, may be organized hierarchically Values or levels of proficiency variables may be selected to support claim distinctions Informed by investigations of the domain and learning within the domain job analyses, cognitive task analyses, learning models, etc.

Example of a Hypothetical Proficiency Model for Driving

Still to Come Illustration of Evidence Modeling and Task Modeling (David) Commentary from an applied perspective (Barbara)

Providing the Evidence in Evidence Centered Design David M. Williamson, Ph.D.

Building Evidence Models Objective: valid inference Understanding what constitutes evidence and why Situations that elicit evidential behaviors Task design Statistical representation of evidential value of observations Integration of scoring with design ECD models – conceptual scoring

Establishing Evidence in a “Perfect World” Targets of Inference Links behaviors and inference Behavior Elicitation Driving Ability Knowledge of RotR Psychomotor Evidence Knowledge declarations Performance Task Requirements Situation Outcomes

Establishing Evidence: Driver’s Exam Knowledge of RotR Weather & Light Traffic Flow Intersections Speed Psychomotor Intersections Speed Turning Parking Driving Ability Knowledge of RotR Psychomotor Intersections Recognizing signs Specifying intersection traffic behaviors Following proper procedure

Building Task Models: Why? Elicits behaviors by predictably manipulating Focus Difficulty Context Targets what we want to measure Provides a framework for generating multiple tasks targeting same proficiencies

Building Task Models: What? To target elicitation of particular evidence, the task model specifies: Tools provided Directions and stimulus materials Indication of data collected and retained for scoring Relevance of specified elements to particular evidence, and particular proficiencies

Evidence as Inference about Proficiency Stage 1: Evidence Identification Identify and summarize response elements Stage 2: Evidence Accumulation Using these elements to estimate ability

Beneficial Outcomes of ECD Development direction w evidential focus Integration of expertise/activities (TD, stat, etc.) in the design process Chain of reasoning Integrated design – implications of design changes Validity argument Item modeling AIG Defensibility Directed item writing – future forms Somewhat more expensive (Initially)

Still to Come Commentary from an applied perspective (Barbara)

Discussant Barbara Showers, Ph.D. Director, Office of Education and Examinations Wisconsin Department of Regulation and Licensing