Show Me an Evidential Approach to Assessment Design Michael Rosenfeld F. Jay Breyer David M. Williamson Barbara Showers.

Slides:



Advertisements
Similar presentations
An Introduction to Test Construction
Advertisements

Performance Assessment
What Behaviors Indicate a Student is Meeting Course Goals and Objectives? “Indicators”
General Information --- What is the purpose of the test? For what population is the designed? Is this population relevant to the people who will take your.
Managing Human Resources, 12e, by Bohlander/Snell/Sherman © 2001 South-Western/Thomson Learning 5-1 Managing Human Resources Managing Human Resources Bohlander.
How well did the assessment task do what we wanted it to do? Janina Drazek Manager — Assessment & Comparability, QCAR Queensland Studies Authority.
Developing Classroom Assessments for the NGSS What evidence of student thinking is needed to determine if a student has met a PE (performance expectation)?
Robert J. Mislevy & Min Liu University of Maryland Geneva Haertel SRI International Robert J. Mislevy & Min Liu University of Maryland Geneva Haertel SRI.
Chapter 4 Validity.
VALIDITY.
SRI Technology Evaluation WorkshopSlide 1RJM 2/23/00 Leverage Points for Improving Educational Assessment Robert J. Mislevy, Linda S. Steinberg, and Russell.
Job Analysis and Rewards
Planning Value of Planning What to consider when planning a lesson Learning Performance Structure of a Lesson Plan.
Meaningful Learning in an Information Age
FERA 2001 Slide 1 November 6, 2001 Making Sense of Data from Complex Assessments Robert J. Mislevy University of Maryland Linda S. Steinberg & Russell.
Chapter 9 Flashcards. measurement method that uses uniform procedures to collect, score, interpret, and report numerical results; usually has norms and.
Classroom Assessment A Practical Guide for Educators by Craig A
Science and Engineering Practices
Conducting a Job Analysis to Establish the Examination Content Domain Patricia M. Muenzen Associate Director of Research Programs Professional Examination.
Adolescent Sexual Health Work Group (ASHWG)
Ch 6 Validity of Instrument
Evidence-Based Practice Current knowledge and practice must be based on evidence of efficacy rather than intuition, tradition, or past practice. The importance.
PDHPE K-6 Using the syllabus for consistency of assessment © 2006 Curriculum K-12 Directorate, NSW Department of Education and Training.
ECD in the Scenario-Based GED ® Science Test Kevin McCarthy Dennis Fulkerson Science Content Specialists CCSSO NCSA June 29, 2012 Minneapolis This material.
ASSESSMENT IN EDUCATION ASSESSMENT IN EDUCATION. Copyright Keith Morrison, 2004 PERFORMANCE ASSESSMENT... Concerns direct reality rather than disconnected.
Foundations of Recruitment and Selection I: Reliability and Validity
ASSESSMENT OF STUDENT LEARNING Manal bait Gharim.
SLB /04/07 Thinking and Communicating “The Spiritual Life is Thinking!” (R.B. Thieme, Jr.)
SELECTION. SELECTION PROCESS BY WHICH AN ORGANIZATION CHOOSES PERSONS WHO BEST MEET THE SELECTION CRITERIA FOR THE POSITION AVAILABLE, CONSIDERING CURRENT.
1 Issues in Assessment in Higher Education: Science Higher Education Forum on Scientific Competencies Medellin-Colombia Nov 2-4, 2005 Dr Hans Wagemaker.
Logo Driving task analysis as a tool in traffic safety reasearch and practice Safety Science(2006)
The present publication was developed under grant X from the U.S. Department of Education, Office of Special Education Programs. The views.
Managing Human Resources, 12e, by Bohlander/Snell/Sherman © 2001 South-Western/Thomson Learning 5-1 Managing Human Resources Managing Human Resources Bohlander.
Competency in Older Adults: Clinical and Legal Perspectives The Role of Cognitive and Neuropsychological Evaluations John Crumlin, PhD Assistant Director,
On Layers and Objects in Assessment Design Robert Mislevy, University of Maryland Michelle Riconscente, University of Maryland Robert Mislevy, University.
Construct-Centered Design (CCD) What is CCD? Adaptation of aspects of learning-goals-driven design (Krajcik, McNeill, & Reiser, 2007) and evidence- centered.
Assessing Your Assessments: The Authentic Assessment Challenge Dr. Diane King Director, Curriculum Development School of Computer & Engineering Technologies.
Software Safety Case Why, what and how… Jon Arvid Børretzen.
Copyright © Allyn & Bacon 2008 Intelligent Consumer Chapter 14 This multimedia product and its contents are protected under copyright law. The following.
LEARNER CENTERED APPROACH
Chapter 3 Human Resource Development
Validity and Item Analysis Chapter 4.  Concerns what instrument measures and how well it does so  Not something instrument “has” or “does not have”
Georgia will lead the nation in improving student achievement. 1 Georgia Performance Standards Day 3: Assessment FOR Learning.
Evidence-Based Practice Evidence-Based Practice Current knowledge and practice must be based on evidence of efficacy rather than intuition, tradition,
Spring 2015 Kyle Stephenson
Lifelong Learning: Construction of competencies and knowledge.
Chapter 6: High-Leverage Practice 1: Formative Evaluation.
AERA April 2005 Models and Tools for Drawing Inferences from Student Work: The BEAR Scoring Engine Cathleen Kennedy & Mark Wilson University of California,
姓名 : 許浩維 學號 :M 日期 : Road Accident: Driver Behaviour, Learning and Driving Task 1.
Research Methodology II Term review. Theoretical framework  What is meant by a theory? It is a set of interrelated constructs, definitions and propositions.
Assessing Math Looking closer at PARCC Task Types 2.
Educational Research Chapter 8. Tools of Research Scales and instruments – measure complex characteristics such as intelligence and achievement Scales.
Knowing What Students Know Ganesh Padmanabhan 2/19/2004.
Regulation of Statutory National Assessments l. Contents ■Ofqual Responsibilities ■Regulation at GCSE ■The Regulatory Framework □Statutory Objectives.
2018 PISA Global Competency Assessment
Assessment Fall Quarter 2007.
VALIDITY by Barli Tambunan/
Using Cognitive Science To Inform Instructional Design
Assessments for Monitoring and Improving the Quality of Education
Smarter Balanced Assessment Results
Airmanship or Temple of “Doom” ? Airmanship.
Clinical Assessment Dr. H
Classroom Assessment Validity And Bias in Assessment.
Reliability and Validity of Measurement
Assessment Directors WebEx March 29, :00-3:00 pm ET
Job Analysis CHAPTER FOUR Screen graphics created by:
From Learning to Testing
Assessment Fall Quarter 2007.
Deanna L. Morgan The College Board
Presentation transcript:

Show Me an Evidential Approach to Assessment Design Michael Rosenfeld F. Jay Breyer David M. Williamson Barbara Showers

A Brief Introduction to Evidence Centered Design (ECD) Michael Rosenfeld, Ph.D. Rosenfeld & Associates

What is Evidence Centered Design? A formal framework for assessment design Emphasizes assessment as an evidential argument Encourages models and representations that strengthen validity arguments Focuses the test development process to better target decisions of interest

The Basic Models of ECD Proficiency Model – What you want to measure Evidence Model – How to recognize & interpret observable evidence of unobservable proficiencies Task Models – How to elicit valid and reliable evidence

ECD Validity Chain Evidence Proficiency Tasks Task Model Task Model Task Model Task Model

What is Gained from ECD? Better understanding of the domain, for new constructs Explicit chain of evidence from assessment goals, to models of evidence, to assessment tasks Strong construct validity argument Rapid and efficient item generation from models automatic item generation Opportunities for innovation, task types other than multiple choice Philosophical shift from content-authoring to evidential argument

Typical Test Development Content Specification Item Writing Test Blueprint Analysis Cut Scores The Job Knowledge & Tasks Major Dimensions Job Analysis Incumbent Survey Content Linking Statistical Modeling

Typical Test Development and Evidence Centered Design Content Specification Item Writing Test Blueprint Analysis Cut Scores Typical Test DevelopmentEvidence Centered Design Proficiency Model Task Model Evidence Model Tasks

Evidence Centered Design Proficiency Model Task Model Evidence Model Tasks Claims (Criterion) Knowledge, Strategies & Skills Reporting Variables Content Linking Evidential Argument Job Analysis Cognitive Task Analysis Statistical Modeling Task Design

Still to Come Example of the Proficiency Modeling process (Jay) Illustration of Evidence Modeling and Task Modeling (David) Commentary from an applied perspective (Barbara)

Proficiency Models in the ECD Framework F. Jay Breyer, Ph.D.

Proficiency Model Combination of Formalization of assessment goals, population and purpose Formal claims to be made on the basis of the assessment He/she is competent to operate an automobile Proficiencies of interest Knowledge of rules, procedures, and laws Ability to implement driving regulations in practice Skill in automobile operation

What is a Proficiency? Proficiencies are knowledge, skills and abilities that provide the basis for making claims about people

Building a Proficiency Model: Assessment Goals Assessment Goals Protect the public health, safety & welfare Examinee population All eligible (age, prior training requirements, etc.) citizens self-selecting into the program Test users/stakeholders State regulatory bodies State resident drivers Other citizens exposed to driving Result Usage/Decisions State issuing the license (to issue or deny) Means of official identification

Building a Proficiency Model: Assessment Claims Formal Claims are developed for assessment goals Claims can be course-grained and general Can safely operate a motor vehicle These general claims can be be supported by a variety of more specific sub-claims in a hierarchy Operates a vehicle in conformance with posted signs Stops at a posted stop sign Obeys posted speed limits The final hierarchy of claims drives the evidential argument and the score reporting needs of the test

Building a Proficiency Model: Proficiencies Proficiency variables are the basis of making claims Proficiency variables, like claims, may be organized hierarchically Values or levels of proficiency variables may be selected to support claim distinctions Informed by investigations of the domain and learning within the domain job analyses, cognitive task analyses, learning models, etc.

Example of a Hypothetical Proficiency Model for Driving

Still to Come Illustration of Evidence Modeling and Task Modeling (David) Commentary from an applied perspective (Barbara)

Providing the Evidence in Evidence Centered Design David M. Williamson, Ph.D.

Building Evidence Models Objective: valid inference Understanding what constitutes evidence and why Situations that elicit evidential behaviors Task design Statistical representation of evidential value of observations Integration of scoring with design ECD models – conceptual scoring

Establishing Evidence in a “Perfect World” Targets of Inference Links behaviors and inference Behavior Elicitation Driving Ability Knowledge of RotR Psychomotor Evidence Knowledge declarations Performance Task Requirements Situation Outcomes

Establishing Evidence: Driver’s Exam Knowledge of RotR Weather & Light Traffic Flow Intersections Speed Psychomotor Intersections Speed Turning Parking Driving Ability Knowledge of RotR Psychomotor Intersections Recognizing signs Specifying intersection traffic behaviors Following proper procedure

Building Task Models: Why? Elicits behaviors by predictably manipulating Focus Difficulty Context Targets what we want to measure Provides a framework for generating multiple tasks targeting same proficiencies

Building Task Models: What? To target elicitation of particular evidence, the task model specifies: Tools provided Directions and stimulus materials Indication of data collected and retained for scoring Relevance of specified elements to particular evidence, and particular proficiencies

Evidence as Inference about Proficiency Stage 1: Evidence Identification Identify and summarize response elements Stage 2: Evidence Accumulation Using these elements to estimate ability

Beneficial Outcomes of ECD Development direction w evidential focus Integration of expertise/activities (TD, stat, etc.) in the design process Chain of reasoning Integrated design – implications of design changes Validity argument Item modeling AIG Defensibility Directed item writing – future forms Somewhat more expensive (Initially)

Still to Come Commentary from an applied perspective (Barbara)

Discussant Barbara Showers, Ph.D. Director, Office of Education and Examinations Wisconsin Department of Regulation and Licensing