FERA 2001 Slide 1 November 6, 2001 Making Sense of Data from Complex Assessments Robert J. Mislevy University of Maryland Linda S. Steinberg & Russell G. Almond Educational Testing Service FERA November 6, 2001

FERA 2001 Slide 2 November 6, 2001 How much can testing gain from modern cognitive psychology? So long as testing is viewed as something that takes place in a few hours, out of the context of instruction, and for the purpose of predicting a vaguely stated criterion, then the gains to be made are minimal. Buzz Hunt, 1986:

FERA 2001 Slide 3 November 6, 2001 Opportunities for Impact Informal / local use Conceptual design frameworks  E.g., Grant Wiggins, CRESST Toolkits & building blocks  E.g., Assessment Wizard, IMMEX Building structures into products  E.g., HYDRIVE, Mavis Beacon Building structures into programs  E.g., AP Studio Art, DISC

FERA 2001 Slide 4 November 6, 2001 For further information, see...

FERA 2001 Slide 5 November 6, 2001 Don Melnick, NBME: “It is amazing to me how many complex ‘testing’ simulation systems have been developed in the last decade, each without a scoring system. “The NBME has consistently found the challenges in the development of innovative testing methods to lie primarily in the scoring arena.”

FERA 2001 Slide 6 November 6, 2001 The DISC Project The Dental Interactive Simulations Corporation (DISC) The DISC Simulator The DISC Scoring Engine Evidence-Centered Assessment Design The Cognitive Task Analysis (CTA)

FERA 2001 Slide 7 November 6, 2001 Evidence-centered assessment design The three basic models

FERA 2001 Slide 8 November 6, 2001 What complex of knowledge, skills, or other attributes should be assessed? (Messick, 1992 ) Evidence-centered assessment design

FERA 2001 Slide 9 November 6, 2001 What complex of knowledge, skills, or other attributes should be assessed? (Messick, 1992 ) Student Model Variables Evidence-centered assessment design

FERA 2001 Slide 10 November 6, 2001 What behaviors or performances should reveal those constructs? Evidence-centered assessment design

FERA 2001 Slide 11 November 6, 2001 What behaviors or performances should reveal those constructs? Work product Evidence-centered assessment design

FERA 2001 Slide 12 November 6, 2001 What behaviors or performances should reveal those constructs? Work product Observable variables Evidence-centered assessment design

FERA 2001 Slide 13 November 6, 2001 What behaviors or performances should reveal those constructs? Observable variables Evidence-centered assessment design

FERA 2001 Slide 14 November 6, 2001 What behaviors or performances should reveal those constructs? Observable variables Evidence-centered assessment design Student Model Variables

FERA 2001 Slide 15 November 6, 2001 What tasks or situations should elicit those behaviors? Evidence-centered assessment design

FERA 2001 Slide 16 November 6, 2001 What tasks or situations should elicit those behaviors? Stimulus Specifications Evidence-centered assessment design

FERA 2001 Slide 17 November 6, 2001 What tasks or situations should elicit those behaviors? Work Product Specifications Evidence-centered assessment design

FERA 2001 Slide 18 November 6, 2001 Implications for Student Model SM variables should be consistent with … The results of the CTA. The purpose of assessment: What aspects of skill and knowledge should be used to accumulate evidence across tasks, for pass/fail reporting and finer-grained feedback?

FERA 2001 Slide 19 November 6, 2001 Simplified Version of the DISC Student Model

FERA 2001 Slide 20 November 6, 2001 Implications for Evidence Models The CTA produced ‘performance features’ that characterize recurring patterns of behavior and differentiate levels of expertise. These features ground generally-defined, re-usable ‘observed variables’ in evidence models. We defined re-usable evidence models for recurring scenarios for use with many tasks.

FERA 2001 Slide 21 November 6, 2001 An Evidence Model

FERA 2001 Slide 22 November 6, 2001 Evidence Models: Statistical Submodel What’s constant across cases that use the EM »Student-model parents. »Identification of observable variables. »Structure of conditional probability relationships between SM parents and observable children. What’s tailored to particular cases »Values of conditional probabilities »Specific meaning of observables.

FERA 2001 Slide 23 November 6, 2001 Evidence Models: Evaluation Submodel What’s constant across cases »Identification and formal definition of observable variables. »Generally-stated “proto-rules” for evaluating their values. What’s tailored to particular cases »Case-specific rules for evaluating values of observables-- Instantiations of proto-rules tailored to the specifics of case.

FERA 2001 Slide 24 November 6, 2001 “Docking” an Evidence Model Evidence Model Student Model

FERA 2001 Slide 25 November 6, 2001 “Docking” an Evidence Model Evidence Model Student Model

FERA 2001 Slide 26 November 6, 2001 Initial Status Expert.28 Competent.43 Novice.28 All.33 Some.33 None.33

FERA 2001 Slide 27 November 6, 2001 Expert.39 Competent.51 Novice.11 All 1.00 Some.00 None.00 Status after four ‘good’ findings

FERA 2001 Slide 28 November 6, 2001 Expert.15 Competent.54 Novice.30 All.00 Some.00 None 1.00 Status after one ‘good’ and three ‘bad’ findings

FERA 2001 Slide 29 November 6, 2001 “Docking” another Evidence Model Evidence Model Student Model

FERA 2001 Slide 30 November 6, 2001 “Docking” another Evidence Model Evidence Model Student Model

FERA 2001 Slide 31 November 6, 2001 Implications for Task Models Task models are schemas for phases of cases, constructed around key features that... the simulator needs for its virtual-patient data base, characterize features we need to evoke specified aspects of skill/knowledge, characterize features of tasks that affect difficulty, characterize features we need to assemble tasks into tests.

FERA 2001 Slide 32 November 6, 2001 Implications for Simulator Once we’ve determined the kind of evidence we need as evidence about targeted knowledge, how must we construct the simulator to provide the data we need? Nature of problems »Distinguish phases in the patient interaction cycle. »Use typical forms of information & control availability. »Dynamic patient condition & cross time cases. Nature of affordances »Examinees must be able to seek and gather data, »indicate hypotheses, »justify hypotheses with respect to cues, »justify actions with respect to hypotheses.

FERA 2001 Slide 33 November 6, 2001 Payoff Re-usable student-model » Can project to overall score for licensing » Supports mid-level feedback as well Re-usable evidence and task models » Can write indefinitely many unique cases using schemas » Framework for writing case-specific evaluation rules Machinery can generalize to other uses & domains

FERA 2001 Slide 34 November 6, 2001 Two ways to “score” complex assessments THE HARD WAY: Ask ‘how do you score it?’ after you’ve built the assessment and scripted the tasks or scenarios. A DIFFERENT HARD, BUT MORE LIKELY TO WORK, WAY: Design the assessment and the tasks/scenarios around what you want to make inferences about, what you need to see to ground them, and the structure of the interrelationships. Part 2 Conclusion

FERA 2001 Slide 35 November 6, 2001 We can attack new assessment challenges by working from generative principles: Principles from measurement and evidentiary reasoning, coordinated with... inferences framed in terms of current and continually evolving psychology, using current and continually evolving technologies to help gather and evaluate data in that light, in a coherent assessment design framework. Grand Conclusion

FERA 2001 Slide 36 November 6, 2001