Maryland Validity ConferenceSlide 1October 10, 2008 Validity from the Perspective of Model-Based Reasoning Robert J. Mislevy Measurement, Statistics and Evaluation University of Maryland, College Park Presented at the conference “The Concept of Validity: Revisions, New Directions and Applications,” University of Maryland, College Park, MD October 9-10, Supported by a grant from the Spencer Foundation.

Maryland Validity ConferenceSlide 2October 10, 2008 Overview of the Talk l Sources of unease l Cognition in terms of patterns l Model-based reasoning l Measurement models as model-based reasoning l Implications for validity l Feeling better now

Maryland Validity ConferenceSlide 3October 10, 2008 Sources of Unease (1) Different models fit the same data lTatsuoka (1983) mixed number subtraction

Maryland Validity ConferenceSlide 4October 10, 2008 Sources of Unease (1) Cognitive diagnosis model for instruction Student characterized by vector of 0/1 variables, say , for which operations she had mastered lTask characterized by which ones the task needed lProbability of correct response via latent class model 2PL IRT model for overall proficiency Student characterized by univariate, continuous , for proficiency in the domain lTasks modeled by difficulty & discrimination lProbability of correct response via IRT model Container metaphor Person B Person D Measurement metaphor Item 1Item 4Item 5Item 3Item 6Item 2 Person APerson BPerson D

Maryland Validity ConferenceSlide 5October 10, 2008 Sources of Unease (2) Summary test scores, and factors based on them, have often been though of as “signs” indicating the presence of underlying, latent traits. … An alternative interpretation of test scores as samples of cognitive processes and contents … is equally justifiable and could be theoretically more useful. Snow & Lohman, 1989, p. 317

Maryland Validity ConferenceSlide 6October 10, 2008 Sources of Unease (2) The evidence from cognitive psychology suggests that test performances are comprised of complex assemblies of component information-processing actions that are adapted to task requirements during performance. Snow & Lohman, 1989, p. 317

Maryland Validity ConferenceSlide 7October 10, 2008 Sources of Unease (2) The implication is that sign-trait interpretations of test scores and their intercorrelations are superficial summaries at best. At worst, they have misled scientists, and the public, into thinking of fundamental, fixed entities, measured in amounts. Snow & Lohman, 1989, p. 317

Maryland Validity ConferenceSlide 8October 10, 2008 Sources of Unease (2) Whatever their practical value as summaries, for selection, classification, certification, or program evaluation, the cognitive psychological view is that such interpretations no longer suffice as scientific explanations of aptitude and achievement constructs. Snow & Lohman, 1989, p. 317

Maryland Validity ConferenceSlide 9October 10, 2008 Sources of Unease (3) What is the nature of parameters like  and  ? Where are they? l What is the interpretation of the probabilities that arise from IRT, latent class / cognitive diagnosis models, and the like? l What does this mean about validity of the data / the models / the uses of them?

Maryland Validity ConferenceSlide 10October 10, 2008 Cognition in Terms of Patterns l The sociocognitive paradigm l Metaphors as foundation l Formal model-based reasoning

Maryland Validity ConferenceSlide 11October 10, 2008 The sociocognitive paradigm Converging ideas from cog psych, neurology, anthropology, linguistics, science ed, etc. Knowledge as patterns, at many levels… Assembled to understand, to interact with, and to create particular situations in the world Developed, strengthened, modified by use Associations of all kinds, including applicability, affordances, procedures, strategies, affect

Maryland Validity ConferenceSlide 12October 10, 2008 Walter Kintsch’s CI Theory of Reading Comprehension More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition. TextText baseSituation ModelContext Context 1 LTM Kintsch is focusing here on “experiential” cognition – not conscious, occurring at the scale of milliseconds. We’ll talk about reflective cognition in a couple minutes. Kintsch is focusing here on “experiential” cognition – not conscious, occurring at the scale of milliseconds. We’ll talk about reflective cognition in a couple minutes.

Maryland Validity ConferenceSlide 13October 10, 2008 Walter Kintsch’s CI Theory of Reading Comprehension More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition. TextText baseLTMSituation ModelActionContext Context 1 Context 2

Maryland Validity ConferenceSlide 14October 10, 2008 Walter Kintsch’s CI Theory of Reading Comprehension More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition. TextText baseLTMSituation ModelActionContext Context 1 Context 2

Maryland Validity ConferenceSlide 15October 10, 2008 Walter Kintsch’s CI Theory of Reading Comprehension More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition. TextText baseLTMSituation ModelActionContext Context 2 Context 3

Maryland Validity ConferenceSlide 16October 10, 2008 Metaphors as foundation Lakoff & Johnson »Metaphors we live by (1980); Philosophy in the flesh (1999) Key idea: »Cognitive machinery builds from capabilities for interacting with the real physical and social world. »We extend and creatively recombine basic patterns and relationships to think about everything from … everyday things to extremely complicated and abstract social, conceptual, philosophical realms True of both experiential and reflective cognition.

Maryland Validity ConferenceSlide 17October 10, 2008 Metaphors as foundation Example: Containers Free Clip Art Provided by Artclips.com

Maryland Validity ConferenceSlide 18October 10, 2008 Metaphors as foundation Example: Containers Everyday experience  Set theory »Very good, mostly. Knowledge as collection of discrete things inside our heads »Usually good and useful, in communication »Sometimes inapt, as sole basis of instructional practice and assessment design (the Jeopardy model of cognition—Rosie Perez in White men can’t jump) Example: Containers

Maryland Validity ConferenceSlide 19October 10, 2008 Metaphors as foundation Example: Cause & Effect

Maryland Validity ConferenceSlide 20October 10, 2008 Metaphors as foundation Example: Cause & Effect Newton’s laws; kinematics; quantitative models of force and motion, esp. F=MA

Maryland Validity ConferenceSlide 21October 10, 2008 Metaphors as foundation Example: Cause & Effect  xjxj IRT & SEM models; quantitative models for response probabilities, esp. Rasch’s P= 

Maryland Validity ConferenceSlide 22October 10, 2008 Metaphors as foundation Example: Cause & Effect Everyday experience  F=MA »Very good, mostly. Teleological theories of history, a la Hegel »Not so good, mostly. Example: Cause & Effect Everyday experience  F=MA »Very good, mostly.

Maryland Validity ConferenceSlide 23October 10, 2008 Model-Based Reasoning Real-World Situation Reconceived Real-World Situation Entities and relationships Representational Form A y=ax+b (y-b)/a=x Representational Form B Mappings among representational systems Mainly semantic Mainly syntactic

Maryland Validity ConferenceSlide 24October 10, 2008 Properties of Models (1) Human way to think about complex unique situations Abstract structure of entities, relationships, processes What’s included, what’s omitted Levels of analysis and grainsize »Newtonian and quantum mechanics »Transmission genetics at level of species, individuals, cells, or molecules

Maryland Validity ConferenceSlide 25October 10, 2008 Properties of Models (2) Can apply different models to same situation »Can view selling car to brother-in-law in terms of economic transaction model vs family relationships model Models tuned to uses / problems / purposes »Mixed number subtraction

Maryland Validity ConferenceSlide 26October 10, 2008 Properties of Models (2) The modeling cycle: Evaluate Revise Model Observe Predict/Use »Fit? »Does it work? »What’s left out? »Adequacy of rationale?

Maryland Validity ConferenceSlide 27October 10, 2008 Models with probabilistic layers Probability from analogy with physical games of chance (Shafer) Probability connects to model representation »Key in model criticism Model posits space for patterns; parameter values characterize them; probability models can characterize … »Variation in patterns »Modeler’s uncertainty about patterns & parameters

Maryland Validity ConferenceSlide 28October 10, 2008 Psychometric / Measurement Models E.g., IRT, CTT, FA, SEM, CDM Model posits space for patterns, parameter values characterize them Semantic layer is cause & effect metaphor »Q: In what sense does  “cause” X? »A: The C&E metaphor grounds productive connection between observations and inferences Modeling patterns across people, not explaining item responses (Snow & Lohman) »Could model within-person processes at finer grainsize

Maryland Validity ConferenceSlide 29October 10, 2008 Some answers What is the nature of parameters like  and  ? Where are they? »These are characterizations of patterns we observe in real-world situations (ones we in part construct for target uses) through the lens of a simplified model we are (provisionally) using to think about those situations and the use situations in which the patterns are apt to be relevant. »So they are in our heads, but they aren’t worth much unless they reflect patterns in examinees’ actions in the world.

Maryland Validity ConferenceSlide 30October 10, 2008 Some answers l What is the interpretation of the probabilities that arise from IRT, latent class / cognitive diagnosis models, and the like? »These are characterizations of patterns we observe in situations and our degree of knowledge about them, again through the lens of a simplified model we are (provisionally) using to think about those situations. »In addition to guiding inference through the model, they provide tools for seeing where the model may be misleading, inadequate.

Maryland Validity ConferenceSlide 31October 10, 2008 Some answers l What does this mean about validity of the data / the models / the uses of them?

Maryland Validity ConferenceSlide 32October 10, 2008 Validity Evidence Real-World Situation Reconceived Real-World Situation Entities and relationships Representational Form A y=ax+b (y-b)/a=x Representational Form B Mappings among representational systems Theory and experience supporting the narrative/scientific frame Empirical evaluation of predictions / outcomes Theoretical and empirical grounding of task design Theoretical and empirical grounding of task-scoring procedures

Maryland Validity ConferenceSlide 33October 10, 2008 Validity Implications, Sense 1 The currently dominant view: Validity is an integrated evaluative judgement of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment. (Messick, 1989) Focus on situated use of data from test Consistent with MBR perspective; i.e., reasoning through psychometric model in particular situations & inferences.

Maryland Validity ConferenceSlide 34October 10, 2008 Validity Implications, Sense 2 Alternative (e.g., Wiley, Borsboom, Lissitz): [A] test is valid for measuring an attribute if and only if (a) the attribute exists and (b) variations in the attribute causally produce variations in the outcomes of the measurement procedure. (Borsboom et al, 2004) MBR view can omit specific uses, but »must consider range of situations and uses that are apt to be thought about effectively via the model. »Broader range consistent with scientific program, in opposition to Snow & Lohman quote. »Is realist but strong correspondence to existence of traits qua traits in individuals is not required.

Maryland Validity ConferenceSlide 35October 10, 2008 I am Feeling Better Now Model-based reasoning provides a way of thinking about validity that … is consistent with the practical methods that have developed to assure quality of inferences from assessments is realist, in constructive-realism and L&J’s “embodied realism” sense is consistent with developments in cognitive psychology, including the nature of scientific reasoning, and the meaning of probability.