1. LTRC 2007 Messick Address Slide 1 June 9, 2007 Toward a Test Theory for the Interactionalist Era Robert J. Mislevy University of Maryland Samuel J. Messick Memorial Lecture Presented at the Language Testing Research Colloquium, Barcelona, Spain, June 9, 2007.

2. LTRC 2007 Messick Address Slide 2 June 9, 2007 Chi, Feltovich, & Glaser, 1981

3. LTRC 2007 Messick Address Slide 3 June 9, 2007 Key Ideas Generative patterns from the domain organized experts’ perceptions, understanding, and actions Experts assembled pieces from patterns to fit particulars of context and purpose »F=MA: Conventional v. situated meaning Critical role of narrative layer »Integrates principles & equations with context »Locus for understanding, planning, & action

4. LTRC 2007 Messick Address Slide 4 June 9, 2007 Why Am I Telling You This? Situative / sociocultural / interactionist perspective is frontier in all domains Analogues between model-based reasoning and using language Test theory is model-based reasoning »Interactionalist perspective on test theory »Bottleneck the narrative layer

5. LTRC 2007 Messick Address Slide 5 June 9, 2007 Outline Messick on assessment arguments Interactionalist perspective »Re language, comprehension, cultural meaning A narrative space / metaphor for assessment in this light »Attention to senses and roles of context Implications for building and using measurement models

6. LTRC 2007 Messick Address Slide 6 June 9, 2007 The Assessment Argument What complex of knowledge, skills, or other attributes should be assessed? What behaviors or performances should reveal those constructs? What tasks or situations should elicit those behaviors? Messick (1994) “The interplay of evidence and consequences in the validation of performance assessments”

7. LTRC 2007 Messick Address Slide 7 June 9, 2007 Perspectives on the L2 Construct What is foregrounded? Trait »Focus on underlying abilities of individuals that are called upon in a wide variety of situations. Behaviorist »Focus on context, from external point of view— success of action in specified classes of situations. Interactionalist …

8. LTRC 2007 Messick Address Slide 8 June 9, 2007 An Interactionalist Perspective Concern with language used in specific discursive practices rather than on language ability independent of context. Focus on the co-construction of discursive practices by all participants... A set of general interactional resources that participants draw upon in specific ways in order to co-construct a discursive practice. (Young, 2000, p. 5)

9. LTRC 2007 Messick Address Slide 9 June 9, 2007 An Interactionalist Perspective successful interaction presupposes not only a shared knowledge of the world, … but also the construction of a shared internal context … that is built through the collaborative efforts of the interactional partners. Kramsch ( 1986, p. 367)

10. LTRC 2007 Messick Address Slide 10 June 9, 2007 Challenges for Assessment Amending the construct of individual ability to accommodate [how] language use in a communicative event reflects dynamic discourse, which is co-constructed among participants; and … reconciling [the notion that language ability is local] with the need for assessments to yield scores to generalize across contextual boundaries. Chalhoub-Deville (2003, p. 373)

11. LTRC 2007 Messick Address Slide 11 June 9, 2007 A Narrative Frame Themes from, e.g., cognitive psychology, literacy, neuroscience, anthropology: »Connectionist metaphor, Associative memory Situated cognition & information processing »Construction-Integration (CI) theory of comprehension (Kintsch and others) Individual  Sociocultural perspectives »A cognitive theory of cultural meaning (Strauss & Quinn, 1997)

12. LTRC 2007 Messick Address Slide 12 June 9, 2007 A Cognitive Theory of Cultural Meaning  “Interactional Resources” External: knowledge “out there”, patterns that exist in use by groups, in tools & processes »Cultural models: What ‘being sick’ means, restaurant script, Newton’s laws, complaints »Linguistic: Grammar, conventions, frames Interactional: enable the co-construction of new shared meanings Internal: patterns in individuals attuned to shared external patterns

13. LTRC 2007 Messick Address Slide 13 June 9, 2007 Inheritance from Schema Theory Knowledge as organized patterns, at many levels… Assembled to understand & to create particular situations in the world Developed, strengthened, modified by use Associations of all kinds, including applicability, affordances, procedures, strategies, affect »“The user’s knowledge of the language rules is interlocked with his knowledge of when, where, and with whom to use them” (Ellis, 1985)

14. LTRC 2007 Messick Address Slide 14 June 9, 2007 B Inside B Inside A Context A A la Kintsch: “Conventional” meaning, or propositional content of text / speech… and all aspects of context…

15. LTRC 2007 Messick Address Slide 15 June 9, 2007 B Inside B Inside A Context A The C in CI theory, Construction: Activation of both relevant and irrelevant … Linguistic models, e.g. Conventions, Rhetorical frames Cultural models, e.g., Equilibrium, Human motivation The C in CI theory, Construction: Activation of both relevant and irrelevant … Linguistic models, e.g. Conventions, Rhetorical frames Cultural models, e.g., Equilibrium, Human motivation If B hasn’t developed a given pattern in past experience, it can’t be activated (although it may get constructed in the interaction). Relevant patterns from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)). If B hasn’t developed a given pattern in past experience, it can’t be activated (although it may get constructed in the interaction). Relevant patterns from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)).

16. LTRC 2007 Messick Address Slide 16 June 9, 2007 B Inside B Inside A Context A The I in CI theory, Integration: Resulting synthesis of reinforced activated cultural / linguistic / situational patterns: the Situation model Akin to ‘stories’ in Larkin’s physics study Situation model is the understanding The I in CI theory, Integration: Resulting synthesis of reinforced activated cultural / linguistic / situational patterns: the Situation model Akin to ‘stories’ in Larkin’s physics study Situation model is the understanding

17. LTRC 2007 Messick Address Slide 17 June 9, 2007 B Inside B Inside A Context A Situation model is also the basis of planning and action.

18. LTRC 2007 Messick Address Slide 18 June 9, 2007 B Inside B Inside A Context A

19. LTRC 2007 Messick Address Slide 19 June 9, 2007 A B Inside B Inside A Context Ideally, participants activate cultural & linguistic models that are compatible in relevant ways…

20. LTRC 2007 Messick Address Slide 20 June 9, 2007 A B Inside B Inside A Context To lead to co-constructed meaning / sufficiently (?) shared understanding Kramsch’s "shared internal context"

21. LTRC 2007 Messick Address Slide 21 June 9, 2007 A B Inside B Inside A Context Preceeding overheads highlight the importance of a common narrative space for thinking about assessment: A = Linguist B = Psychometrician Preceeding overheads highlight the importance of a common narrative space for thinking about assessment: A = Linguist B = Psychometrician “shared internal context” re nature & use of knowledge would help ground compatible views of assessment purpose, design, analysis, and use for the job at hand.

22. LTRC 2007 Messick Address Slide 22 June 9, 2007 Senses and roles of “context” A B Inside B Inside A Context Can distinguish external and internal aspects of context (e.g., Douglas, 2000) Some external aspects of context are public & objective, e.g., Setting Physical attributes Directives Some external aspects of context are public & objective, e.g., Setting Physical attributes Directives Target language use (TLU) features Assessment task features (Bachman & Palmer) Target language use (TLU) features Assessment task features (Bachman & Palmer)

23. LTRC 2007 Messick Address Slide 23 June 9, 2007 A B Inside B Inside A Context Some aspects of context can be interpreted by an external observer through the lens of targeted cultural/linguistic models. e.g., apology situation. In assessment, we can often arrange for this to be the case. Some aspects of context can be interpreted by an external observer through the lens of targeted cultural/linguistic models. e.g., apology situation. In assessment, we can often arrange for this to be the case. The question at issue in assessment is whether the examinee activates targeted compatible l/c models, then constructs and acts accordingly through a corresponding situation model. The question at issue in assessment is whether the examinee activates targeted compatible l/c models, then constructs and acts accordingly through a corresponding situation model. Note the need to activate many other l/c models in order to construct a situation model, plan, and carry out action. Many places to slip, but others to compensate. “Alternative explanations” in assessment argument. Note the need to activate many other l/c models in order to construct a situation model, plan, and carry out action. Many places to slip, but others to compensate. “Alternative explanations” in assessment argument.

24. LTRC 2007 Messick Address Slide 24 June 9, 2007 A B Inside B Inside A Context Some aspects of context can be interpreted by an external observer through the lens of targeted cultural/linguistic models. e.g., apology situation. In assessment, we can sometimes arrange for this to be the case; sometimes watch for it to happen. Some aspects of context can be interpreted by an external observer through the lens of targeted cultural/linguistic models. e.g., apology situation. In assessment, we can sometimes arrange for this to be the case; sometimes watch for it to happen. As assessment designers, we use these situations that call upon targeted linguistic/ cultural models to determine what examinee actions would signal recognition, comprehension, action through them. This sense of context plays a key role in Evaluation of performance, hence Observable variables that go into a measurement model. This sense of context plays a key role in Evaluation of performance, hence Observable variables that go into a measurement model.

25. LTRC 2007 Messick Address Slide 25 June 9, 2007 What can we say about individuals? Use of interactional resources in appropriate contexts in appropriate ways; i.e., Attunement to targeted cultural/linguistic patterns: Recognize markers of externally-viewed patterns? Construct internal meanings in their light? Act in ways appropriate to targeted c/l models in the assessment contexts? What are the ranges and circumstances of activation? (variation of performance across contexts)

26. LTRC 2007 Messick Address Slide 26 June 9, 2007 Implications for measurement models Basic form: Probability of aspects of performance X ij given parameters for person i and situation j (all could be vector-valued) Way too simple No explicit connection with CI comprehension model, interaction processes, etc. Apparent separation of person and situation characteristics These are indeed properties of the conventional meaning of the measurement model and parameters.

27. LTRC 2007 Messick Address Slide 27 June 9, 2007 An Interactionalist Perspective: Instantiation in a Context Xs result from particular persons calling upon resources in particular contexts (or not, or how) Mechanically  s simply accumulate info across situations Our chosing situations and what to observe drives their situated meaning. Situated meaning of  s are tendencies toward these actions in these situations that call for certain interactional resources, via l/c models.

28. LTRC 2007 Messick Address Slide 28 June 9, 2007 An Interactionalist Perspective: Instantiation in a Context Inference to criterion contexts (TLU) depends on analysis of what l/c models are called upon in assessment use argument… What is similar, what is different, re the resources task & criterion situations call for? To what degree does activation and success in task context correspond to activation and success in criterion context? (e.g., Bachman, Chalhoub-Deville, Douglas, Chapelle)

29. LTRC 2007 Messick Address Slide 29 June 9, 2007 An Interactionalist Perspective: Instantiation in a Context What demands do we minimize via task design, so needn’t model? What resources do we already know examinees can draw upon, so tasks can require them but we needn’t model? »“Hidden” but essential to meaning »Occupational English Test (McNamara) »Analogous to ‘focus on forms’ learning

30. LTRC 2007 Messick Address Slide 30 June 9, 2007 An Interactionalist Perspective: Instantiation in a Context How to model inconsistent performance? If “unmotivated,” it’s noise; via probability model Promising direction: Model individual’s degree or pattern in variation in terms of context features If “motivated”: Model in terms of  s »Divide & Conquer: Multiple unidimensional tests (OET) »Exploratory multidimensional: Discover patterns in data. »Controlled: Structured multidimensional models (e.g., Embretson, Adams & Wilson, von Davier)

31. LTRC 2007 Messick Address Slide 31 June 9, 2007 Structured Multidimensional Models Still way too simple, but … »purposeful modeling of motivated variation in complex tasks when persons differ in targeted ways »exploit what is known about examinees but not modeled Controlled mixes of demand features »E.g., in OET-like situations, wrt medical knowledge, complexity of stimulus language, complexity of language to be produced. “Throwing the data over the wall” won’t work

32. LTRC 2007 Messick Address Slide 32 June 9, 2007 Structured Multidimensional Models Probabilities modeled in terms of task features: »Which dimensions are relevant for which observables from which tasks? (Robinson’s “difficulty” features) »Task parameters modeled in terms of Robinson’s “complexity” features. Hence a priori structure of patterns to interpret »Can organize  s in terms of traits or context features Coordinated task design and measurement model »Create tasks within task models

33. LTRC 2007 Messick Address Slide 33 June 9, 2007 How much can testing gain from modern cognitive psychology? So long as testing is viewed as something that takes place in a few hours, out of the context of instruction, and for the purpose of predicting a vaguely stated criterion, then the gains to be made are minimal. Buzz Hunt (1986) Conclusion

34. LTRC 2007 Messick Address Slide 34 June 9, 2007 I have argued that we need to capitalize on [method effects] by designing tests for specific populations -- tests that contain instructions, content, genre, and language directed toward that population. The goal is to produce tests … that would provide information interpretable as evidence of communicative competence in context. Douglas (1998) Conclusion

35. LTRC 2007 Messick Address Slide 35 June 9, 2007 Conclusion Interactionalist view of test theory… for assembling, analyzing, and interpreting assessments, for arguments in interactionalist view of language Methods and exemplars needed, but more pressing need is narrative frame … »To connect view of language proficiency with the machinery of test theory, »Toward modeling purposeful variations in a coherent design space.