(1) If Language is a Complex Adaptive System, What is Language Assessment? Presented at “Language as a Complex Adaptive System”, an invited conference.

Slides:



Advertisements
Similar presentations
Understanding American Citizenship
Advertisements

Second Language Acquisition
Contrastive Analysis, Error Analysis, Interlanguage
KEMENTERIAN PENDIDIKAN DAN KEBUDAYAAN BADAN PENGEMBANGAN SUMBER DAYA MANUSIA PENDIDIKAN DAN KEBUDAYAAN DAN PENJAMINAN MUTU PENDIDIKAN AUTHENTIC ASSESSMENT.
1 Content-based Interpretations of Test Scores Michael Kane National Conference of Bar Examiners Maryland Assessment Research Center for Education Success.
Module 2 Text Comprehension
Why this Research? 1.High School graduates are facing increased need for high degree of literacy, including the capacity to comprehend texts, but comprehension.
Cognitive Theories and Reading Comprehension Building Blocks of the Reading Process.
Theories of Second language Acquisition
Robert J. Mislevy & Min Liu University of Maryland Geneva Haertel SRI International Robert J. Mislevy & Min Liu University of Maryland Geneva Haertel SRI.
SRI Technology Evaluation WorkshopSlide 1RJM 2/23/00 Leverage Points for Improving Educational Assessment Robert J. Mislevy, Linda S. Steinberg, and Russell.
University of Maryland Slide 1 July 6, 2005 Presented at Invited Symposium K3, “Assessment Engineering: An Emerging Discipline” at the annual meeting of.
Ethnography. In ethnography, the researcher  Participates in people's daily lives for an extended period of time  Watches everyday happenings  Listens.
SLRF 2010 Slide 1 Oct 16, 2010 What is the construct in task-based language assessment? Robert J. Mislevy Professor, Measurement, Statistics and Evaluation.
Constructivism Constructivism — particularly in its "social" forms — suggests that the learner is much more actively involved in a joint enterprise with.
Consistency/Reliability
LTRC 2007 Messick Address Slide 1 June 9, 2007 Toward a Test Theory for the Interactionalist Era Robert J. Mislevy University of Maryland Samuel J. Messick.
AERA 2010 Robert L. Linn Lecture Slide 1 May 1, 2010 Integrating Measurement and Sociocognitive Perspectives in Educational Assessment Robert J. Mislevy.
Common Core State Standards Professional Learning Module Series
TASK-BASED INSTRUCTION Teresa Pica, PhD Presented by Reem Alshamsi & Kherta Sherif Mohamed.
Presented by: Mohsen Saberi and Sadiq Omarmeli  Language testing has improved parallel to advances in technology.  Two basic questions in testing;
On Knowing a Language1 Today Find out your own beliefs about language learning and teaching Start Chapter 1: What is it to know a language? Standards used.
FERA 2001 Slide 1 November 6, 2001 Making Sense of Data from Complex Assessments Robert J. Mislevy University of Maryland Linda S. Steinberg & Russell.
Historical Thinking Skills
Case Study Research By Kenneth Medley.
Science and Engineering Practices
Second language acquisition
ADL Slide 1 December 15, 2009 Evidence-Centered Design and Cisco’s Packet Tracer Simulation-Based Assessment Robert J. Mislevy Professor, Measurement &
Chapter 17 Ethnographic Research Gay, Mills, and Airasian
Foreign language and English as a Second Language: Getting to the Common Core of Communication. Are we there yet? Marisol Marcin
ESL Phases & ESL Scale Curriculum Corporation 1994.
WORLD LANGUAGES : A Year of Transition. Today’s Outcomes  Celebrate the start of the school year  Greet new teachers  Explore areas of focus.
NSW Curriculum and Learning Innovation Centre Draft Senior Secondary Curriculum ENGLISH May, 2012.
Some Implications of Expertise Research for Educational Assessment Robert J. Mislevy University of Maryland National Center for Research on Evaluation,
ELD Transition Sessions
ASSESSING LANGUAGE SKILLS
ELA Common Core Shifts. Shift 1 Balancing Informational & Literary Text.
1 Duschl, R & Osborne, J ”Supporting and Promoting Argumentation Discourse in Science Education” in Studies in Science Education, 38, Ingeborg.
Academic Needs of L2/Bilingual Learners
ATTRIBUTEDESCRIPTION Focal Knowledge, Skills, Abilities The primary knowledge / skills / abilities (KSAs) targeted by this design pattern. RationaleHow/why.
Historical Thinking Skills
REVISIONS TO GENERAL EDUCATION STUDENT LEARNING OUTCOMES Auburn University Senate Information Item, August 2014.
Advanced English - Modules
Historical Thinking Skills
Unpacking the Elements of Scientific Reasoning Keisha Varma, Patricia Ross, Frances Lawrenz, Gill Roehrig, Douglas Huffman, Leah McGuire, Ying-Chih Chen,
VALUE/Multi-State Collaborative (MSC) to Advance Learning Outcomes Assessment Pilot Year Study Findings and Summary These slides summarize results from.
Historical Thinking Skills A.P. World History Mr. Schabo Crestwood High School All info care of College Board:
Historical Thinking Skills. Skill Type I: Chronological Reasoning Skill 1: Historical Causation Historical thinking involves the ability to identify,
Anchor Standards ELA Standards marked with this symbol represent Kansas’s 15%
Robert J. Mislevy University of Maryland National Center for Research on Evaluation, Standards, and Student Testing (CRESST) NCME San Diego, CA April 15,
Common Core.  Find your group assignment.  As a group, read over the descriptors for mastery of this standard. (The writing standards apply to more.
Reliability performance on language tests is also affected by factors other than communicative language ability. (1) test method facets They are systematic.
First Language Acquisition
Chapter 6 - Standardized Measurement and Assessment
Paper III Qualitative research methodology.  Qualitative research is designed to reveal a specific target audience’s range of behavior and the perceptions.
Common Core State Standards in English/Language Arts What science teachers need to know.
Module 4—Literacy Strands Arts Education. Learning Outcomes Participants will: explore the relationship between the new Essential Standards and the Common.
Discourse Analysis Week 10 Riggenbach (1999) Chapter 1 - Quotes.
Eng. 429 What is Discourse Analysis? Lecture 1 Source: Paltridge (2006), p.p. 1-4.
Interactive Lecture 2: Discourse, Competency, Proficiency and the Implications for Methodology Dr. Douglas Fleming Faculty of Education.
Discourse analysis May 2012 Carina Jahani
Objectives of session By the end of today’s session you should be able to: Define and explain pragmatics and prosody Draw links between teaching strategies.
ASSESSMENT PRACTICES IN THE POST-COMMUNICATIVE ERA: A MULTILITERACIES PERSPECTIVE Heather Willis Allen – University of Wisconsin - Madison Beatrice Dupuy.
Common Core.  Find your group assignment.  As a group, read over the descriptors for mastery of this standard. (The writing standards apply to more.
Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 47 Critiquing Assessments.
Dr Anie Attan 26 April 2017 Language Academy UTMJB
Using Cognitive Science To Inform Instructional Design
OSEP Leadership Conference July 28, 2015 Margaret Heritage, WestEd
Historical Thinking Skills
Historical Thinking Skills
Presentation transcript:

(1) If Language is a Complex Adaptive System, What is Language Assessment? Presented at “Language as a Complex Adaptive System”, an invited conference celebrating the 60 th Anniversary of Language Learning, at the University of Michigan, Ann Arbor, MI, November 7-9, The first author was supported by a grant from the Spencer Foundation. Robert J. Mislevy University of Maryland Chengbin Yin Center for Applied Linguistics

(2) Key Ideas Assessment as evidentiary argument, not simply as measurement. Arguments constructed around … »View of the nature of proficiency. »Situations and ways people acquire it and use it. Relevant work taking place in language testing from an interactionist perspective. Reconceiving measurement models

(3) The Assessment Argument (Messick, 1994) What complex of knowledge, skills, or other attributes should be assessed? What behaviors or performances should reveal those constructs? What tasks or situations should elicit those behaviors? We’ll look at a more technical representation in a little while.

(4) Snow & Lohman, 1989 Summary test scores, and factors based on them, have often been though of as “signs” indicating the presence of underlying, latent traits. [  ] … An alternative interpretation of test scores as samples of cognitive processes and contents … is equally justifiable and could be theoretically more useful.

(5) LaCAS and Assessment Arguments Interactionalist perspective in language testing: »Communicative competence »Contextual features of tasks »Language tests for specific purposes

(6) An Interactionalist Perspective (Young, 2000, 2008) … language used in specific discursive practices rather than … language ability independent of context. Focus on the co-construction of discursive practices by all participants... A set of general interactional resources that participants draw upon in specific ways in order to co-construct a discursive practice.

(7) An Interactionalist Perspective (Young, 2000, 2008) Relationship between participants’ employment of interactional resources and the context in which they are employed. Varying with the practice and the participants…

(8) Challenges for Assessment (Chalhoub-Deville, 2003) Amending the construct of individual ability to accommodate [how] language use in a communicative event reflects dynamic discourse, which is co-constructed among participants; and … reconciling [the notion that language ability is local] with the need for assessments to yield scores to generalize across contextual boundaries.

(9) Sociocognitive Foundations Themes from, e.g., cognitive psychology, literacy, neuroscience, anthropology: »Connectionist metaphor, associative memory Situated cognition & information processing »Construction-Integration (CI) theory of comprehension (Kintsch and others) Individual  Sociocultural perspectives »A cognitive theory of cultural meaning (Strauss & Quinn, 1997)

(10) A Cognitive Theory of Cultural Meaning  “Interactional Resources” Extrapersonal: »Cultural models: What ‘being sick’ means, restaurant script, Newton’s laws, complaints »Linguistic: Grammar, conventions, constructions Intrapersonal: »Patterns from experience at many levels »Schemas / frames / understandings / assumptions Interplay  Situated understandings »Access to, and ways of interacting with, shared structures in order to accomplish goals The user’s knowledge of the language rules is interlocked with his knowledge of when, where, and with whom to use them. (R. Ellis, 1985)

(11) B Inside B Inside AA observablenot observable

(12) B Inside B Inside A Context A A la Kintsch: Propositional content of text / speech… and internal and external aspects of context …

(13) B Inside B Inside A Context A The C in CI theory, Construction: Activation of both relevant and irrelevant bits of cultural models, experiences, e.g., Restaurant script, Human motivation Guided in part by linguistic models, e.g. Conventions, constructions, rhetorical frames Content of utterance History with interlocutor Conversation thus far The C in CI theory, Construction: Activation of both relevant and irrelevant bits of cultural models, experiences, e.g., Restaurant script, Human motivation Guided in part by linguistic models, e.g. Conventions, constructions, rhetorical frames Content of utterance History with interlocutor Conversation thus far E.g., tasks in Occupational English Test (OET; McNamara, 1996) call upon patterns re language, but also genre, medical knowledge, use of information in clinical settings. If a pattern hasn’t been developed in past experience, it can’t be activated (although it may get constructed in the interaction). A relevant pattern from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)). If a pattern hasn’t been developed in past experience, it can’t be activated (although it may get constructed in the interaction). A relevant pattern from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)).

(14) B Inside B Inside A Context A The I in CI theory, Integration: Situation model: synthesis of coherent / reinforced activated cultural / linguistic / situational patterns Situation model is basis of understanding The I in CI theory, Integration: Situation model: synthesis of coherent / reinforced activated cultural / linguistic / situational patterns Situation model is basis of understanding

(15) B Inside B Inside A Context A Situation model is also the basis of planning and action.

(16) B Inside B Inside A Context A Previous situation models are input to subsequent situation models.

(17) B Inside B Inside A Context A Ideally, activation of relevant and compatible cultural & linguistic models…

(18) B Inside B Inside A Context A to lead to (sufficiently) shared understanding; i.e., co-constructed meaning. Kramsch’s "shared internal context "

(19) Comments about context… A B Inside B Inside A Context Can distinguish external and internal aspects of context (e.g., Douglas, 2000) External / pubic aspects of context, e.g., Setting Physical attributes External / pubic aspects of context, e.g., Setting Physical attributes Re assessment, Target language use (TLU) features Task features Re assessment, Target language use (TLU) features Task features

(20) A B Inside B Inside A Context Aspects of cultural/linguistic/interaction context as interpreted by an external observer. Used to determine what actions signal recognition, comprehension, action through targeted cultural /linguistic models. As such, in assessment, plays role in Evaluation, hence Observable variables As such, in assessment, plays role in Evaluation, hence Observable variables

(21) What can we say about individuals? Use of resources in appropriate contexts in appropriate ways; i.e., Attunement to cultural/linguistic patterns: Recognize markers of externally-viewed patterns? Construct internal meanings in their light? Act in ways appropriate to targeted cultural/linguistic models? What is the range and circumstances of activation? (variation of performance across contexts)

(22) The Assessment Argument (Messick, 1994) What complex of knowledge, skills, or other attributes should be assessed? What behaviors or performances should reveal those constructs? What tasks or situations should elicit those behaviors?

(23) Toulmin’s Argument Claim Backing unless since Warrant Alternative explanation so Data Structure

(24) Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation

(25) Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation In interactive task, performance flows in time, performance changes situation, may or may not be series of task and observable variables In interactive task, performance flows in time, performance changes situation, may or may not be series of task and observable variables

(26) Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Concerns features of (possibly evolving) context as seen from the view of the assessor – in particular, those seen as relevant to targets of inference.

(27) Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Evaluation of performance concerns context features indirectly: clues that suggest attunement to features of cultural / linguistic models of interest. (did examinee recognize, comprehend, act accordingly?)

(28) Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation “Hidden” aspects of context—not in test theory model but essential to argument: What attunements to cultural / linguistic models can be presumed among examinees, to condition inference re targeted l/c models? “Hidden” aspects of context—not in test theory model but essential to argument: What attunements to cultural / linguistic models can be presumed among examinees, to condition inference re targeted l/c models? Fundamental to situated meaning of student variables in measurement models; Both critical and implicit. Fundamental to situated meaning of student variables in measurement models; Both critical and implicit.

(29) Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Assessment context always has its own features that activate some cultural / linguistic models and suppress others in different ways for different examinees. (i.e., “method effects”) Assessment context always has its own features that activate some cultural / linguistic models and suppress others in different ways for different examinees. (i.e., “method effects”) Important for … Alternative explanations Variable performance Important for … Alternative explanations Variable performance

(30) Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Design Argument

(31) Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation (Bachman)

(32) Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation

(33) Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation

(34) Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation

(35) Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation Shared backing for test and use arguments grounds warrant for presumed appropriate activation in TLU context. What features do tasks and TLUs share? Implicit in trait arguments Explicit in interactionalist arguments What features do tasks and TLUs share? Implicit in trait arguments Explicit in interactionalist arguments This is the essence of warrant for claim in use argument.

(36) Claim about student in use situation Other information concerning student vis a vis use situation Warrant concerning use situation since on account of Alternative explanations unless Design Argument Use Argument Data concerning use situation Student acting in assessment situation on account of Backing concerning assessment situation Alternative explanations unless Warrant concerning assessment since Warrant concerning evaluation since Warrant concerning task design since Other information concerning student vis a vis assessment situation so Claim about student Data concerning student performance Data concerning task situation Backing concerning use situation What features do tasks and TLUs not have in common? Questions of validity / generalizability: TLU features that call for other cultural / linguistic models that weren’t in task and may or may not be in examinee’s resources. Target models not activated in LTM in TLU context. Questions of validity / generalizability: TLU features that call for other cultural / linguistic models that weren’t in task and may or may not be in examinee’s resources. Target models not activated in LTM in TLU context. Knowing about target examinees and TLUs is key to strong inferences (Douglas, 1998)

(37) Implications for measurement models Basic form: Probability of aspects of performance X ij given parameters for person i and situation j (all could be vector-valued) Way too simple No explicit connection with CI comprehension model, interaction processes, etc. Apparent separation of person and situation characteristics These are indeed properties of the conventional meaning of the measurement model and parameters.

(38) An Interactionalist Perspective: Instantiation in a Context Xs result from particular persons calling upon resources in particular contexts (or not, or how) Mechanically  s simply accumulate info across situations Our choosing situations and what to observe drives the situated meaning of  s. Situated meaning of  s are tendencies toward these actions in these situations that call for certain interactional resources, via l/c models.

(39) How to model inconsistent performance? Traditional: Model as “noise” / unreliability Promising direction: Model individual’s degree or pattern in variation in terms of context features If “motivated”: Model in terms of  s »Divide & Conquer: Multiple unidimensional tests (OET) »Exploratory multidimensional models »Controlled : Structured multidimensional models »Critical importance of what else you know Structured multidimensional

(40) How much can testing gain from modern cognitive psychology? So long as testing is viewed as something that takes place in a few hours, out of the context of instruction, and for the purpose of predicting a vaguely stated criterion, then the gains to be made are minimal. Buzz Hunt (1986) Conclusion

(41) I have argued that we need to capitalize on [method effects] by designing tests for specific populations -- tests that contain instructions, content, genre, and language directed toward that population. The goal is to produce tests … that would provide information interpretable as evidence of communicative competence in context. Douglas (1998) Conclusion

(42) Conclusion Interactionalist view of test theory… for arguments in interactionalist view of language to assemble, analyze, & interpret assessments in light of context and purpose. Methods and exemplars needed, but pressing need is interpretive frame … »To connect view of language proficiency with the machinery of test theory, »Toward modeling students’ (inter)actions in purposeful variations in task contexts.

(43) Thank you!