Brain signatures of game play and language processing Robert V. Reichle rreichle@niu.edu Department of Foreign Languages Center for the Interdisciplinary Study of Language and Literacy Northern Illinois University

L2 PROCESSING For late L2 learners, L2 processing is difficult Corresponding features or constructions in the native language do not ensure success in the L2 Variability is typical Native-like attainment is not typical Previously thought to relate to putative critical period for L2 acquisition

L2 PROCESSING Among other explanations for divergence in L1/L2 processing, Ullman’s (2001, 2004) Declarative/Procedural model Two domain-general memory systems Declarative: facts, associations between form and meaning, non-compositional words Procedural: rule-governed processes, morphosyntactic composition, cognitive-motor skills Some L2 learners become more nativelike as they come to rely more on procedural memory in the L2

EVENT-RELATED POTENTIALS (ERPs) Lexical/semantic processing: N400 Morphosyntactic/grammatical processing: LAN Differential ERPs for L1 syntax/semantics interpreted as evidence for DP model (Ullman, 2004) (e.g., graphs from Osterhout et al., 2006, p. 204 and Neville et al., 1991; Friederici, Steinhauer, & Frisch, 1999; Friederici, 2002; Hahne & Friederici, 1999, 2001; Hagoort & Brown, 1999; Kaan et al., 2000; Kutas & Hillyard, 1980; Osterhout, Bersick, & McLaughlin, 1997; Osterhout & Holcomb, 1992; Osterhout & Mobley, 1995; Steinhauer, Halter, & Friederici, 1999; Friederici, Pfeifer, & Hahne, 1993; Kluender & Kutas, 1993; Münte, Heinze, Matzke, Wieringa, & Johannes, 1998)

EVENT-RELATED POTENTIALS (ERPs) Proficiency effects on sensitivity to inflectional morphology and syntax Artificial language learning: with language training, N400 LAN/P600 (e.g., Friederici, Steinhauer, & Pfeifer, 2002; Morgan-Short, Sanz, Steinhauer, & Ullman, 2010; Morgan- Short, Steinhauer, Sanz, & Ullman, 2012; Morgan-Short, Finger, Grey, & Ullman, 2012) Real language learning: with increasing proficiency, N400 LAN/P600 (e.g., Bowden et al., 2007; Hahne et al., 2006; Osterhout et al., 2006; Rossi et al., 2006; Steinhauer et al., 2006; Gillon Dowens, Vergara, Barber, & Carreiras, 2010; McLaughlin, Osterhout, & Kim, 2004; McLaughlin, Tanner, Pitkänen, Frenck-Mestre, Inoue, Valentine, & Osterhout, 2010; Tanner, Osterhout, & Herschensohn, 2009; Ojima, Nakata, & Kakigi, 2005)

EVENT-RELATED POTENTIALS (ERPs) Proficiency effects on sensitivity to inflectional morphology and syntax Artificial language learning: with language training, N400 LAN/P600 (e.g., Friederici, Steinhauer, & Pfeifer, 2002; Morgan-Short, Sanz, Steinhauer, & Ullman, 2010; Morgan- Short, Steinhauer, Sanz, & Ullman, 2012; Morgan-Short, Finger, Grey, & Ullman, 2012) Real language learning: with increasing proficiency, N400 LAN/P600 (e.g., Bowden et al., 2007; Hahne et al., 2006; Osterhout et al., 2006; Rossi et al., 2006; Steinhauer et al., 2006; Gillon Dowens, Vergara, Barber, & Carreiras, 2010; McLaughlin, Osterhout, & Kim, 2004; McLaughlin, Tanner, Pitkänen, Frenck-Mestre, Inoue, Valentine, & Osterhout, 2010; Tanner, Osterhout, & Herschensohn, 2009; Ojima, Nakata, & Kakigi, 2005)

brocanto training game Artificial language (BROCANTO) Fulfills syntactic requirements of natural language Contains morphosyntactic features that differ from L1 of participants Participants trained in lexicon of artificial language One group trained to proficiency in BROCANTO via a two-player computer board game Played against each other in pairs Orally expressed all game moves in BROCANTO Received feedback on valid game moves and grammatical BROCANTO utterances The structure of the language not dependent on the structure of the game (Friederici, Steinhauer, & Pfeifer, 2002; for similar BROCANTO2 design, see also Morgan-Short, Sanz, Steinhauer, & Ullman, 2010; Morgan-Short, Steinhauer, Sanz, & Ullman, 2012; Morgan-Short, Finger, Grey, & Ullman, 2012)

brocanto training game aaf trul prez nöri aak füne plox the trul-piece captures horizontally the round plox-piece aaf trul prez nöri *rix füne ploxthe trul-piece captures horizontally *buy round plox-piece ERPs ERP data collection using syntactically felicitous and anomalous items took place after participants in the training group reached a 95% accuracy threshold. Training group had biphasic response to violations that resembled LAN/P600 of L1 processing

brocanto training game Why was this a successful training paradigm?

brocanto training game Why was this a successful training paradigm? Gee (2003): active, critical learning cooperation competition a requirement of successful learning in order to progress feedback meta-awareness of gameplay mechanics

brocanto training game Why was this a successful training paradigm? Declarative/Procedural model nativelike syntactic processing should be handled by procedural memory learning a cognitive-motor skill (like the rules of the training game) should also be handled by this system Did these two types of learning engage similar cognitive processes? Does “words and rules” have an equivalent “sprites and rules” (Reichle, 2012)? P600 not necessarily fully language-specific — also observed for structural revision in music (e.g. Patel et al., 1998)

Research questions Does the execution of gameplay skills engage the same processes as syntactic processing? Specifically, examining violations of syntax and violations of expected gameplay behavior. If the same processes, then could aid learning, but could also present confound to interpretation of results. Does processing linguistic stimuli in the context of a game lead to a task effect compared to linguistic stimuli in isolation?

Hypotheses Violations of expected gameplay behavior will elicit LAN and P600 effects comparable to those seen for violations of morphosyntax. Processing violations of morphosyntax in the context of a game causes a task effect compared to violations of morphosyntax in isolation.

ERP Experiment 1 23 participants (5 rejected due to artifacts) native speakers of English pre-screened for knowledge of rules of movement in chess EEG data were recorded from nine sites (F3, Fz, F4, C3, Cz, C4, P3, Pz, P4)Four conditions A: Control. Legal moves followed by grammatical sentences B: Morphosyntax violation. Legal moves followed by sentences with a morphosyntax violation C: Gameplay mismatch. A chess move that is legal for a piece other than the one shown in the image, followed by a grammatical sentence D: Gameplay violation. A chess move that is never legal for any piece, followed by a grammatical sentence

A1. Control: The queen takes the pawn.

A: The white queen takes the black pawn. A1. Control: The queen takes the pawn. A: The white queen takes the black pawn.

B: The white queen *taking the black pawn. A1. Control: The queen takes the pawn. B: The white queen *taking the black pawn.

F1. Game Rule Violation: The queen takes the pawn.

C: The white queen takes the black pawn. F1. Game Rule Violation: The queen takes the pawn. C: The white queen takes the black pawn.

F1. Game Rule Violation: The queen takes the pawn.

D: The white queen takes the black pawn. F1. Game Rule Violation: The queen takes the pawn. D: The white queen takes the black pawn.

ERP Experiment 1 Stimulus presentation each image onscreen for 1 sec 600ms pause word-by-word presentation: 290 ms plus 30 ms per letter, up to a maximum of 590 ms, with a 150 ms interval between words Analysis At fourth word (takes/*taking) Mean amplitude data in 200-500ms and 500-800ms windows Repeated-measures ANOVAs: 4 (Condition) x 3 (Anteriority) x 3 (Laterality)

F3 Fz F4 C3 Cz C4 P3 Pz P4

F3 Fz F4 Omnibus 200-500ms: Cond * Lat, p < .05 C3 Cz C4 P3 Pz P4

F3 Fz F4 C3 Cz C4 P3 Pz P4 Morph. vs Control 500-800ms: Cond * Lat, p < .05 C3 Cz C4 P3 Pz P4

F3 Fz F4 C3 Cz C4 P3 Pz P4 Game rule mismatch vs Control 200-500ms: NS

F3 Fz F4 C3 Cz C4 P3 Pz P4 Game rule violation vs Control 500-800ms: Cond * Ant * Hem, p = .022 C3 Cz C4 P3 Pz P4

Results P600 effect for morphosyntax violations No effect for game rule mismatch Game rule violation Negativity effect, strongest in left and mid central and frontal sites Interpreted as possible variation of LAN Extended into later time window than typical biphasic LAN But, not biphasic — no P600 effect followed

Discussion Hypothesis: Violations of expected gameplay behavior will elicit LAN and P600 effects comparable to those seen for violations of morphosyntax. Not fully supported — no P600. Consistent with generally language-specific nature of P600. Partially supported by presence of LAN. But work still remains on reproducibility and functional significance of LANs (e.g. Steinhauer, Drury).

Discussion Implications Possibility remains that processing gameplay rules engages similar cognitive processes as processing morphosyntax (LAN) No evidence that gameplay elicits P600 Doubtful that P600s seen in BROCANTO studies were indices of gameplay-related processing

Hypotheses Violations of expected gameplay behavior will elicit LAN and P600 effects comparable to those seen for violations of morphosyntax. Processing violations of morphosyntax in the context of a game causes a task effect compared to violations of morphosyntax in isolation.

ERP Experiment 2 Show morphosyntactic anomalies within and without game context 2 groups Image group: Saw chess move after reading sentence NoImage group: Did not see chess moves 12 participants (6 per group) native speakers of English pre-screened for knowledge of rules of movement in chess EEG data were recorded from nine sites (F3, Fz, F4, C3, Cz, C4, P3, Pz, P4)

ERP Experiment 2 2 Conditions Control: The white queen takes the black pawn. Morphosyntax violation: The white queen taking the black pawn. 2 fillers Progressive filler: The queen is taking the black pawn. Game rule violation: The black queen takes the white pawn.

ERP Experiment 2 Stimulus presentation word-by-word presentation: 290 ms plus 30 ms per letter, up to a maximum of 590 ms, with a 150 ms interval between words for Image group, presentation of game images 1 sec after final word Analysis At fourth word (takes/taking) Mean amplitude data in 200-500ms and 500-800ms windows Repeated-measures ANOVAs: 2 (Condition) x 3 (Anteriority) x 3 (Laterality)

NoImage group Control Morphosyntax violation

Image group Control Morphosyntax violation

Between-group omnibus: 400-900ms: Cond * Group, p = .26 Image group Control Morphosyntax violation

Image group: 400-900ms: Cond, p = .25 400-900ms: Cond * Lat, p = .19 400-900ms: Cond * Lat * SRTT, p = .2 Image group Control Morphosyntax violation

Results Trend toward between-group difference (task effect) Trend toward P600 for Image group Data collection still in progress — more participants needed

Discussion Hypothesis: Processing violations of morphosyntax in the context of a game causes a task effect compared to violations of morphosyntax in isolation. Preliminary results trend toward support of hypothesis. Attributable to “motivation” in the broadest sense. Additional studies needed to separate out Gee’s factors (competition, meta-awareness, etc.) from simpler explanations (i.e. more attentive due to presence of images)

CONCLUSIONS Evidence of an LAN for the processing of gameplay violations suggests gameplay may engage cognitive processes also used in language processing. Lack of P600 for gameplay processing indicates that L2 studies using game training paradigms are not likely to confound indices of gameplay with indices of morphosyntactic processing. Preliminary data suggest trend toward game-related task effect, suggesting increased motivation or attention within game context. Additional studies needed to further tease apart variables.

Acknowledgments This research has been supported by the NIU Division of Research and Graduate Studies, the NIU Center for the Interdisciplinary Study of Language and Literacy, and the NIU Center for the Study of Family Violence and Sexual Assault. The author thanks Cody Happ and Regina Hiraoka for their assistance with materials creation and data collection, and all participants for their cooperation.

Thank you