Anticipatory eye-movements in a visual world: Effects of Context Heather Ferguson Tony Sanford & Christoph Scheepers GLASGOW LANGUAGE PROCESSING

Counterfactuals  Counterfactual thinking: - Cases of possibly valid reasoning from premises that are false in actuality (Fauconnier & Turner, 2003) “If I had woken up earlier this morning then I would have eaten breakfast”

Counterfactuals in cognitive science  Fauconnier (1994; 2002) - Forced incompatibility between two mental spaces: reality & hypothetical  Leslie (1987) - Counterfactual thinking is a special case of theory of mind  Kahneman & Miller (1986) - Modifying past experiences manipulates expected norms

Anomaly detection in reading  Semantic anomalies & eye-movements (Ni et al, 1998; Braze et al, 2002): –generate longer reading times prior to a gradual increase in regressive eye-movements  Semantic anomalies & ERP (Kutas & Hillyard, 1980): –N400 effect to pragmatic violations  Discourse-dependent context anomalies & ERP (Van Berkum, Hagoort & Brown, 1999): –elicit a large N400 effect

Counterfactuals: Experimental items [1] RW- inconsistent If cats are hungry they usually pester their owners until they get fed. Families could feed their cat a bowl of carrots and it would gobble it down happily. [2] RW- consistent If cats are hungry they usually pester their owners until they get fed. Families could feed their cat a bowl of fish and it would gobble it down happily. [3] CW- inconsistent If cats were vegetarians they would be cheaper for owners to look after. Families could feed their cat a bowl of fish and it would gobble it down happily. [4] CW- consistent If cats were vegetarians they would be cheaper for owners to look after. Families could feed their cat a bowl of carrots and it would gobble it down happily.

Eye-movements: Summary  RW violations can be ‘neutralised’ within an appropriate pre-specified CW context  RW-congruent items can lead to the experience of an anomaly following an inconsistent CW context - prior context is rapidly utilised

 Disruption to early processing for RW violations regardless of prior context  RW inconsistencies are detected faster than CW inconsistencies –effects for a RW inconsistency are detected in the first-pass RT Eye-movements: Summary

ERPs: Summary  ‘world’ inconsistencies have been detected - BUT, gives limited information on interference from RW knowledge N400 effect

 Strong frontal theta following ‘world’ inconsistencies  Temporal gamma for RW violations Time Frequency: Summary RW-consistentRW-inconsistent CW-consistentCW-inconsistent

 Counterfactual reasoning could account, at least in part, for theory of mind understanding  Significant correlations have been found between false belief and counterfactual reasoning tasks  BUT, ability in counterfactual reasoning is a necessary but not sufficient component of successful performance in the false belief tasks (Peterson & Bowler, 2000) Counterfactuals & Theory of Mind

 The ability to understand and predict other peoples’ behaviour by attributing independent mental states to them  Neurology of Theory of mind  Limitations to methods of research to date –See Saxe (2006), ‘Why and how to study Theory of Mind with fMRI’ –No on-line studies of ToM processing Theory of Mind

 Visual-world research: –Eye-movements around a visual scene are closely time-locked with related auditory input Visual World Paradigm - Altmann & Kamide (1999): “The boy will move the cake.” “The boy will eat the cake.”

Visual World Paradigm  Participant looked faster to the cake in the “eat” condition than in the “move” condition.  Participants anticipate forthcoming linguistic reference to objects in the scene - i.e. they looked at the cake before the could hear it  Altmann & Kamide concluded that visual attention is drawn to critical objects in the scene before they are actually mentioned in the auditory input

Can people use their knowledge of the wider discourse to over-ride real-world knowledge to predict specific upcoming words as the current sentence unfolds?  24 experimental items  28 native English speakers (16 female)  Eyelink II head-mounted eye-tracker  Within-subjects 2 (CW/ RW) x 2 (consistent/ inconsistent) design Visual World Paradigm

Counterfactuals: Experimental items [1] RW- inconsistent If cats are hungry they usually pester their owners until they get fed. Families could feed their cat a bowl of carrots and it would gobble it down happily. [2] RW- consistent If cats are hungry they usually pester their owners until they get fed. Families could feed their cat a bowl of fish and it would gobble it down happily. [3] CW- inconsistent If cats were vegetarians they would be cheaper for owners to look after. Families could feed their cat a bowl of fish and it would gobble it down happily. [4] CW- consistent If cats were vegetarians they would be cheaper for owners to look after. Families could feed their cat a bowl of carrots and it would gobble it down happily.

Counterfactuals: Example Sentence 1 + RW context CW context

Sentence 2 Counterfactuals: Example RW- anomalous RW- congruent

 Temporal location of fixations was recorded at 20ms intervals for each condition  k-Means cluster analysis: - groups similar eye-movement patterns in the region preceding and immediately following the verbal onset of the target word for statistical analysis Methods of analysis

Counterfactuals: Results  (-200ms) – 400ms: Main effect of context  460ms – 2000ms: Main effect of consistency  No (context x consistency) interaction at any point

 Participants anticipated towards a contextually relevant referent from 200ms prior to critical word onset  Shortly after the onset of the target word, however, the relevant ‘appropriate’ referent was visually favoured  The results provide the first evidence that real- world biases in the visual world paradigm can be ‘neutralised’ by a counterfactual context  No effects of gender were detected Counterfactuals: Summary

Can people use their knowledge of the wider discourse to over-ride real-world knowledge to predict specific upcoming words as the current sentence unfolds?  24 experimental items  28 native English speakers (16 female)  Eyelink II head-mounted eye-tracker  Within-subjects 2 (Reality/ Belief) x 2 (inconsistent/ consistent) design Visual World Paradigm

Janet unpacked the belongings and put the postcard in the cupboard. [1] Reality-inconsistent Later, Janet saw Barry move the postcard from the cupboard to the drawer. Later, Janet wanted to see the postcard so she looked in the cupboard and smiled. [2] Reality-consistent Later, Janet saw Barry move the postcard from the cupboard to the drawer. Later, Janet wanted to see the postcard so she looked in the drawer and smiled. [3] Belief-inconsistent While Janet was busy, Barry moved the postcard from the cupboard to the drawer. Later, Janet wanted to see the postcard so she looked in the drawer and smiled. [3] Belief-consistent While Janet was busy, Barry moved the postcard from the cupboard to the drawer. Later, Janet wanted to see the postcard so she looked in the cupboard and smiled. Beliefs of others: Experimental items

Beliefs of others: Example Sentence 1

Beliefs of others: Example Sentence 2 Reality context Belief context

Beliefs of others: Example Sentence 3 RW- anomalous RW- congruent

Beliefs of others: Results  (-560ms) – 500ms: Main effect of context  860ms – 1500ms: Main effect of context  280ms – 1500ms: Main effect of consistency  No (context x consistency) interaction at any point

Gender effect  (-1500ms) – (-560ms): (gender x context) interaction Beliefs of others: Results

Beliefs of others: Gender

 As a group, participants anticipated towards a contextually relevant referent from 560ms prior to critical word onset  Shortly after the onset of the target word, the relevant ‘appropriate’ referent was visually favoured  HOWEVER, this prediction of others’ behaviour seems to have been led by female participants  Here, females show better performance at predicting others’ actions based on their intentions, beliefs and desires than males Beliefs of others: Summary

 First on-line investigation into the comprehension of the beliefs of others  Also shows evidence to support a gender bias in cognition (see Baron-Cohen, 2002) - Females are superior than males on tests of social judgement and measures of empathy and cooperation (Baron-Cohen, 1996; Charman, Ruffman & Clements, 2002) - Males excel in practical skills, e.g. mathematical reasoning, mental rotation, spatial tasks and target- directed motor skills Beliefs of others: Summary

 EM, ERP and time frequency data support a two-stage mental model theory of discourse comprehension  In a visual world, prior context is integrated even faster and participants expect context-relevant continuations –Real-world biases in the visual world paradigm can be ‘neutralised’ by a prior context –Although this does not mean that the RW violation has been missed all together  Results support a theory of a gender bias in cognition and suggest an advantage for females on a theory of mind task  This significant gender bias was not present when predicting events in a counterfactual world, suggesting different processes underlying the two tasks Conclusions

Thank you!