1. How do we process text with spatial information? Marijn E. Struiksma*, Matthijs L. Noordzij**, Bas F.W. Neggers*** & Albert Postma* *Universiteit Utrecht **Radboud Universiteit Nijmegen *** Universitair Medisch Centrum Utrecht

2. Spatial language: spatial configuration of the world Simple sentences Search directions, identifying people Complex route finding instructions Generate mental map Also blind people Spatial Language

3. Aim: gain insight in underlying process and nature of mental representations Research with sighted and blind Different input modalities Spatial Language

4. Propositional model Verbal strategy Strategic model Verbal strategy or Visuo-spatial strategy Spatial Language Processing

5. Noordzij et al. (2005) Sentence-sentence Sentence-picture 80% expectancy Different Strategies

6. + triangle left of circle + triangle left of circle 80%20% Spatial condition

7. Different Strategies + triangle and circle + triangle and circle 20%80% Non-spatial condition

8. Different Strategies Spatial: slower on unexpected pictures Propositional model: unexpected stimuli – general switch cost Identical for spatial and non-spatial Strategic model: unexpected stimuli – general switch cost and incompatibility for both sentence and picture Fig. 1 from Noordzij et al. (2005)

9. Automatic propositional representation Additional visual-spatial representation Strategically dependent on context Neuroimaging parietal areas: Understanding spatial terms Visual-spatial representation Dual-representational model

10. Noordzij et al. 2006 Similar sentence-sentence and sentence-picture paradigm Similar behavioral pattern Event-related potentials (ERPs) EEG: processing spatial sentences

11. ERPs EEG: processing spatial sentences Fig. 4 from Noordzij et al. (2006) Parieto-occipital activation for spatial sentences, expecting a picture. visual-spatial strategy

12. Sentence-sentence and sentence- picture paradigm 100% expectancy for S2 RT faster for picture than sentence, especially for spatial fMRI: processing spatial sentences

13. fMRI fMRI: processing spatial sentences Fig. 2 from Noordzij et al. (2008) Spatial > Non-spatial for both stimulus modalities: activity in left Supramarginal gyrus (SMG)

14. How is spatial language processed in the absence of vision? Is language processing different for different input modalities? Is the nature of mental representations modality specific? Nature of mental representations

15. Sentence-sentence paradigm Comparing spatial and non-spatial Auditory version Scanner-details: 3T Philips Achieva scanner PRESTO-SENSE sequence TR = 500ms Voxel-size: 4*4*4 mm fMRI blind & sighted

16. 13 congenitally blind Age 36.5 ± 9.8 7 male, 6 female 5 right-handed, 5 left-handed, 3 ambidexter 13 sighted controls Age 37.2 ± 11.2 8 male, 5 female 6 right-handed, 4 left-handed, 3 ambidexter Subjects

17. Compare two sentences Do they describe the same situation? Spatial Sentence Comprehension

18. Block-design: 4 sessions, 18 blocks per session, 2 trials of 7.5s per block 4 conditions SpatialNonspatial Compound preposition left of right of (LR) together with (TW) Compound adverb smaller than taller than (ST) older than younger than (YO) Spatial Sentence Comprehension

19. Position = Left of/Right of Size = Taller than/Smaller than Conjunction = Together with Age = Older than/Younger than Position (15s) Interval (6-9s) Size (15s) Age (15s) Conjunction (15s) Interval (6-9s) Presented pseudo-randomly Spatial Sentence Comprehension

20. Performance Performance is good No difference CB or SC

21. Behavioral Results: RT Main effects of Space and Category No group difference

22. Contrast: Position > Combination ROI around SMG from Noordzij et al. (2008) Conjunction CB and SC Results: Supramarginal Gyrus ROI, T=2.5, p =.045 (corrected)

23. Contrast: Relational > Combination ROI around SMG from Noordzij et al. (2008) Conjunction CB and SC Results: Supramarginal Gyrus ROI, T=2.5, p =.045 (corrected)

24. Contrast Estimates SMG

25. fMRI results Evidence for reorganization Difference CB and SC for task vs. rest Occipital areas No dimension modulation Whole brain, T=4.0,k=10 voxels, p < 0.03

26. Behavioral results: sighted can generate propositional and visual- spatial representations Target area: left SMG Left SMG not influenced by context (picture or sentence) input modality (visual or auditory) Conclusions 1

27. Blind also activate left SMG and visual areas Visual areas are not modulated by Space or Category Left SMG important for processing spatial prepositions Other function: ordering on 1 dimension Activation in left SMG is modality- independent + hardwired Conclusions 2

28. Questions ?

29. Block-design 3 types of blocks Rest Non-words Words Verb Generation

30. Results Conjunction CB+SC Language areas bilateral Broca, Wernicke Covert verb generation: precentral gyrus supplementary motor area Cerebellum Word - Nonword Whole brain, T=4.0, k=20 voxels, p < 0.007

31. Results: Reorganization Difference CB-SC CB significantly activate occipital cortex BA 19, 31, 37 CB vs. SC Whole brain, T=4.5, k=10 voxels, p < 0.019