1. Readers routinely represent implied object rotation: The role of visual experience Wassenberg & Zwaan, in press, QJEP Brennan Payne Psych 525 10.27.10

2. Theories of discourse comprehension Construction-Integration (Kintsch & van Dijk, 1978; Kintsch, 1998) Structure-Building (Gernsbacher, 1990, 1997) Event-Indexing Model (Zwaan et al., 1995) Resonance Model (O’Brien et al., 1995, 1998) Shared assumption that discourse comprehension can be modeled as the integration of abstract and amodal representations.

3. Theories of discourse comprehension Construction-Integration (Kintsch & van Dijk, 1978; Kintsch, 1998) Structure-Building (Gernsbacher, 1990, 1997) Event-Indexing Model (Zwaan et al., 1995) Resonance Model (O’Brien et al., 1993, 1995, 1998) Shared assumption that discourse comprehension can be modeled as the integration of abstract and amodal representations.

4. Construction-Integration Computational model Different levels of representation: Surface form of the text Text base: propositional information from the text Situation model: representation of situation implied with the text; derived from propositional text base Proposition: “idea unit”; smallest unit of knowledge. Follows predicate argument form: PREDICATE(ARGUMENT 1, ARGUMENT 2 ) 2a. The carpenter pounded the nail into the wall. 2b. The carpenter pounded the nail into the floor. [POUNDED(CARPENTER, NAIL)], [INTO (NAIL, WALL)] [POUNDED(CARPENTER, NAIL)], [INTO(NAIL, FLOOR)] 1a. The ranger saw the eagle in the sky. [SAW(RANGER, EAGLE)], [IN (EAGLE, SKY)] 1b. The ranger saw the eagle in its nest. [SAW(RANGER, EAGLE)], [IN (EAGLE, NEST)]

5. Alternative Account 1. The carpenter pounded the nail into the wall. 2. The carpenter pounded the nail into the floor. [POUNDED (CARPENTER, NAIL)], [IN (NAIL, WALL)] [POUNDED (CARPENTER, NAIL)], [IN (NAIL, FLOOR)] Proposition Account: Highly Identical, only difference is N specifying orientation Perceptual Symbol Account (Barsalou, 1999a,b; Stanfield & Zwaan, 2001; Zwaan et al., 2002): Sentences are very different in perceptual representation that is implied.

6. Do readers represent perceptual information? John put the pencil in the cup. Sentence-picture verification task Was this mentioned in the previous sentence? Significant differences in RT latencies when objects matched vs. mismatched. 838 (331) 882 (329) * Stanfield & Zwaan (2001); Psych. Sci.

7. Perceptual Traces (Zwaan & Kaschak, 2008) Orientation (Stanfield & Zwaan, 2001) Shape (Zwaan et al., 2002; Dijkstra et al., 2004) Size (Taylor & Zwaan, 2010) Movement and Motion (Kaschak et al., 2005; 2006) Color (Richter et al., 2009; Therriault et al., 2009) Previous Research: Nature of language representation Sentence  Picture Can this perceptual information affect online language processing? Picture  Sentence Do readers represent perceptual information?

8. Current Study Does a recent visual exposure to an object in a specific orientation affect later language comprehension? Wassenburg & Zwaan (in press); QJEP. A 3-phase “visual memory” paradigm (Zwann et al., 2010) 1.Word-picture verification task -Experimental items shown in vertical or horizontal orientation 2.15- minute filler task 3.Eye-tracking session Three phases are presented as unrelated experiments to deter some kind of strategy use. Predict a match/ mismatch effect: fixation times on the prepositional phrase (into the wall/ in the cup) that implies the object orientation should be sensitive to the orientation of the previously seen image.

9. Method Participants: N = 34; N= 28 after track loss/errors. 50%Female Age= 20.3 (18-24). Native Dutch Speakers Materials P1 80 word-picture items 60 fillers; 20 critical items Each critical item formed a match with its word Orientation (H;V) counterbalanced across participants over 2 lists P2 ??? Maybe the flag test P3 Tobii 2150 eye tracker 40 Dutch Sentences 20 filler; 20 critical sentences using critical words from P1 Half of each orientation matched and half mismatched

10. Method Aunt Karen finally found the toothbrush in the sink of the bathroom. Phase 3: Toothbrush Phase 1: Phase 2: VSP rotation filler task Approx. 15 min. 1234*5 Procedure

11. Results * † †p =.06 *p <.05

12. Much in the way that language affects later visual processing (Stanfield & Zwaan, 2001), visual memory also influences language processing. Prior exposure to a picture of an object in a particular orientation affects later reading times for phrases that imply the orientation of that object. Match/mismatch effects occur on first pass measures on the disambiguating PP and diminish quickly, suggesting that these effects are both early and immediate. Reading comprehension may be multimodal, not only using linguistic representations, but sensory/perceptual representations as well. Conclusion