Cognition – 2/e Dr. Daniel B. Willingham Chapter 9: Visual Imagery PowerPoint by Glenn E. Meyer, Trinity University ©2004 Prentice Hall

2 What Purpose Does Visual Imagery Serve? Two Functions: Two Functions: Memory- making the visual properties of objects available Memory- making the visual properties of objects available Problem solving – allowing us to try out changes in position of objects in our mind’s eye Problem solving – allowing us to try out changes in position of objects in our mind’s eye Imagery in Early Psychology Imagery in Early Psychology Imagery in Early Psychology Imagery in Early Psychology Imagery Reenters Psychology Imagery Reenters Psychology Imagery Reenters Psychology Imagery Reenters Psychology Imagery and Perception Imagery and Perception Imagery and Perception Imagery and Perception

©2004 Prentice Hall3 Imagery in Early Psychology Introspectionists (Wundt, 1894) Introspectionists (Wundt, 1894)  Small set of irreducible elements could be combined to create mental images  Imagery was tantamount to thought Imageless Thought Controversy: Imageless Thought Controversy:  Debate over whether it is possible to have thoughts that are not accompanied by images. This debate occurred in the late 19th century and was critical to the introspectionist program because they studied imagery as a window into thought processes. If some thought was not accompanied by images, it was not clear how it could be studied – Galton Behaviorism – John Watson (1913): Behaviorism – John Watson (1913):  Studying the mind was fruitless and study of imagery abandoned  Woodworth and Scholsberg (1954) –Images are interesting but hard to study objectively, so we don’t

©2004 Prentice Hall4 Imagery Reenters Psychology Major Turning Point - Allan Paivio’s (1971) book Major Turning Point - Allan Paivio’s (1971) book  Refuted claims that imagery was impossible to study  Demonstrated imagery had functional role – as seen in memory tasks like paired associate learning.  Found (1963) concrete words easier to recall than abstract words due to imagery. Similar result in Paivio and Foth (1970) o Abstract word: One that does not refer to a physical object o Concrete word: Refer to real objects in the word (e.g., pencil)  Suggest Dual Coding Hypothesis: o Paivio’s proposal that concepts can be encoded verbally, in terms of mental images, or both as seen in Fig 9.1 o Verbal stimuli codes as logogens: Term for representations that support mental images in Paivio’s dual-coding hypothesis o Nonverbal stimuli representation as imagens: Term for verbal representation in Paivio’s dual-coding hypothesis as seen in Fig. 9.2  Key point of Paivio’s work – imagery impacts memory

©2004 Prentice Hall5 Imagery and Perception Studies Linking Perception to Imagery Studies Linking Perception to Imagery  Brooks (1968) – had subjects examine codes in primary memory and found visual-visual interference but little visual-verbal interference. Argues for dual representation  Shepard and Metzler (1971) – Subjects had to match 3D images of objects to rotated versions. Found a linear relationship in reaction time to degree of rotation. Subjects report using imagery – A seminal study! o Highlights important role of imagery in dealing with transformations o Psychologists excited by orderly nature of data and suggestion of a single process can explain it  Shepard and Chipman (1970) suggested images are second-order isomorphism: They suggested that the parts of images have the same functional relationship to one another that the parts of pictures have to one another  Neural Imagery Findings: Kosslyn et al. (1997) – PET scans during perceptual or imagery tasks found some overlap in secondary visual areas

©2004 Prentice Hall6 Are Visual Images Supported by a Separate Representation System? Pylyshyn’s Criticisms of Imagery Based Theories Pylyshyn’s Criticisms of Imagery Based Theories  Propositional versus Analog Representation Propositional versus Analog Representation Propositional versus Analog Representation  The Metaphor is Misleading The Metaphor is Misleading The Metaphor is Misleading  Demand Characteristics and Tacit Knowledge Demand Characteristics and Tacit Knowledge Demand Characteristics and Tacit Knowledge The Brain and the End of the Imagery Debate The Brain and the End of the Imagery Debate The Brain and the End of the Imagery Debate The Brain and the End of the Imagery Debate

©2004 Prentice Hall7 Propositional versus Analog Representation Analog representation: One that has important properties of pictures (e.g., that it occurs in a spatial medium) but is not itself a picture. Mental images are usually referred to as analog representations Analog representation: One that has important properties of pictures (e.g., that it occurs in a spatial medium) but is not itself a picture. Mental images are usually referred to as analog representations Proposition: A verbal representation of knowledge. It is the most basic unit of meaning that has a truth value Proposition: A verbal representation of knowledge. It is the most basic unit of meaning that has a truth value Images not needed as data can be handled by propositions Images not needed as data can be handled by propositions A. Kosslyn (1980) – Five properties that differentiate propositions from images 1. Propositions are relational 2. Proposition has syntax 3. Proposition has truth value 4. Proposition is abstract 5. Image occurs in a spatial medium, proposition does not – as seen in Fig 9.5 (Kosslyn, 1973) B. Pylyshyn argues sense of using image is epiphemonological C. Pylyshyn argues having only one representation scheme (propositions) is parsimonious as in the principle of Occam’s Razor D. Response by imagery researchers: studies of Kosslyn and his colleagues (1980, 1978, 1975, 1976) more parsimoniously explained by subjects using a spatial representation in imagery

©2004 Prentice Hall8 The Metaphor is Misleading 1. Metaphor is misleading –  Looking at a mental image is similar to having having a homunculus in the system: A small person inside the head who performs cognitive functions such as looking at images on a screen and then pulling levers for behavior.  Related to Picture Theory of Imagery: The experience of visual imagery is created by activating a memory representation. This memory representation was created by viewing objects in the real world. Kosslyn (1980) provided detailed theory of imagery representation

©2004 Prentice Hall9 Demand Characteristics & Tacit Knowledge Pylyshyn argues subjects could guess what was to happen in most imagery experiments because of : Pylyshyn argues subjects could guess what was to happen in most imagery experiments because of :  Tacit knowledge: In the imagery debate, tacit knowledge is a participant’s knowledge of how objects in the real world move. It was suggested by some that participants used this tacit knowledge to simulate real-world movement and thereby produce results in imagery experiments that match real-world phenomena  Demand Characteristics: Anything about the way the experiment is conducted that signals to the participant what the desired, appropriate, or expected behavior is Experimental evidence does not support this view: Experimental evidence does not support this view:  Jolicouer and Kosslyn (1985) – subjects told to expect a ‘U’ shaped function in a imagery scanning experiment but still produced the previous found linear function  Denis and Carfantan (1985) – found most subjects could not predict outcome of an imagery experiment using mental rotation

©2004 Prentice Hall10 The Brain & the End for the Imagery Debate The Debate: If imagery is propositional then you might see activation of language areas during imagery tasks. If it is more an analog process, you would expect activation in visual areas. The Debate: If imagery is propositional then you might see activation of language areas during imagery tasks. If it is more an analog process, you would expect activation in visual areas. Bihan, et al. (1983) – fMRI founding imaging patterns lead to significant activation of primary visual cortex Bihan, et al. (1983) – fMRI founding imaging patterns lead to significant activation of primary visual cortex Similar results indicate activation of primary or secondary visual cortex by several researchers (Charlot, et al., 1992; Chen, et al., 1998; Fletcher, et al. 1995; Sabbah, et al. 1995; Mellet, et al., 1995) Similar results indicate activation of primary or secondary visual cortex by several researchers (Charlot, et al., 1992; Chen, et al., 1998; Fletcher, et al. 1995; Sabbah, et al. 1995; Mellet, et al., 1995) Conclusion: Activation observed in same areas known to support visual perception rather than language. This speaks against the propositional model Conclusion: Activation observed in same areas known to support visual perception rather than language. This speaks against the propositional model

©2004 Prentice Hall11 How Does Visual Imagery Work? Image Generation Image Generation Image Generation Image Generation Image Inspection Image Inspection Image Inspection Image Inspection Image Transformation Image Transformation Image Transformation Image Transformation

©2004 Prentice Hall12 Image Generation Images are generate one piece at a time as seen in Kosslyn, et al. (1988), Fig. 9.8 Images are generate one piece at a time as seen in Kosslyn, et al. (1988), Fig. 9.8 Visual vs. Spatial Aspects of Image Generation (as seen in the tasks in Table 9.1): Visual vs. Spatial Aspects of Image Generation (as seen in the tasks in Table 9.1):  Anatomically separate processes support the generation of image parts (visual imagery) and their configuration into correct spatial locations (spatial imagery): o Farah, et al. (1988): case of L.H., temporal lobe damage leading to agnosia (A deficit of vision caused by brain damage in which the patient can appreciate shapes but cannot identify objects based on visual cues alone). Damage was in the “what” visual stream o Levine, et al. (1985): Patient with damage in “where” stream had difficulty with localizing objects from memory o PET data support separation conjecture (Alivisatos and Petrides, 1997; Mellet, et al., 1997)

©2004 Prentice Hall13 Image Generation - Continued  Image Maintenance – o Images fade quickly and need maintenance o Probably because it shares processes with vision that are constantly needed  Imagery has capacity limitations as shown by Kerr (1987) and seen in the data in Table 9.2. Study indicate use of chunking of image parts by subjects

©2004 Prentice Hall14 Image Inspection Image inspection: Processes engaged to better know the visual characteristics of an image Image inspection: Processes engaged to better know the visual characteristics of an image Visual perception and imagery interfere with each other – Baddeley, et al. (1975). Memory for sentences with imagery components was worse with a concurrent tracking task than with sentences with no imagery component Visual perception and imagery interfere with each other – Baddeley, et al. (1975). Memory for sentences with imagery components was worse with a concurrent tracking task than with sentences with no imagery component Farah (1985) – imagery can facilitate perception if they share some visual property Farah (1985) – imagery can facilitate perception if they share some visual property Finke, Pinker and Farah (1989) - subjects can manipulate mental images of simple figures and recognize the resulting patterns as something different from what they started with Finke, Pinker and Farah (1989) - subjects can manipulate mental images of simple figures and recognize the resulting patterns as something different from what they started with Subjects able to examine parts of images in isolation, such as Kosslyn’s (1975) demonstration that subjects can zoom in on an image part. Subjects able to examine parts of images in isolation, such as Kosslyn’s (1975) demonstration that subjects can zoom in on an image part.

©2004 Prentice Hall15 Image Inspection - Continued Patients with hemispherical neglect also show the deficit in imagery tasks (Bisiach and Luzzati (1978) Patients with hemispherical neglect also show the deficit in imagery tasks (Bisiach and Luzzati (1978)  Hemispherical neglect - patients ignore half the visual world, usually the left side Eye movements when scanning an image similar to those when scanning a real stimuli. Demonstrated with real or imaged complex patterns (Brandt and Stark, as seen in Figure 9.9 Eye movements when scanning an image similar to those when scanning a real stimuli. Demonstrated with real or imaged complex patterns (Brandt and Stark, as seen in Figure 9.9

©2004 Prentice Hall16 Image Transformation Question - Do image transformations mirror the real world? Question - Do image transformations mirror the real world? Known Image Transformations (as seen in Table 9.4): Known Image Transformations (as seen in Table 9.4):  RotationShepard & Metzler (1971)  ExpansionBundesen & Larsen (1975)  Sequenced TransformationSekuler & Nash (1972)  Folding Parts to Make a WholeShepard & Feng (1972)  Transforming ColorWatkins & Schiano (1982) Rotation rate of images does not vary according to image complexity Cooper (1975) as seen in Fig Rotation rate of images does not vary according to image complexity Cooper (1975) as seen in Fig Image rotations take long if they would in the real world, Parsons (1987 a & b) Image rotations take long if they would in the real world, Parsons (1987 a & b) Rotating images have momentum (Freyd and Finke, 1984) Rotating images have momentum (Freyd and Finke, 1984) Neuroscientific data indicates that images rotate smoothly, occupying all intermediate positions (Georgopoulous, et al., 1989) Neuroscientific data indicates that images rotate smoothly, occupying all intermediate positions (Georgopoulous, et al., 1989)