1. SYNESTHESI A Multimodal Communication 2008 Ingvar Lind sight taste smell hearing touch

2. Synesthesia - Multimodal Communication 2008 Ingvar Lind Muldimodality in communication Different modalities are used simultaneously in multimodal communication. The modalities interact in certain ways. There are production and reception modalities. The receiving modalities are received through the senses, which translate physical properties in the world to mental modules. The mental modules are both experiences and unconcious simulation processes that together combine to the world we know and feel. The modalities are combined in an indirect fashion in multimodal communication. In synesthesia, the combination of modalities is direct and unchanging.

3. Synesthesia - Multimodal Communication 2008 Ingvar Lind What is Synesthesia?

4. Synesthesia - Multimodal Communication 2008 Ingvar Lind Definitions of Synesthesia:

5. Synesthesia - Multimodal Communication 2008 Ingvar Lind Definitions of Synesthesia: A condition in which stimulation of one sense also evokes another sensory pathway in the brain.

6. Synesthesia - Multimodal Communication 2008 Ingvar Lind Definitions of Synesthesia: A condition in which stimulation of one sense also evokes another sensory pathway in the brain. It is involuntary.

7. Synesthesia - Multimodal Communication 2008 Ingvar Lind Definitions of Synesthesia: A condition in which stimulation of one sense also evokes another sensory pathway in the brain. It is involuntary. It is automatic (consistent and permanent).

8. Synesthesia - Multimodal Communication 2008 Ingvar Lind Definitions of Synesthesia: A condition in which stimulation of one sense also evokes another sensory pathway in the brain. It is involuntary. It is automatic (consistent and permanent). It can be cross-modal and inter-modal (=cross-sub- modal).

9. Synesthesia - Multimodal Communication 2008 Ingvar Lind Definitions of Synesthesia: A condition in which stimulation of one sense also evokes another sensory pathway in the brain. It is involuntary. It is automatic (consistent and permanent). It can be cross-modal and inter-modal (=cross-sub- modal). It is highly memorable.

10. Synesthesia - Multimodal Communication 2008 Ingvar Lind Definitions of Synesthesia: A condition in which stimulation of one sense also evokes another sensory pathway in the brain. It is involuntary. It is automatic (consistent and permanent). It can be cross-modal and inter-modal (=cross-sub- modal). It is highly memorable. It is paired with emotion/mood.

11. Synesthesia - Multimodal Communication 2008 Ingvar Lind Definitions of Synesthesia: A condition in which stimulation of one sense also evokes another sensory pathway in the brain. It is involuntary. It is automatic (consistent and permanent). It can be cross-modal and inter-modal (=cross-sub- modal). It is highly memorable. It is paired with emotion/mood. It is most often unique to one person, but similarities exist between the synesthetic types.

12. Synesthesia - Multimodal Communication 2008 Ingvar Lind Definitions of Synesthesia: A condition in which stimulation of one sense also evokes another sensory pathway in the brain. It is involuntary. It is automatic (consistent and permanent). It can be cross-modal and inter-modal (=cross-sub- modal). It is highly memorable. It is paired with emotion/mood. It is most often unique to one person, but similarities exist between the synesthetic types. The synesthetic phenomena is an extension of perception rather than a limitation, as in color blindness.

13. Synesthesia - Multimodal Communication 2008 Ingvar Lind Definitions of Synesthesia: A condition in which stimulation of one sense also evokes another sensory pathway in the brain. It is involuntary. It is automatic (consistent and permanent). It can be cross-modal and inter-modal (=cross-sub- modal). It is highly memorable. It is paired with emotion/mood. It is most often unique to one person, but similarities exist between the synesthetic types. The synesthetic phenomena is an extension of perception rather than a limitation, as in color blindness. It is directional, from one modality to another.

14. Synesthesia - Multimodal Communication 2008 Ingvar Lind Examples of Synesthesia: numbers -> shapes/colors/textures/movements Extraordinary people - Daniel Tammet Part 2: Beginning 3 min. http://www.youtube.com/watch?v=ydiBnm1ejhM&NR Part 3: Beginning 3 min. http://www.youtube.com/watch?v=4BxhYPKXXnQ

15. Synesthesia - Multimodal Communication 2008 Ingvar Lind Examples of Synesthesia: graphemes (letters) -> sounds Sounds of the Alphabet, 1 min. http://www.youtube.com/watch?v=XOuU0ppCcn8

16. Synesthesia - Multimodal Communication 2008 Ingvar Lind Examples of Synesthesia: drugs -> synesthesia Synesthesia caused by drugs, 2 min. http://www.youtube.com/watch?v=XZ7jqgEYSLU

17. Synesthesia - Multimodal Communication 2008 Ingvar Lind Examples of Synesthesia: music -> colors graphemes (numbers and letters) -> colors taste -> shape speech sound -> taste time units -> positions in space Seeing Life in Colors: Crosswired Senses, ABC NEWS 3 min. http://www.youtube.com/watch?v=KApieSGlyBk

18. Synesthesia - Multimodal Communication 2008 Ingvar Lind Examples of simulated Synesthesia: sound -> moving shapes music -> moving lines Synthesizer Synesthesia, 1.22 min. http://www.youtube.com/watch?v=l4_T5VYMHqc Toccata and Fugue in D minor, 8.34 min. http://www.youtube.com/watch?v=ipzR9bhei_o

19. Synesthesia - Multimodal Communication 2008 Ingvar Lind Colors Tastes Smells Touch Temperature Vibration, skin Shape in 3D Movements Pain Positions Synesthesia Modalities and Modules Concepts (language) Concepts (mathematics) Emotions Music Sound Time units Graphemes Phonemes Faces Bodies etc.

20. Synesthesia - Multimodal Communication 2008 Ingvar Lind Categorical perception and Synesthesia Categorical perception is perception of a distinct number of kinds, opposed to a perception of continuous dimensions in experience. The hue circle is an example of a continuous dimension that perception labels with a small number of hue categories: red, orange, yellow, green, blue, purple. Categorical perception is a necessare function of perception to scale down the complexity of information from the world into a relatively small number of labels. Synesthesia are often connections between categories of modalities rather than between continuous dimensions. This suggests that language, or other more abstract brain functions has a major role in synesthesia.

21. Synesthesia - Multimodal Communication 2008 Ingvar Lind Synesthesia Questions Can structures/patterns of different modalities be described and compared in a functional way (similarities, differences of neural patterns underlying functional structures)?

22. Synesthesia - Multimodal Communication 2008 Ingvar Lind Synesthesia Questions Can structures/patterns of different modalities be described and compared in a functional way (similarities, differences of neural patterns underlying functional structures)? 1.Cross-sensory metaphores (blue music, blue feelings)

23. Synesthesia - Multimodal Communication 2008 Ingvar Lind Synesthesia Questions Can structures/patterns of different modalities be described and compared in a functional way (similarities, differences of neural patterns underlying functional structures)? 1.Cross-sensory metaphores (blue music, blue feelings) 2.Geometrical properties are common between the modalities (2D structures, 3D-structures (places, movements, textures and directions in a room in vision, auditory and tactile modalities)

24. Synesthesia - Multimodal Communication 2008 Ingvar Lind Synesthesia Questions Can structures/patterns of different modalities be described and compared in a functional way (similarities, differences of neural patterns underlying functional structures)? 1.Cross-sensory metaphores (blue music, blue feelings) 2.Geometrical properties are common between the modalities (2D structures, 3D-structures (places, movements, textures and directions in a room in vision, auditory and tactile modalities) 3.The Bouba-Kiki effect

25. Synesthesia - Multimodal Communication 2008 Ingvar Lind Synesthesia Questions Can structures/patterns of different modalities be described and compared in a functional way (similarities, differences of neural patterns underlying functional structures)? 1.Cross-sensory metaphores (blue music, blue feelings) 2.Geometrical properties are common between the modalities (2D structures, 3D-structures (places, movements, textures and directions in a room in vision, auditory and tactile modalities) 3.The Bouba-Kiki effect (unknown words suggest certain shapes) 4.Temperature and color (blue feels cold, red feels warm)

26. Synesthesia - Multimodal Communication 2008 Ingvar Lind Synesthesia Questions The mind-body-dualism

27. Synesthesia - Multimodal Communication 2008 Ingvar Lind Synesthesia Questions The mind-body-dualism 1.The mind studied with introspection, interviews and psychophysics. Only the mental functions are considered.

28. Synesthesia - Multimodal Communication 2008 Ingvar Lind Synesthesia Questions The mind-body-dualism 1.The mind studied with introspection, interviews and psychophysics. Only the mental functions are considered. 2.The body as controlled by a computerlike brain, with neural network models explaining what goes on between input and output. Can the two perspectives be fused into one? Can synesthesia show us how the mind works?

29. Synesthesia - Multimodal Communication 2008 Ingvar Lind Synesthesia Questions Theories of synesthesia 1.Module theory Cognitive psychology: modules have separate functions in the brain. Some sensory modules are linked in synesthesia.

30. Synesthesia - Multimodal Communication 2008 Ingvar Lind Synesthesia Questions Theories of synesthesia 1.Module theory Cognitive psychology: modules have separate functions in the brain. Some sensory modules are linked in synesthesia. 2. Matching theory Cross-modal matching, one modality can process information patterns that another modality process: size, shape, intensity, texture, quality, time, space.

31. ”Synesthesia”-model for a connection between color and sound Vertical direction: Brightness Central vertical arrow: neutral colors (black, gray, white, shiny) Horizontal circle: Hues (red, orange, yellow, green, blue, purple, red) Vertical direction: mental Volume of sound (or maybe Pitch) Central vertical arrow: neutral vowels (central in the formant space) Horizontal circle: Vowel sounds (timbre) (a, e, i, y, u, o, a) 1-1 mapping Dotted line: strength of Hue or strength of Vowel

32. Synesthesia - Multimodal Communication 2008 Ingvar Lind An experiment in Synesthesia Make a synesthetic one-to-one connection model between the 2D vowel space and the 2D hue space, using the computer programs Praat and ColorSpace. What is the most natural connection between the two? Explain why. Vowel space: Praat (Macintosh, Pc, Linux, SGI) http://www.fon.hum.uva.nl/praat/ Praat objects – New – Sound – Create Sound from Voweleditior Hue space: ColorSpace (Windows program, similar to Macintosh Color Picker) http://www.codeguru.com/cpp/w-d/dislog/commondialogs/article.php/c1861 View – Color Space (Color Picker)

33. Synesthesia - Multimodal Communication 2008 Ingvar Lind References Sean Day, “Trends in colored letter synesthesia”, research on colored letters: http://web.mit.edu/synesthesia/www/trends.html Sensequence, a huge webpage on synesthesia: http://www.sensequence.de/indexen.html "Art and Synesthesia: in search of the synesthetic experience": http://www.doctorhugo.org/synesthesia/art/index.html ” The Phenomenology of Synaesthesia”, V.S. Ramachandran and E.M. Hubbard: http://www.imprint.co.uk/pdf/R_H-follow-up.pdf “The perceptual reality of synesthetic colors”: http://www.psy.vanderbilt.edu/faculty/blake/PDFs/PalmeriEtAl_PNAS_2002.pdf Cognitive, Affective, & Behavioral Neuroscience 2004, 4 (3), 335-34,” Not all synesthetes are created equal: Projector versus associator synesthetes” http://watarts.uwaterloo.ca/~src/papers/Dixon.CABN.2004.pdf Consciousness and Cognition 16 (2007) 913–931 “Varieties of grapheme-colour synesthesia: A new theory of phenomenological and behavioural differences” http://home.comcast.net/~sean.day/Ward%20et%20al%202007.pdf The neuropsychology of synesthesia: http://scienceblogs.com/neurophilosophy/2007/09/the_neuropsychology_of_synest.php

34. Synesthesia - Multimodal Communication 2008 Ingvar Lind References, books Richard E. Cytowic Synesthesia, A Union of the Senses, 1989 Richard E. Cytowic The Man Who Tasted Shapes, 1993 Monica Vester En värld av nyanser, 2004