2. Multimodality Alistair D N Edwards Department of Computer Science http://www-users.cs.york.ac.uk/~alistair/teaching/AMSmasterclass/Multimodality.pptx

3. Multi-X MULTI-media, MULTI-modal, MULTI- channel... there are many terms that can be prefixed by the qualifier 'multi', so that there is a danger of getting diverted into debates about the meanings of the terms and distracted from the real questions. At the same time, these discussions often centre on computers; a linguist or a playwright or a singer would not even use the terms, far less wonder whether what they do should be prefixed by 'multi'.

4. Multi-X -Why is it that the debate arises in the context of computer interaction? -Why is it that it does not arise in other contexts of communication? -Are there are other contexts in which the 'multi-X' question arises? -Are we progressing towards a situation in which the question will become obsolete or redundant?

5. How many senses do we have?

6. Five

7. ‘Extra’ senses http://www.bbc.co.uk/radio4/science/extrasenses.shtml 1Pain 2Balance 3Time 4Temperature 5Digestion

8. Class of 2015 9 Body position 10 Heat 11 Danger 12 Circadian Rhythms 13 Hunger 14 Bodily needs 15 Time 16 Pain 17 Fatigue How many senses do we have?

9. Class of 2008-10 1 Sight 2 Smell 3 Taste 4 Hearing 5 Touch 6 Balance 7 Direction 8 Humour 9 Body position 10 Heat 11 Danger 12 Circadian Rhythms 13 Hunger 14 Bodily needs 15 Time/Rhythm 16 Pain 17 Fatigue 18 Pressure 19 Empathy 20 Intuition 21 Fear 22 Hunger 23 Tiredness 24 Thirst

10. What is a sense?

11. What is a human sense? (Let’s not get distracted by questions as to whether animals or even plants have senses in the way we mean)

12. What is a human sense? An input channel? Only for external information? …or internal too?

13. What is a human sense? An input channel? Only for external information? …or internal too?

14. Objectives What modes do we use in interaction? Which others could we use? Why? How?

15. Visual dominance Psychology Neurology User interfaces The McGurk effect An example of a cross-modal effect

16. Visual veracity

18. The rotating mask illusion https://www.youtube.com/watch?v=sKa0eaKsdA0 We cannot help but see the illusion The brain ‘knows’ that it is ‘impossible’ The eye is not just a camera

19. Cross-modality We’re looking at separate modalities but the total is not necessarily the sum of the parts The role of auditory cues in modulating the perceived crispness and staleness of potato chips Journal of Sensory Studies, 19: 347–363. doi: 10.1111/j.1745-459x.2004.080403.x http://onlinelibrary.wiley.com/doi/10.1111/j.1745-459x.2004.080403.x/full

20. ‘…the perception of both the crispness and staleness was systematically altered by varying the loudness and/or frequency composition of the auditory feedback elicited during the biting action. The potato chips were perceived as being both crisper and fresher when either the overall sound level was increased, or when just the high frequency sounds (in the range of 2 kHz−20 kHz) were selectively amplified’

21. Some practical(?) possibilities Sounds Haptics Smell

22. Sounds Speech synthesized Non-speech earcons auditory icons sonification data mapping audification programs

23. Earcons Symbolic sounds with arbitrary mappings to their meanings ‘Paint’ Open Open paint

24. Auditory icon A ‘cartoon’ sound that resembles the thing it represents Sonic finder

25. Spearcons A cross between speech and non-speech sounds Speech speeded up fast but no need to learn

26. Spearcons Examples 1 2 3 http://sonify.psych.gatech.edu/research/auditorymenus/

27. Spearcons Examples 1 2 3 http://sonify.psych.gatech.edu/res earch/auditorymenus/ elephant

28. Spearcons Examples 1 2 3 http://sonify.psych.gatech.edu/res earch/auditorymenus/ elephant elevator

29. Spearcons Examples 1 2 3 http://sonify.psych.gatech.edu/researc h/auditorymenus/ elephant elevator parking meter

30. Sonification

31. Sounds Why do we not make more use of sounds in interfaces?

32. http://www.dcs.gla.ac.uk/~stephen/papers/CHI2000.pdf TermDefinition Proprioceptive Relating to sensory information about the state of the body (including cutaneous, kinesthetic, and vestibular sensations). Haptic Relating to the sense of touch. Vestibular Pertaining to the perception of head position, acceleration, and deceleration Kinesthetic Meaning the feeling of motion. Relating to sensations originating in muscles, tendons and joints Cutaneous Pertaining to the skin itself or the skin as a sense organ. Includes sensation of pressure, temperature, and pain. Tactile Pertaining to the cutaneous sense but more specifically the sensation of pressure rather than temperature or pain. Force Feedback Relating to the mechanical production of information sensed by the human kinesthetic system. Haptics - definitions

33. Geomagic Touch http://www.sensable.com/ Formerly the Sensable Phantom Omni

34. Trisenx Scent Dome http://www.trisenx.com/

35. iSmell

37. smells Why do we not make more use of smells in interfaces?

38. Conclusions Why do we not make more use of other senses/channels/modes in interaction? Which ones could we practically use more? What would be the benefits?