2. Multimodality Alistair D N Edwards Department of Computer Science http://www-module.cs.york.ac.uk/advt/classes/class01/materials/Multimodality.pptx

3. But first… ADVT Tutorials Lectures Classes

4. ADVT tutorials Autumn term Taken by supervisor Weeks 2, 3, 4, 8, 9, 10 How to write

5. ADVT lectures Alistair Autumn term Week 5 Academic Writing Week Scientific Writing

6. ADVT classes Autumn Term Week 2 Alistair: Multimodality Week 3 Paul: Games Week 4 Helen: Web Accessibility Week 5 Chris: To be announced

7. ADVT classes Spring Term Weeks 2 – 6 Contents to be announced but last year… http://www-module.cs.york.ac.uk/advt/2014-15/index.php See also http://www-module.cs.york.ac.uk/advt/

8. Multimodality: How many senses do we have? Alistair D N Edwards Department of Computer Science http://www-module.cs.york.ac.uk/advt/classes/class01/materials/Multimodality.pptx

9. Multi-X MULTI-media, MULTI-modal, MULTI-channel... there are many terms that can be prefixed by the qualifier 'multi', so that there is a danger of getting diverted into debates about the meanings of the terms and distracted from the real questions. At the same time, these discussions often centre on computers; a linguist or a playwright or a singer would not even use the terms, far less wonder whether what they do should be prefixed by 'multi'.

10. Multi-X -Why is it that the debate arises in the context of computer interaction? -Why is it that it does not arise in other contexts of communication? -Are there are other contexts in which the 'multi-X' question arises? -Are we progressing towards a situation in which the question will become obsolete or redundant?

11. How many senses do we have?

12. Five

13. ‘Extra’ senses http://www.bbc.co.uk/radio4/science/extrasenses.shtml 1Pain 2Balance 3Time 4Temperature 5Digestion

14. Class of 2015 9 Body position 10 Heat 11 Danger 12 Circadian Rhythms 13 Hunger 14 Bodily needs 15 Time 16 Pain 17 Fatigue How many senses do we have?

15. Previous answers 1 Sight 2 Smell 3 Taste 4 Hearing 5 Touch 6 Balance 7 Direction 8 Humour 9 Body position 10 Heat 11 Danger 12 Circadian Rhythms 13 Hunger 14 Bodily needs 15 Time/Rhythm 16 Pain 17 Fatigue 18 Pressure 19 Empathy 20 Intuition 21 Fear 22 Hunger 23 Tiredness 24 Thirst

16. What is a sense?

17. What is a human sense? (Let’s not get distracted by questions as to whether animals or even plants have senses in the way we mean)

18. What is a human sense? An input channel? Only for external information? …or internal too?

19. What is a human sense? An input channel? Only for external information? …or internal too?

20. Senses External Internal Compound? A synthesis of inputs fear humour empathy ?

21. Objectives What modes do we use in interaction? Which others could we use? Why? How? Individual senses Combining senses/modes

22. Some practical(?) possibilities Sounds Haptics Smell

23. Sounds Speech synthesized Non-speech earcons auditory icons sonification data mapping audification programs

24. Synthesized speech Who is this? …and this? Which is the better voice? How can we make synthetic voices more acceptable? Better use of pauses? Newell, C (2009) Place, authenticity, and time: a framework for liveness in synthetic speech University of York PhD Thesis Newell, C., Edwards, A. D. N. & Cairns, P. (2011) 'Liveness' in human-machine interaction, International Journal of Performance Arts and Digital Media, 7, 2, pp.221-237.

25. Earcons Symbolic sounds with arbitrary mappings to their meanings ‘Paint’ Open Open paint

26. Earcons Steve Brewster, Glasgow Brewster, S.A., Wright, P.C. & Edwards, A.D.N. (1992). A detailed investigation into the effectiveness of earcons. In G. Kramer (Ed.), Auditory display, sonification, audification and auditory interfaces. The Proceedings of the First International Conference on Auditory Display, Santa Fe Institute, Santa Fe, NM: Addison-Wesley, pp. 471-498 Brewster, S. A (1991). Providing a Model for the Use of Sound in User Interfaces, University of York DPhil thesis

27. Algebra earcons

28. Stevens, R.D., Brewster, S.A., Wright, P.C. & Edwards, A.D.N. (1994). Providing an audio glance at algebra for blind readers. In G. Kramer & S. Smith (Eds.), Proceedings of ICAD'94, Santa Fe Institute, Santa Fe: Addison-Wesley, pp. 21-30 Stevens, R. D. (1996) Principles for the design of auditory interfaces to present complex information to blind people, University of York DPhil thesis

29. Auditory icon A ‘cartoon’ sound that resembles the thing it represents Sonic finder

30. Auditory icon Gaver, W. W. (1989). The SonicFinder: An interface that uses auditory icons. Human-Computer Interaction, 4(1), 67- 94

31. Spearcons Somewhere between speech and an earcon speeded up speech Spearcon 1 Spearcon 1 Spearcon 2 Spearcon 2 Spearcon 3 Spearcon 3

32. Spearcons Walker, B. N., & Kogan, A. (2009). Spearcons enhance performance and preference for auditory menus on a mobile phone. Invited paper in Proceedings of the 5th International Conference on Universal Access in Human-Computer Interaction (UAHCI) at HCI International 2009, San Diego, CA, USA, (19-24 July).

33. Sonification

34. Edwards, A. D. N., Hines G. and Hunt, A. (2008). Segmentation of biological cell images for sonification. 2008 International Congress on Image and Signal Processing (CISP 2008), Sanya, China, IEEE Computer Society, pp. 128-132.

35. Sounds Why do we not make more use of sounds in interfaces?

36. Annoyance? What makes a sound annoying?

37. Experiment Sample 1 Sample 2 Sample 3 Sample 4

38. Experiment Sample 1 Sample 2 Sample 3 Sample 4

39. Experiment Sample 1 Sample 2 Sample 3 Sample 4

40. Experiment Sample 1 Sample 2 Sample 3 Sample 4

41. Haptics

42. http://www.dcs.gla.ac.uk/~stephen/papers/CHI2000.pdf TermDefinition Proprioceptive Relating to sensory information about the state of the body (including cutaneous, kinesthetic, and vestibular sensations). Haptic Relating to the sense of touch. Vestibular Pertaining to the perception of head position, acceleration, and deceleration Kinesthetic Meaning the feeling of motion. Relating to sensations originating in muscles, tendons and joints Cutaneous Pertaining to the skin itself or the skin as a sense organ. Includes sensation of pressure, temperature, and pain. Tactile Pertaining to the cutaneous sense but more specifically the sensation of pressure rather than temperature or pain. Force Feedback Relating to the mechanical production of information sensed by the human kinesthetic system. Haptics - definitions

43. Geomagic Touch https://youtu.be/nmrviXro65g http://www.sensable.com/ Formerly the Sensable Phantom Omni

44. Invisible 3-D haptics http://phys.org/news/2014-12-team-invisible-d-haptic-video.html

45. Invisible 3-D haptics Carter, T., Seah, S. A., Long, B., Drinkwater, B., & Subramanian, S. (2013, October). Ultrahaptics: multi-point mid-air haptic feedback for touch surfaces. In Proceedings of the 26th annual ACM symposium on User interface software and technology (pp. 505-514). ACM.

46. Olfactory

47. Trisenx Scent Dome http://www.trisenx.com/

48. iSmell

50. Olfactory device ode Older people who may forget to eat Releases food smells at meal times

51. ode Fragrances Fresh Orange Juice Cherry Bakewell Tart Homemade Curry Pink Grapefruit Braised Beef Casserole Black Forest Gateau

52. Smells Why do we not make more use of smells in interfaces?

53. Combining senses

54. The ‘bandwidth problem’

55. Vision How much information do we receive visually? The real power is in being able to focus

56. Vision versus audition How many screen objects could we hear? ~ 8 Pitt, I. J. and Edwards, A. D. N. (1995). Pointing in an auditory interface for blind users. in Intelligent Systems for the 21st Century: Proceedings of the 1995 IEEE International Conference on Systems, Man and Cybernetics, (Vancouver), IEEE. pp. 280-285. Can use auditory streams Bregman, A, (1990) Auditory Scene Analysis, MIT Press Mitsopoulos, E. N. and Edwards, A. D. N. (1999). A principled design methodology for auditory interactionMitsopoulos, E. N. and Edwards, A. D. N. (1999). A principled design methodology for auditory interaction. in M. A. Sasse and C. Johnson (eds.), Proceedings of Interact 99, (Edinburgh), IOS Press. pp. 263-271 Interact 99,

57. Combining senses Can the whole be greater than the sum of the parts?

58. T4: turning talking tactile tablet Touch Speech Non-speech Rotation haptics (Vision) https://youtu.be/4nW9-zKmKUw

59. Conflicting senses McGurk Effect https://youtu.be/G-lN8vWm3m0

60. Visual dominance Psychology Neurology User interfaces

61. Conflicting senses Poppadoms

62. Experiment Sample 1 Crisp poppadom, crisp sounds Sample 2 Soft poppadom, crisp sounds Sample 3 Crisp poppadom, white noise Sample 4 Soft poppadom, white noise

63. Conclusions Why do we not make more use of other senses/channels/modes in interaction? Which ones could we practically use more? What would be the benefits?

64. Visual veracity