Multimodality Alistair D N Edwards Department of Computer Science

But first… ADVT Tutorials Lectures Classes

ADVT tutorials Autumn term Taken by supervisor Weeks 2, 3, 4, 8, 9, 10 How to write

ADVT lectures Alistair Autumn term Week 5 Academic Writing Week Scientific Writing

ADVT classes Autumn Term Week 2 Alistair: Multimodality Week 3 Paul: Games Week 4 Helen: Web Accessibility Week 5 Chris: To be announced

ADVT classes Spring Term Weeks 2 – 6 Contents to be announced but last year… See also

Multimodality: How many senses do we have? Alistair D N Edwards Department of Computer Science

Multi-X MULTI-media, MULTI-modal, MULTI-channel... there are many terms that can be prefixed by the qualifier 'multi', so that there is a danger of getting diverted into debates about the meanings of the terms and distracted from the real questions. At the same time, these discussions often centre on computers; a linguist or a playwright or a singer would not even use the terms, far less wonder whether what they do should be prefixed by 'multi'.

Multi-X -Why is it that the debate arises in the context of computer interaction? -Why is it that it does not arise in other contexts of communication? -Are there are other contexts in which the 'multi-X' question arises? -Are we progressing towards a situation in which the question will become obsolete or redundant?

How many senses do we have?

Five

‘Extra’ senses 1Pain 2Balance 3Time 4Temperature 5Digestion

Class of Body position 10 Heat 11 Danger 12 Circadian Rhythms 13 Hunger 14 Bodily needs 15 Time 16 Pain 17 Fatigue How many senses do we have?

Previous answers 1 Sight 2 Smell 3 Taste 4 Hearing 5 Touch 6 Balance 7 Direction 8 Humour 9 Body position 10 Heat 11 Danger 12 Circadian Rhythms 13 Hunger 14 Bodily needs 15 Time/Rhythm 16 Pain 17 Fatigue 18 Pressure 19 Empathy 20 Intuition 21 Fear 22 Hunger 23 Tiredness 24 Thirst

What is a sense?

What is a human sense? (Let’s not get distracted by questions as to whether animals or even plants have senses in the way we mean)

What is a human sense? An input channel? Only for external information? …or internal too?

What is a human sense? An input channel? Only for external information? …or internal too?

Senses External Internal Compound? A synthesis of inputs fear humour empathy ?

Objectives What modes do we use in interaction? Which others could we use? Why? How? Individual senses Combining senses/modes

Some practical(?) possibilities Sounds Haptics Smell

Sounds Speech synthesized Non-speech earcons auditory icons sonification data mapping audification programs

Synthesized speech Who is this? …and this? Which is the better voice? How can we make synthetic voices more acceptable? Better use of pauses? Newell, C (2009) Place, authenticity, and time: a framework for liveness in synthetic speech University of York PhD Thesis Newell, C., Edwards, A. D. N. & Cairns, P. (2011) 'Liveness' in human-machine interaction, International Journal of Performance Arts and Digital Media, 7, 2, pp

Earcons Symbolic sounds with arbitrary mappings to their meanings ‘Paint’ Open Open paint

Earcons Steve Brewster, Glasgow Brewster, S.A., Wright, P.C. & Edwards, A.D.N. (1992). A detailed investigation into the effectiveness of earcons. In G. Kramer (Ed.), Auditory display, sonification, audification and auditory interfaces. The Proceedings of the First International Conference on Auditory Display, Santa Fe Institute, Santa Fe, NM: Addison-Wesley, pp Brewster, S. A (1991). Providing a Model for the Use of Sound in User Interfaces, University of York DPhil thesis

Algebra earcons

Stevens, R.D., Brewster, S.A., Wright, P.C. & Edwards, A.D.N. (1994). Providing an audio glance at algebra for blind readers. In G. Kramer & S. Smith (Eds.), Proceedings of ICAD'94, Santa Fe Institute, Santa Fe: Addison-Wesley, pp Stevens, R. D. (1996) Principles for the design of auditory interfaces to present complex information to blind people, University of York DPhil thesis

Auditory icon A ‘cartoon’ sound that resembles the thing it represents Sonic finder

Auditory icon Gaver, W. W. (1989). The SonicFinder: An interface that uses auditory icons. Human-Computer Interaction, 4(1),

Spearcons Somewhere between speech and an earcon speeded up speech Spearcon 1 Spearcon 1 Spearcon 2 Spearcon 2 Spearcon 3 Spearcon 3

Spearcons Walker, B. N., & Kogan, A. (2009). Spearcons enhance performance and preference for auditory menus on a mobile phone. Invited paper in Proceedings of the 5th International Conference on Universal Access in Human-Computer Interaction (UAHCI) at HCI International 2009, San Diego, CA, USA, (19-24 July).

Sonification

Edwards, A. D. N., Hines G. and Hunt, A. (2008). Segmentation of biological cell images for sonification International Congress on Image and Signal Processing (CISP 2008), Sanya, China, IEEE Computer Society, pp

Sounds Why do we not make more use of sounds in interfaces?

Annoyance? What makes a sound annoying?

Experiment Sample 1 Sample 2 Sample 3 Sample 4

Experiment Sample 1 Sample 2 Sample 3 Sample 4

Experiment Sample 1 Sample 2 Sample 3 Sample 4

Experiment Sample 1 Sample 2 Sample 3 Sample 4

Haptics

TermDefinition Proprioceptive Relating to sensory information about the state of the body (including cutaneous, kinesthetic, and vestibular sensations). Haptic Relating to the sense of touch. Vestibular Pertaining to the perception of head position, acceleration, and deceleration Kinesthetic Meaning the feeling of motion. Relating to sensations originating in muscles, tendons and joints Cutaneous Pertaining to the skin itself or the skin as a sense organ. Includes sensation of pressure, temperature, and pain. Tactile Pertaining to the cutaneous sense but more specifically the sensation of pressure rather than temperature or pain. Force Feedback Relating to the mechanical production of information sensed by the human kinesthetic system. Haptics - definitions

Geomagic Touch Formerly the Sensable Phantom Omni

Invisible 3-D haptics

Invisible 3-D haptics Carter, T., Seah, S. A., Long, B., Drinkwater, B., & Subramanian, S. (2013, October). Ultrahaptics: multi-point mid-air haptic feedback for touch surfaces. In Proceedings of the 26th annual ACM symposium on User interface software and technology (pp ). ACM.

Olfactory

Trisenx Scent Dome

iSmell

Olfactory device ode Older people who may forget to eat Releases food smells at meal times

ode Fragrances Fresh Orange Juice Cherry Bakewell Tart Homemade Curry Pink Grapefruit Braised Beef Casserole Black Forest Gateau

Smells Why do we not make more use of smells in interfaces?

Combining senses

The ‘bandwidth problem’

Vision How much information do we receive visually? The real power is in being able to focus

Vision versus audition How many screen objects could we hear? ~ 8 Pitt, I. J. and Edwards, A. D. N. (1995). Pointing in an auditory interface for blind users. in Intelligent Systems for the 21st Century: Proceedings of the 1995 IEEE International Conference on Systems, Man and Cybernetics, (Vancouver), IEEE. pp Can use auditory streams Bregman, A, (1990) Auditory Scene Analysis, MIT Press Mitsopoulos, E. N. and Edwards, A. D. N. (1999). A principled design methodology for auditory interactionMitsopoulos, E. N. and Edwards, A. D. N. (1999). A principled design methodology for auditory interaction. in M. A. Sasse and C. Johnson (eds.), Proceedings of Interact 99, (Edinburgh), IOS Press. pp Interact 99,

Combining senses Can the whole be greater than the sum of the parts?

T4: turning talking tactile tablet Touch Speech Non-speech Rotation haptics (Vision)

Conflicting senses McGurk Effect

Visual dominance Psychology Neurology User interfaces

Conflicting senses Poppadoms

Experiment Sample 1 Crisp poppadom, crisp sounds Sample 2 Soft poppadom, crisp sounds Sample 3 Crisp poppadom, white noise Sample 4 Soft poppadom, white noise

Conclusions Why do we not make more use of other senses/channels/modes in interaction? Which ones could we practically use more? What would be the benefits?

Visual veracity