Multimodality Alistair D N Edwards Department of Computer Science

Slides:

Advertisements

Similar presentations

Rebecca Dalton Rayvant Kohli Jenna Raithel Megan Saracino Irsham Zaman.

Advertisements

The Perception of Speech. Speech is for rapid communication Speech is composed of units of sound called phonemes –examples of phonemes: /ba/ in bat, /pa/

Hearing, Touch, Taste and Smell. Hearing Audition – the sense of hearing.

EXPLORING PSYCHOLOGY EIGHTH EDITION IN MODULES David Myers PowerPoint Slides Aneeq Ahmad Henderson State University Worth Publishers, © 2011.

Sponge: Set up Cornell Notes on pg. 41 Topic: 12.1 General and Special Senses Essential Question: 2.1 Atoms, Ions, and Molecules 12.1 General and Special.

Supporting Children’s Diverse Learning. All of these children are demonstrating signs of problems with sensory integration: Thomas covers his ears when.

Charles Spence Department of Experimental Psychology, Oxford University New Perspectives from the Human Sciences Consumer Focus Workshop (November, 2001)

TOK: Ways of Knowing Sense Perception. We perceive the world through our 5 senses. Our 5 senses are: Sight Sight Hearing Hearing Touch Touch Smell Smell.

Teleoperation Interfaces. Introduction Interface between the operator and teleoperator! Teleoperation interface is like any other HMI H(mobile)RI = TI.

Designing a User Interface for People with Disabilities u u

SENSATION AND PERCEPTION. DEFINITIONS  Sensation: the process in which our sensory receptors and nervous system receive and represent stimulus energies.

THE FIVE SENSES Mrs.Whitmore CCSD Standard- (3)2.2 use and identify five senses, matching the appropriate body part to each sense.

Structure of sense organs

Sensory Integration Dysfunction By Ricca Klein. Sensory Integration Normal Sensory Integration –Neurological process of organizing info from body and.

Energy Senses Vision The eye Transduction In the brain Theories of Color Vision Trichromatic theory Opponent-process theory.

How do we choose a skill to perform? What decisions do we need to make about performing a skill? Need to decide when to perform the skill Need to decide.

 Can you explain an example of touch or skin related disorder?  Can you notice the differences between vestibular, kinesthetic and cutaneous sense?

산업경영공학과 IMEN 315 인간공학 5. Auditory System SOUND: THE AUDITORY STIMULUS  sound – a vibration of the air molecules  a sine wave with amplitude (loudness)

Sensation and Perception

Somatic and Special Senses

Sensing self motion Key points: Why robots need self-sensing Sensors for proprioception in biological systems in robot systems Position sensing Velocity.

Touch: 1. State the purpose of pain, and describe the biopsychosocial approach to pain. 2. Describe the sense of taste, and explain the principle of sensory.

Vision Hearing Other Senses Perception 1 Perception 2.

IAS2223: Human Computer Interaction Chapter 5: Universal Design.

Sensory Smarts Occupational Therapy Department. The Seven Senses Visual Olfactory Gustatory Auditory Tactile Propriopection Vestibular.

Sensory Integration Mazyad Alotaibi. 2 Definition of Normal Sensory Integration 1.Neurological process of organizing information from body and environment.

Sensation and Perception Chapter 4

Sensory Modalities General Senses: 1. Somatic (Exteroceptors) a. Touch b. Pressure c. Temperature d. Proprioception e. Pain 2. Visceral (Interoceptors)

The Remaining Senses Unit 6 Lesson 3. Objectives Review the physical properties of sound and light waves. Compare and contrast the senses of taste and.

Sensory Training. Sensory Processing: Sensory Processing difficulties occurs when sensory information coming in from the senses is not interpreted efficiently.

Animal senses: how they detect stimulus

Fundamentals of Sensation and Perception ERIK CHEVRIER SEPTEMBER 9 TH, 2015.

VIRTUAL REALITY (VR) INTRODUCTION AND BASIC APPLICATIONS الواقع الافتراضي : مقدمة وتطبيقات Dr. Naji Shukri Alzaza Assist. Prof. of Mobile technology Dean.

Low cost tactile feedback platform for teleoperation and VR sensing Human Machine Interaction & Low cost technologies Adrien Moucheboeuf - July 8 th, 2015.

Multimodality Alistair D N Edwards Department of Computer Science

1 Challenges visual perception auditory perception speech perception lack of invariance no distinct segments lightness contrast pitch depends on loudness.

Auditory & tactile displays EGR 412 Human Factors Engineering ISE

Applying Sensory Processing Techniques to Positively Impact Behavior Part 1: Sensory Processing and Dysfunction Amanda Martinage OTR/L, M.Ed

Immersive Displays The other senses…. 1962… Classic Human Sensory Systems Sight (Visual) Hearing (Aural) Touch (Tactile) Smell (Olfactory) Taste (Gustatory)

Sensory Systems Lesson 14. Sensory Information n Detection of changes in environment l external or internal n 4 main functions l perception l control.

Perception in Motor Development. What is Perception? A multistage process that takes place in the BRAIN. It includes – Selection – Processing – Organization.

11/3 & 11/4 Do Now: Take out your TOK ESSAY!!!! Do Now: Take out your TOK ESSAY!!!! Make sure your NAME & TITLE are at the top!!!! Make sure your NAME.

How many senses do we have? An introduction to multimodality

Science Foundation The Human Body. We have talked a lot about nature and how nature is science. Did you know that our bodies are science too?

Seeinghearing speakingthinking PET Scans. Frontal Temporal Parietal Occipital Cerebellum The Boundaries of the Lobes.

DID YOU HEAR THAT?. DID YOU HEAR THAT? Volunteer Time!!

Other Senses. THE SKIN SENSES  Pressure, Temperature, Pain  Gate Theory: only a certain amount of information can be processed by the nervous system.

Tactile (Touch) Sense Two different systems: Discriminatory: Tells you where and what you are touching. So that we don’t have to rely on visual cues. Protective:

Multimodal and Natural computer interaction Evelina Stanevičienė.

Proprioception Sben Grans-Korsh. It is not an exteroceptive senses (sight, taste, smell, touch, hearing, and balance) Proprioception is a sense that gives.

SIE 515 Touch and Haptics Class 19.

Human Computer Interaction Lecture 20 Universal Design

THE SENSE OF TOUCH The sense of touch can be broken down into four distinct skin sensations: pressure, warmth, cold, and pain → like the other senses,

Aim: What factors influence our sensitivity

Chapter 4 Section 4 & 5 Goal Four: Explain how the skin, chemical, kinesthetic, and vestibular senses work.

Aim: What factors influence our sensitivity

Nervous System Perception Memory Planning …… Sensory inputs:

Perception is a multistage process that takes place in the BRAIN

DID YOU HEAR THAT?.

October 27, 2013 Eq- How is information from our sensory organs processed by the brain? Standard- BF 2 Table of Contents: 42. The 7 senses 43. Chapter.

Touch 3.11 How does the skin provide pleasant and unpleasant

The Nervous System By: Mr. Hunter.

The Senses: Introduction and Receptors

Multimodal Human-Computer Interaction New Interaction Techniques 22. 1

Sensory Systems Lecture 12.

universal design (web accessibility)

Sensation and Perception

What is the last dream you remember that is school appropriate?

Psychology “the science of mind and behaviour”

Touch The body or somatic senses includes skin senses, which detect touch, temperature, and pain. Pacinian corpuscles, located beneath the skin, detect.

Presentation transcript:

Multimodality Alistair D N Edwards Department of Computer Science

Multi-X MULTI-media, MULTI-modal, MULTI- channel... there are many terms that can be prefixed by the qualifier 'multi', so that there is a danger of getting diverted into debates about the meanings of the terms and distracted from the real questions. At the same time, these discussions often centre on computers; a linguist or a playwright or a singer would not even use the terms, far less wonder whether what they do should be prefixed by 'multi'.

Multi-X -Why is it that the debate arises in the context of computer interaction? -Why is it that it does not arise in other contexts of communication? -Are there are other contexts in which the 'multi-X' question arises? -Are we progressing towards a situation in which the question will become obsolete or redundant?

How many senses do we have?

Five

‘Extra’ senses 1Pain 2Balance 3Time 4Temperature 5Digestion

Class of Body position 10 Heat 11 Danger 12 Circadian Rhythms 13 Hunger 14 Bodily needs 15 Time 16 Pain 17 Fatigue How many senses do we have?

Class of Sight 2 Smell 3 Taste 4 Hearing 5 Touch 6 Balance 7 Direction 8 Humour 9 Body position 10 Heat 11 Danger 12 Circadian Rhythms 13 Hunger 14 Bodily needs 15 Time/Rhythm 16 Pain 17 Fatigue 18 Pressure 19 Empathy 20 Intuition 21 Fear 22 Hunger 23 Tiredness 24 Thirst

What is a sense?

What is a human sense? (Let’s not get distracted by questions as to whether animals or even plants have senses in the way we mean)

What is a human sense? An input channel? Only for external information? …or internal too?

What is a human sense? An input channel? Only for external information? …or internal too?

Objectives What modes do we use in interaction? Which others could we use? Why? How?

Visual dominance Psychology Neurology User interfaces The McGurk effect An example of a cross-modal effect

Visual veracity

The rotating mask illusion We cannot help but see the illusion The brain ‘knows’ that it is ‘impossible’ The eye is not just a camera

Cross-modality We’re looking at separate modalities but the total is not necessarily the sum of the parts The role of auditory cues in modulating the perceived crispness and staleness of potato chips Journal of Sensory Studies, 19: 347–363. doi: /j x x

‘…the perception of both the crispness and staleness was systematically altered by varying the loudness and/or frequency composition of the auditory feedback elicited during the biting action. The potato chips were perceived as being both crisper and fresher when either the overall sound level was increased, or when just the high frequency sounds (in the range of 2 kHz−20 kHz) were selectively amplified’

Some practical(?) possibilities Sounds Haptics Smell

Sounds Speech synthesized Non-speech earcons auditory icons sonification data mapping audification programs

Earcons Symbolic sounds with arbitrary mappings to their meanings ‘Paint’ Open Open paint

Auditory icon A ‘cartoon’ sound that resembles the thing it represents Sonic finder

Spearcons A cross between speech and non-speech sounds Speech speeded up fast but no need to learn

Spearcons Examples

Spearcons Examples earch/auditorymenus/ elephant

Spearcons Examples earch/auditorymenus/ elephant elevator

Spearcons Examples h/auditorymenus/ elephant elevator parking meter

Sonification

Sounds Why do we not make more use of sounds in interfaces?

TermDefinition Proprioceptive Relating to sensory information about the state of the body (including cutaneous, kinesthetic, and vestibular sensations). Haptic Relating to the sense of touch. Vestibular Pertaining to the perception of head position, acceleration, and deceleration Kinesthetic Meaning the feeling of motion. Relating to sensations originating in muscles, tendons and joints Cutaneous Pertaining to the skin itself or the skin as a sense organ. Includes sensation of pressure, temperature, and pain. Tactile Pertaining to the cutaneous sense but more specifically the sensation of pressure rather than temperature or pain. Force Feedback Relating to the mechanical production of information sensed by the human kinesthetic system. Haptics - definitions

Geomagic Touch Formerly the Sensable Phantom Omni

Trisenx Scent Dome

iSmell

smells Why do we not make more use of smells in interfaces?

Conclusions Why do we not make more use of other senses/channels/modes in interaction? Which ones could we practically use more? What would be the benefits?