Multimodal Virtual Environments: Response Times, Attention, and Presence B 陳柏叡

Introduction Efficiently combine sensory information from two or three channels Three channels: vision, audio, haptic Differentiates from older technologies - communicating only via a single channel The underlying cognitive mechanisms are still elusive and unknown

Environments Richness Results in a Complete and Coherent Experience Single channel conveys limited and insufficient information to the senses - Engenders a lower sense of presence Multimodal VE provide a greater extent of sensory information to the observer - Sense of being present is felt stronger

Multimodal VE are Mimicking Reality Better Many of daily experiences in the real world are fundamentally multimodal Grasping an object – visual, haptic Gastronomic pleasure – taste, smell, vision Multimodal VE have a clear advantage, in mimicking a multimodal phenomenon

Better Integration and Filling in of Missing Information Synthetic VEs provide fewer sensory cues than physical environments User needs to interpolate sensory stimuli to create a functional mental modal Result in an enhanced sense of presence

Faster Mental Processing Enables Deeper and Richer Experience Using Reaction Time to compare the brain processing speed of unimodal, bimodal, and trimodal signals Faster processing at the perceptual stage allow users more time in consequent cognitive stages Contributing to the great sense of presence

Experimental Design Materials – touch-enabled computer interface, haptic device(force-feedback) Participants – 16 students(unaware of the purpose of the experiments) Under the guidelines of the ethical committee and with its approval

Stimuli Holding the stylus in their hand as if they were writing On each trail the computer generated a sensory stimulation – uni, bi, trimodal Visual – changing color Auditory – compound sound haptic – resisting force

Procedure Pressing a button on the stylus as soon as they detected the stimuli Computer generated the stimulation randomly Consisted of six blocks of trails (three with dominant hand, three with the other hand) Each of blocks consisted of 105 single trail, in which each of the seven conditions

Result-Unimodal V – dominant hand 430ms (SD=94) not dominant 436ms (SD=92) A – dominant hand 330ms (SD=103) not dominant 326ms (SD=76) H – dominant hand 318ms (SD=99) not dominant 334ms (SD=91)

Result-Bimodal VA – dominant hand 302ms (SD=78) not dominant 304ms (SD=70) HV – dominant hand 294ms (SD=75) not dominant 306ms (SD=77) HA – dominant hand 272ms (SD=81) not dominant 280ms (SD=69)

Result-Trimodal dominant hand 430ms (SD=94) not dominant436ms (SD=92) Between-Hand comparisons - Revealed insignificant differences in the VA and HA conditions, and only marginal difference in HV conition

Multimodal Stimulation and Attention Multisensory enhancement is modulated by attention Hypothesizing that the brain allocates a greater amount of attention to multimodal and employing a larger neural network

Attention and Presence The link between attention and presence is already documented in literature

Conclusion At the initial perceptual stage – Multimodal information is processed faster Activating larger neural network, increasing user ’ s attention At the consequent cognitive stages, more time to Acquire a wider range of details and subtle for the display Fill in missing information Integrate all these informative cues Contributing to a greater sense of presence