Virtual Worlds: Audio and Other Senses

VR Worlds: Output Overview Visual Displays: –Visual depth cues –Properties –Kinds: monitor, projection, head-based, handheld Aural Displays: –Aural cues –Properties, including spatialization –Kinds: headphones, speakers Haptic (touch) Displays: –Properties –Kinds Other Displays: vestibular, olfactory

Audio Displays: Human Perception Outer ear (inc the pinna) collects sound as air vibrations, converts them to mechanical vibrations in the middle ear (vibrations of the tympanic membrane produce vibrations in the 3 bones called the ossicles) which transmit vibrations to the inner ear’s cochlea (filled with fluid, motion of the fluid causes the hairs of the basilar membrane to vibrate) and then sent via nerve impulses to the brain

Audio Displays: Sound Sound is the propagation of waves (pressure variations) through a medium; humans hear sounds from 20 Hz (cycles per second) to 20 kHz; low frequency are deeper Sound properties include amplitude, frequency (rate at which the pressure varies) and phase (where they are in time), spectral components which determine timbre (type or quality) Sounds have a temporal component; there can be masking Pitch and loudness are perceptions related to the sound properties (frequency and amplitude); for example, the A that orchestras tune to is 440 Hz Humans can do auditory scene analysis (grouping into discrete objects) and auditory stream segregation (isolating a sequence of sounds as one event) and cocktail party effect Localization of sound: ability to pinpoint the source

Questions What issues are important for localized sound? When would it be important to have spatialization of sound?

Cues for Localization of Sound Interaural level difference (ILD): different volumes reach each ear; better for high frequencies where the head provides interference Interaural time difference (ITD): same sound reaches the ears at different times; better for low frequencies (wavelength is large relative to head size) Called duplex theory; one problem is the cone of confusion where ITD is the same; head movement can help

Cues for Localization of Sound (con’t) People with hearing loss in one ear can localize so there are other factors Doppler effects: intensity increases means getting closer; intensity decreases = receding Reverberation/reflection Acoustic characteristics of speech Pinna filtering and head-related transfer functions (HTRF)

Head-Related Transfer Functions: HRTF Pinna filtering (outer ear): distorts incoming sounds; depends on frequencies and position of sound Experiments done: microphones to measure the changes that happen from different locations (~30 years ago) Produced mathematical functions to change the sound “Trick” the ear into localization Should be individualized but can be general; not as good when sound is behind or in low frequency range

Auralization: simulation of sounds Room or environment has acoustic properties (RIR: room impulse response); uses wave- based modeling and geometric modeling (sound is traced) Acoustic environment and the listener together are the BRIR (binaural room impulse response): generally done by combining the RIR and the HTRF, with the majority falling on the HTRF Generally use standard HTRF; doesn’t take into account different size pinna, diff ways to measure HTRF, perturbation of microphones, changes in head position Another techniques is to have arrays of microphones to reproduce sounds

Aural Presentation Properties Less computationally intensive than CG Number of channels: 1, 2 or more Sound stage: world referenced (loudspeakers) or head-referenced (headphones) Localization: HRFT, can simulate interaural differences, reverb Masking Amplification

Aural Displays Headphones: immersive, cables, HRTF, tracking needed Headphones can isolate listener but cumbersome and only good for one person and sounds originate “inside the head” Loudspeakers: environment, masking by projection screens, world-referenced, good for more people Loudspeakers: more difficult to get diff signals for diff ears, amplitude panning to simulate ILD (volume) Loudspeakers with wave field synthesis; large number of closely-packed loudspeakers

Aural Logistic Properties Noise pollution User mobility because of cables Interference from tracking equipment Environment needs: reflections Combination with other displays (visual, haptic) Portability Throughput: number of people, time to change headphones Encumbrance Safety Cost

Vestibular and Other Senses Vestibular sense: inner ear – to sense equilibrium, acceleration, gravity –Flight simulators, motion platforms, shaking, low-gravity –Issues with nausea, dizziness Olfactory: difficult to describe, individual variance Taste?

Olfactory Sense Hardware to generate odors: inkjet, solenoid valves, mass-flow controllers, autosamplers- blending of odor components Odor concentration Odor duration and strength: continuous air flow, air pump, headsets with nose interface Odor sensing an individual differences

Sources Understanding Virtual Reality by Sherman & Craig, Morgan Kaufman, 2003 Computer Graphics and Virtual Environments by Slater et al Virtual Audio Systems by Kapralos et al, Presence, Dec 2008 Cooking up an Olfactory Game Display by Nakamoto et al, IEEE Compputer Graphics and Applications, 2008