1. Immersive Displays The other senses…

2. 1962…

3. Classic Human Sensory Systems Sight (Visual) Hearing (Aural) Touch (Tactile) Smell (Olfactory) Taste (Gustatory)

4. Relevance to VR #1 – Sight #2 – Hearing #3 – Touch #4 – Smell #5 – Taste 1,2,3 are well studied but still have plenty of research left 4 and 5 are incredibly difficult, but some examples exist

5. Other relevant sensors Temperature Sensors Proprioceptive sensors (gravity) Stretch sensors found in muscles, skin, and joints Vestibular (inner ear) sensors Which can we control in VR? ◦ Cue conflicts cause nausea, vomiting

6. Audio (Sound Rendering) Easiest way to improve a VR system ◦ Think of watching a movie without sound Easy to use (Sound APIs) Cheap to produce great results (headphones) <$100

7. Audio Displays An arrangement of speakers ◦ Spatially Fixed – Loudspeakers (many types) ◦ Head-Mounted – Headphones (many types) Speaker quality affects your ability to generate sound wave frequencies, loudness ◦ Amplifiers very important for good results

8. Immersive Audio Our hearing system can sense the 3D source of a sound ◦ A VR system should be able to produce what the ears should hear from a 3D source Binaural recordings in real life (like stereoscopic video) 3D sound rendering in the virtual world (like stereoscopic rendering) ◦ Works best with headphones

9. Head Related Transfer Function (HRTF) In the frequency domain, at frequency f ◦ H(f) = Output (f) / Input (f) HRTF is dependent on spatial position, X,Y,Z, or in the far field, direction. Complex HRTF caused by the Pinnae of the ears ◦ Unique to each person HRTF learned by each person from childhood to sense 3D source

10. 3D sound rendering In the API (what you program) ◦ position, velocity, intensity of source ◦ position, velocity, *orientation* of listener Dependent on your renderer capabilities ◦ HRTF of actual listener for best results  Measure with molds or in-ear microphones  Default HRTF is identity (basically you only get left- right distinction) ◦ Reverb (echoing) or other effects ◦ Speaker arrangement (usually defined in OS)

11. Sound API OpenAL and DirectSound are popular ◦ Sort of like OpenGL and Direct3D API for talking to a 3D renderer (usually hardware) ◦ Similar to the idea of OpenGL Allows you to load sounds (utility toolkit), specify 3D sound properties, and specify listener properties. ◦ Must use single-channel sound files! Multi- channel sound files do not make sense. The renderer “generates” multi-channel sound. Example

12. Haptics (Touch rendering) Reproduction of forces exerted on the human body ◦ Striking a surface (e.g. hitting a ball) ◦ Holding an object ◦ Texture of a surface Lack of touch rendering is the #1 problem in VR systems ◦ Enormous actuation area  The entire surface of the human body Existing solutions are encumbering and task specific

13. Categories of Haptic Displays Passive vs Active ◦ Passive – Can stop motion but cannot create it ◦ Active – Can generate motion Fixed vs Sourceless ◦ Fixed – Mounted to the environment (e.g. a wand) ◦ Sourceless – Mounted to the user (e.g. a glove) Forces, torques, vibrations ◦ Types of output a Haptic device can be capable of

14. Haptic Rendering Specify forces, torques, rotations at actuation points ◦ Most commonly one APIs are available ◦ From manufacturer ◦ OpenHL? Very similar to physics rendering, except much more difficult ◦ Requires extremely high update rates (1000hz for imperceptibility)