WPI HIVE: Introduction to Virtual Reality Interesting Issues, Open Problems Prof. Robert W. Lindeman Worcester Polytechnic Institute Department of Computer Science

R.W. Lindeman - WPI Dept. of Computer Science2 Overview  Administrative Stuff  What is Virtual Reality?  Why study Interaction in VR?  Open Problems

R.W. Lindeman - WPI Dept. of Computer Science3 Motivation  Much excitement (and hype) about how VR was going to change things VR has not made inroads into everyday life  Lagging technology  Lack of understanding of usability issues  Lack of “killer app” Still remains mainly in research labs Video games show great promise Training scenarios - surgery, military, therapy  Long-Term Goal Make VR more usable

R.W. Lindeman - WPI Dept. of Computer Science4 Nothing New? SENSORAMA by Morton Heilig (1960)

R.W. Lindeman - WPI Dept. of Computer Science5 Background  VR defined: Fooling the senses into believing they are experiencing something they are not actually experiencing  Virtual reality systems consist of: Graphical/audio/haptic/... rendering Content Tracking of people and objects Collision detection Interaction techniques Optional, but common:  Networking  Autonomous agents

R.W. Lindeman - WPI Dept. of Computer Science6 Keys to Success  High fidelity (or realism) Graphics, audio, haptics, behaviors, etc.  Low latency Tracking Collision detection Rendering Networking  Ease of use Low cumber for users Easy integration for programmers  Compelling Content

R.W. Lindeman - WPI Dept. of Computer Science7 The Senses  See (Visual Sense): Visuals are excellent!  Hear (Aural Sense): Spatialized audio is very good!  Smell (Olfactory Sense): Very hard! Too many types of receptors.  Touch (Haptic Sense): Application specific and cumbersome  Taste (Gustatory Sense): We know the base tastes, but that is it!

R.W. Lindeman - WPI Dept. of Computer Science8 See: Head-Mounted Displays

R.W. Lindeman - WPI Dept. of Computer Science9 See: Projection-Based Environments

R.W. Lindeman - WPI Dept. of Computer Science10 See: Projection-Based Environments (cont.)

R.W. Lindeman - WPI Dept. of Computer Science11 Hear: Sound in VR  Display techniques Multi-speaker output (sound cube) Headphones Bone-Conduction  Waveform filtering Simple balance & volume control Head-Related Transfer Functions  Software "Standards" OpenAL A3D from Aureal (RIP!) VRSonic.com

R.W. Lindeman - WPI Dept. of Computer Science12 Smell: Olfactory Sense  Two main problems Scent generation  Tens of thousands of receptor types Scent delivery  Easier problem

R.W. Lindeman - WPI Dept. of Computer Science13 Smell: Air Cannon (Yanagida, 2004)

R.W. Lindeman - WPI Dept. of Computer Science14 Touch: Haptic Feedback in VR  Tactile: Surface properties Most densely populated area is the fingertip (okay, it's the tongue)  Kinesthetic: Muscles, Tendons, etc. Also known as proprioception

R.W. Lindeman - WPI Dept. of Computer Science15 Vibrotactile Feedback Projects TactaBoard and TactaVest

R.W. Lindeman - WPI Dept. of Computer Science16 The TactaBoard 2.1

R.W. Lindeman - WPI Dept. of Computer Science17 Empirical Studies  TactaChair experiments Vibration location identification Priming for a visual search task Intensity matching  TactaVest experiments Exposure during room clearing tasks Spatial awareness Team member location for team training Robot tele-operation

R.W. Lindeman - WPI Dept. of Computer Science18 TactaChair Experiments

R.W. Lindeman - WPI Dept. of Computer Science19 Exposure Experiments  Looking at the use of spatialized vibrotactile feedback as a training aid on "victim" search tasks

R.W. Lindeman - WPI Dept. of Computer Science20 Exposure Experiments (cont.)  Use vibration to convey exposure  Results to appear in ACM CHI 2005

R.W. Lindeman - WPI Dept. of Computer Science21 Issues to be Addressed  Transfer effects from virtual to real environments How do subjects perform after training in VR?  Psychophysical issues Sensory substitution  Cognitive Issues Does the addition of haptic cues increase cognitive load?  Multi-modal integration

R.W. Lindeman - WPI Dept. of Computer Science22 Interaction in VR  Use of a keyboard and mouse is not tractable Can't see them Want to move around No good 3D mappings  How can we allow easy interaction that takes advantage of real-world experience? This is the problem that we need to solve!

R.W. Lindeman - WPI Dept. of Computer Science23 Basic Interaction Tasks in VR (Bowman et al.)  Object Selection What do I want to manipulate?  Object Manipulation How can I manipulate it?  Navigation Wayfinding: How do I know where I am, and how to get where I am going? Travel: How do I get there? (locomotion)  System Control How do I change system parameters?  Symbolic Input Inputting text and numbers

R.W. Lindeman - WPI Dept. of Computer Science24 Oh, I forgot One (Lindeman)  Killing

R.W. Lindeman - WPI Dept. of Computer Science25 Dealing with Objects  Problems Ambiguity Distance  Selection Approaches Direct / enhanced grabbing Ray-casting techniques Image-plane techniques  Manipulation Approaches Direct position / orientation control Worlds in miniature Skewers Surrogates Courtesy: D. Bowman

R.W. Lindeman - WPI Dept. of Computer Science26 Navigation: Wayfinding  People get lost/disoriented easily  Traditional tools Maps (North-up vs. Forward-up) Landmarks Spoken directions  Non-traditional Callouts Zooming Images:

R.W. Lindeman - WPI Dept. of Computer Science27 Navigation: Travel  Problems Limited physical space, unlimited virtual space Cables  Approaches Fly where you point/look Treadmills Walking in place Big track ball Image: CLIP

R.W. Lindeman - WPI Dept. of Computer Science28 System Control  Need to manipulate widgets Lighting effects Object representation Data filtering  Approaches Floating windows Hand-held windows Gestures Menus on fingers

R.W. Lindeman - WPI Dept. of Computer Science29 System Control Examples Courtesy: D. Bowman Courtesy: R. Lindeman

R.W. Lindeman - WPI Dept. of Computer Science30 User, Task & Environment  The "optimal" interface will depend on the capabilities of the user, the nature of the task being performed, and the constraints of the environment.  User Dexterity, level of expertise  Task Granularity and complexity of task  Environment Stationary, moving, noisy, etc.

R.W. Lindeman - WPI Dept. of Computer Science31 Primary Interface Tasks  Object Selection  Object Manipulation Direct Manipulation Indirect Manipulation  Navigation and Wayfinding  System Control  Symbolic Input  Can compare different techniques empirically

R.W. Lindeman - WPI Dept. of Computer Science32 Desktop Interaction: SensAble PHANToM

R.W. Lindeman - WPI Dept. of Computer Science33 Direct Manipulation Courtesy: Virginia Tech

R.W. Lindeman - WPI Dept. of Computer Science34 Can We Do WIMP in VR? Courtesy: Virginia Tech

R.W. Lindeman - WPI Dept. of Computer Science35 Wearable Interaction with Haptics: Immersion CyberGrasp

R.W. Lindeman - WPI Dept. of Computer Science36 Wearable Interaction: Rob's Hand-Held Windows

R.W. Lindeman - WPI Dept. of Computer Science37 How Do We Do Menus? Courtesy: Virginia Tech

R.W. Lindeman - WPI Dept. of Computer Science38 Interface Devices Courtesy: Virginia Tech

R.W. Lindeman - WPI Dept. of Computer Science39 Augmented Reality (AR) Courtesy: Virginia Tech

R.W. Lindeman - WPI Dept. of Computer Science40 Applications  Data perceptualization Map variables to tactors  Spatial awareness Driver warning system (vibrotactile Bott's dots)  Navigational aid Firefighter guidance  Non-verbal communication Map hand signals to vibrotactile patterns

R.W. Lindeman - WPI Dept. of Computer Science41 Summary of Research Activities  Human-Computer Interaction Usability studies  Computer Graphics  Virtual Reality and Gaming Interesting input devices Haptic/tactile feedback Multi-modal interaction and integration  Pervasive and Wearable Computing  Robot Tele-Operation

R.W. Lindeman - WPI Dept. of Computer Science42 Open Problems in VR  Tracking sucks! Optical, magnetic, ultrasonic, inertial, hybrid  Integration of sensory stimuli  Good combination of direct and indirect user interaction  User studies! Tough to design Easy to get subjects

R.W. Lindeman - WPI Dept. of Computer Science43 More Info   