1. Interaction in Virtual Environments Benjamin Lok This Lecture contains notes created by Doug Bowman Virginia Tech Sherman & Craig, pp. 332-363

2. 3D Interfaces We live in a 3D world We live in a 3D world Natural interfaces are better Natural interfaces are better Therefore 3D interfaces will be the ultimate Therefore 3D interfaces will be the ultimate What’s wrong with the above? What’s wrong with the above? Natural interfaces aren’t always better! Natural interfaces aren’t always better! Making the interface simple (thus unnatural) often aids performance Making the interface simple (thus unnatural) often aids performance Constrains movement Constrains movement Limiting possible actions Limiting possible actions Depends on application and goal of the user interface Depends on application and goal of the user interface Surgery simulation Surgery simulation Military simulation (general vs. soldier training) Military simulation (general vs. soldier training) Architecture, education, product design Architecture, education, product design Video games Video games

3. 3D Interfaces What we really want are enhanced interfaces What we really want are enhanced interfaces Give us powers we don’t normally have Give us powers we don’t normally have Flying, x-ray vision, teleportation, undo, etc. Flying, x-ray vision, teleportation, undo, etc. Be careful we don’t become overzealous Be careful we don’t become overzealous Air traffic control 3D display Air traffic control 3D display Library interfaces using a books on shelves (what is it good for? What is it poor for?) Library interfaces using a books on shelves (what is it good for? What is it poor for?) Hurts performance Hurts performance Study results: 3D Bar charts don’t help Study results: 3D Bar charts don’t help So what is helped by 3D? So what is helped by 3D?

4. Good 3D Social interfaces + 3D can be very powerful Social interfaces + 3D can be very powerful MMORPG (EveQuest) MMORPG (EveQuest) ActivedWorlds ActivedWorlds The Sims Online The Sims Online Experiences Experiences Art gallary Art gallary 3D Desktops (Mac’s latest) 3D Desktops (Mac’s latest) Office metaphors did not take off (BOB, Task Gallary) Office metaphors did not take off (BOB, Task Gallary) 3D Webbrowsing. Sure you can arrange 16 web pages spatially, but why? 3D Webbrowsing. Sure you can arrange 16 web pages spatially, but why? Compromises to provide 3D interfaces might be undermine usability Compromises to provide 3D interfaces might be undermine usability Think RTS games Think RTS games Discussion: Is the interface holding back 3D? Discussion: Is the interface holding back 3D?

5. 3D Interfaces Use occlusion, shadows, perspective carefully Use occlusion, shadows, perspective carefully Improves use of spatial memory (Ark ’98) Improves use of spatial memory (Ark ’98) Distracting and confusing Distracting and confusing Minimize navigation steps Minimize navigation steps Keep text readable (good contrast, 30 degree tilt max) Keep text readable (good contrast, 30 degree tilt max) Simple user movement (why lock to a floor?) Descent vs Quake Simple user movement (why lock to a floor?) Descent vs Quake Prevent Errors (put in guides to help) Prevent Errors (put in guides to help) Simplify object movement (connecting two parts, for example, can be abstracted… most of the time) Simplify object movement (connecting two parts, for example, can be abstracted… most of the time) Organize groups of items into alignments that facilitate visual search and recall (allow user choice) Organize groups of items into alignments that facilitate visual search and recall (allow user choice)

6. 3D Interface Development Developments that show promise: Developments that show promise: 3D sound 3D sound Stereo display (Ware and Frank ’96) Stereo display (Ware and Frank ’96) Haptic feedback (mouse) Haptic feedback (mouse) 3D can help by: 3D can help by: Provide overviews to see big picture Provide overviews to see big picture Rapid teleportation (context shifts) Rapid teleportation (context shifts) Zooming (aid disabled) Zooming (aid disabled) Multiple coordinated views (3dsmax) Multiple coordinated views (3dsmax) 3D icons can represent abstract or recognizable concepts 3D icons can represent abstract or recognizable concepts

7. Teleoperation Combines: Combines: Direct Manipulation Direct Manipulation Process Control Process Control Human operators control physical processes in complex environments Human operators control physical processes in complex environments Example applications: Mars rover control, flying airplanes (Predator), manufacturing, medicine (surgery) Example applications: Mars rover control, flying airplanes (Predator), manufacturing, medicine (surgery) Supervisory control (Sheridan ’92) Supervisory control (Sheridan ’92) Different levels of human control (automation) Different levels of human control (automation)

8. Teleoperation Direct Manipulation Issues Direct Manipulation Issues Adequate feedback (data quality, latency (transmission and operation delays), incomplete, interference) Adequate feedback (data quality, latency (transmission and operation delays), incomplete, interference) Presence Presence Point and click or more natural interaction vs. typing Point and click or more natural interaction vs. typing Example project: Nanomanipulator (show video) Example project: Nanomanipulator (show video)

9. VR Interaction Trying to simulate reality or an experience Trying to simulate reality or an experience Training, Learning, Exploring Training, Learning, Exploring Expensive Expensive Dangerous Dangerous Logistically Difficult Logistically Difficult Best interaction? Best interaction? Flight simulators (they can cost $10 mil, but that’s still a good deal!) Flight simulators (they can cost $10 mil, but that’s still a good deal!) Why? Why? Why do video game flight sims not cut it? (only $40!) Why do video game flight sims not cut it? (only $40!) Okay, we have monitors that show 3D worlds, what else do we possibly need? Okay, we have monitors that show 3D worlds, what else do we possibly need?

10. Why 3D interaction? 3D / VE apps. should be useful 3D / VE apps. should be useful Immersion Immersion Natural skills Natural skills Immediacy of visualization Immediacy of visualization But, current VE apps are But, current VE apps are Not complex interactively Not complex interactively Have serious usability problems Have serious usability problems

11. What makes 3D interaction difficult? Spatial input Spatial input Lack of constraints Lack of constraints Lack of standards Lack of standards Lack of tools Lack of tools Lack of precision Fatigue Layout more complex Perception You’ve seen this in creating Project #2!

12. Let’s look at this example (ISAAC)

13. Universal interaction tasks Navigation Navigation Travel – motor component Travel – motor component Wayfinding – cognitive component Wayfinding – cognitive component Selection Selection Manipulation Manipulation System control System control

14. Goals of interaction design Performance (efficiency, accuracy, productivity) Performance (efficiency, accuracy, productivity) Usability (ease of use, ease of learning, user comfort) Usability (ease of use, ease of learning, user comfort) Usefulness (users focus on tasks, interaction helps users meet system goals) Usefulness (users focus on tasks, interaction helps users meet system goals)

15. Philosophies of interaction design Artistic approach Artistic approach Intuition about users, tasks Intuition about users, tasks Heuristics, metaphors Heuristics, metaphors Aesthetics Aesthetics Adaptation Adaptation Scientific approach Formal analysis Formal evaluation Performance requirements

16. Travel Motor component of navigation Motor component of navigation Movement between 2 locations, setting the position (and orientation) of the user’s viewpoint Movement between 2 locations, setting the position (and orientation) of the user’s viewpoint Most basic and common VE interaction technique Most basic and common VE interaction technique Used in almost any large- scale VE Used in almost any large- scale VE

17. Types of Travel Tasks Exploration Exploration Travel which has no specific target Travel which has no specific target Build knowledge of environment Build knowledge of environment Search Search Naive: travel to find a target whose position is not known Naive: travel to find a target whose position is not known Primed: travel to a target whose position is known Primed: travel to a target whose position is known Build layout knowledge Build layout knowledge Move to task location Move to task location Maneuvering Maneuvering Travel to position viewpoint for task Travel to position viewpoint for task Short, precise movements Short, precise movements

18. Steering metaphor Continuous specification of direction of motion Continuous specification of direction of motion Gaze-directed Gaze-directed Pointing Pointing Physical device (steering wheel, flight stick) Physical device (steering wheel, flight stick)

19. Target-based metaphor Discrete specification of goal Discrete specification of goal Point at object Point at object Choose from list Choose from list Enter coordinates Enter coordinates E.g. Google Earth E.g. Google Earth

20. Route-planning metaphor One time specification of path One time specification of path Place markers in world Place markers in world Move icon on map Move icon on map Manual manipulation of viewpoint Manual manipulation of viewpoint “Camera in hand” “Camera in hand” Fixed object manipulation Fixed object manipulation

21. “Natural” travel metaphors Walking techniques Walking techniques Treadmills Treadmills Bicycles Bicycles Other physical motion Other physical motion VMC / Magic carpet VMC / Magic carpet Disney’s river raft ride Disney’s river raft ride Simulation of flying Simulation of flying

22. Technique classification Travel Start to move Stop moving Indicate position Indicate orientation position velocity acceleration Target specification Route specification Continuous specification

23. Alternate Technique classification Travel Direction/Target Selection Velocity/Accel. Selection Conditions of Input gaze-directed pointing physical props gesture slow in, slow out physical props start/stop buttons automatic start/stop constant movement Ontologies by Bowman99 and Hand97

24. Evaluation results Steering techniques have similar performance on absolute motion tasks Steering techniques have similar performance on absolute motion tasks Non-head-coupled steering better for relative motion Non-head-coupled steering better for relative motion “Teleportation” can lead to significant disorientation “Teleportation” can lead to significant disorientation Environment complexity affects information gathering Environment complexity affects information gathering Travel interaction technique and user’s strategies affect spatial orientation Travel interaction technique and user’s strategies affect spatial orientation

25. Evaluation results – 3 Manipulation-based techniques efficient for relative motion Manipulation-based techniques efficient for relative motion Manipulation-based techniques not requiring an object efficient for search, but tiring Manipulation-based techniques not requiring an object efficient for search, but tiring Steering techniques best for naïve and primed search Steering techniques best for naïve and primed search Map-based techniques not effective in unfamiliar environments, or when any precision is required Map-based techniques not effective in unfamiliar environments, or when any precision is required

26. Myths There is one optimal travel technique for VEs. There is one optimal travel technique for VEs. A “natural” technique will always be better than another technique. A “natural” technique will always be better than another technique. Desktop 3D, workbench, and CAVE applications should use the same travel ITs as HMD-based VEs. Desktop 3D, workbench, and CAVE applications should use the same travel ITs as HMD-based VEs.

27. Design guidelines Make simple travel tasks simple (target-based techniques for motion to an object, steering techniques for search). Make simple travel tasks simple (target-based techniques for motion to an object, steering techniques for search). Provide multiple travel techniques to support different travel tasks in the same application. Provide multiple travel techniques to support different travel tasks in the same application.

28. More design guidelines Use transitional motions if overall environment context is important. Use transitional motions if overall environment context is important. Train users in sophisticated strategies to help them acquire survey knowledge. Train users in sophisticated strategies to help them acquire survey knowledge. Consider integrated (“cross-task”) ITs if travel is used in the context of another task (e.g. manipulation). Consider integrated (“cross-task”) ITs if travel is used in the context of another task (e.g. manipulation).

29. Wayfinding Cognitive process of defining a path through an environment using and acquiring spatial knowledge Cognitive process of defining a path through an environment using and acquiring spatial knowledge 6DOF makes wayfinding hard 6DOF makes wayfinding hard Human beings have different abilities to orient themselves in an environment Human beings have different abilities to orient themselves in an environment Observing wayfinding as a decision making process Observing wayfinding as a decision making process

30. Wayfinding tasks General, explorative search Search without target General, explorative search Search without target Naïve search target position unknown Naïve search target position unknown Primed search target seen before (known) Primed search target seen before (known) Specified trajectory movement Predefined path Specified trajectory movement Predefined path

31. Cognitive Map During wayfinding, a person makes use of three kinds of knowledge to built up a cognitive map of the environment: During wayfinding, a person makes use of three kinds of knowledge to built up a cognitive map of the environment:  Landmark knowledge  Procedural knowledge  Survey knowledge

32. Reference frames Egocentric reference frame: position, orientation, movement of object with respect to position and orientation of the: Egocentric reference frame: position, orientation, movement of object with respect to position and orientation of the: eyes eyes head head body body Exocentric reference frame: position, orientation and movement are defined in coordinates external to body Exocentric reference frame: position, orientation and movement are defined in coordinates external to body object shape object shape object orientation object orientation object motion object motion

33. Travel technique effects Steering technique with good strategy helps spatial orientation Steering technique with good strategy helps spatial orientation A good travel technique will integrate aids to wayfinding A good travel technique will integrate aids to wayfinding Jumping between points disturbs spatial orientation Jumping between points disturbs spatial orientation

34. Support of spatial knowledge acquisition Allow a wide field of view Allow a wide field of view Provide motion cues for judging depth and direction of movement Provide motion cues for judging depth and direction of movement Audio could enhance visual spatial perception Audio could enhance visual spatial perception Support sense of presence: it could strengthen the construction of a cognitive map Support sense of presence: it could strengthen the construction of a cognitive map

35. Support of spatial knowledge acquisition Design legible environments - allow the user to easily see the spatial organisation of an environment, enabling the establishment of a cognitive map. Design legible environments - allow the user to easily see the spatial organisation of an environment, enabling the establishment of a cognitive map.  Divide a large-scale environment into parts with a distinct character

36. Support of spatial knowledge acquisition  Create a simple spatial organisation in which the relations between the parts are clear  Support the matching process between the egocentric and exocentric frames of reference by (visual) cues, including directional cues

37. Support of Spatial Knowledge Acquisition Use real-world wayfinding principles to build up your environment Use real-world wayfinding principles to build up your environment Natural environment principles Natural environment principles Urban design principles [Lynch] Urban design principles [Lynch] Architectural design principles Architectural design principles Artificial cues Artificial cues

38. Examples of wayfinding aids

39. Map usage guidelines Provide you are here marker Provide you are here marker Provide grid Provide grid Choose either north-up or forward-up map, depending on task Choose either north-up or forward-up map, depending on task Example: World-in-Miniature Example: World-in-Miniature

40. Manipulation Techniques Wand-based Wand-based Go-Go Go-Go Ray Casting Ray Casting HOMER (Hand- centered Object Manipulation Extending Ray- casting) HOMER (Hand- centered Object Manipulation Extending Ray- casting)

41. Two Handed and Body- Centered Interaction What can you do with two hands? What can you do with two hands? What if you use your body as a reference point? What if you use your body as a reference point? Mine, Mark, Frederick P. Brooks Jr., and Carlo Sequin (1997). Moving Objects in Space: Exploiting Proprioception in Virtual-Environment Interaction. Proceedings of SIGGRAPH 97, Los Angeles, CA. (133K pdf version) Mine, Mark, Frederick P. Brooks Jr., and Carlo Sequin (1997). Moving Objects in Space: Exploiting Proprioception in Virtual-Environment Interaction. Proceedings of SIGGRAPH 97, Los Angeles, CA. (133K pdf version)Moving Objects in Space: Exploiting Proprioception in Virtual-Environment Interaction.(133K pdf version)Moving Objects in Space: Exploiting Proprioception in Virtual-Environment Interaction.(133K pdf version)

42. What can you do with two hands? What can you do with two hands? What if you use your body as a reference point? What if you use your body as a reference point? Mine, Mark, Frederick P. Brooks Jr., and Carlo Sequin (1997). Moving Objects in Space: Exploiting Proprioception in Virtual-Environment Interaction. Proceedings of SIGGRAPH 97, Los Angeles, CA. (133K pdf version) Mine, Mark, Frederick P. Brooks Jr., and Carlo Sequin (1997). Moving Objects in Space: Exploiting Proprioception in Virtual-Environment Interaction. Proceedings of SIGGRAPH 97, Los Angeles, CA. (133K pdf version)Moving Objects in Space: Exploiting Proprioception in Virtual-Environment Interaction. (133K pdf version)Moving Objects in Space: Exploiting Proprioception in Virtual-Environment Interaction. (133K pdf version) Two Handed and Body- Centered Interaction

43. Other Manipulation Approaches Tablet/hand-held Tablet/hand-held Pinch Glove Pinch Glove