Interaction Techniques for Common Tasks in Immersive Virtual Environments Design, Evaluation, and Application Doug A. Bowman April 27, 1998

Doug Bowman - VE Interaction Techniques2 Vision Immersive VEs for productivity Complex applications for real work Example: immersive modeling and design Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques3 Definitions Interaction Technique (IT): Method used to complete a task via a human-computer interface (hardware & software) Immersive VE: A real-time 3D synthetic environment that appears to surround the user in space HMD with head tracking, CAVE XFishtank VR, MUDs, Multimedia apps Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques4 A Brief History of VEs 1968: Sutherlands Ultimate Display Hardware advances –displays– trackers– 3D graphics –input devices– haptics– 3D audio Software advances –view culling– level of detail –VE toolkits– collision detection Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques5 VE Applications In Use: –architectural walkthrough –phobia treatment –games (e.g. 1st person shooter) Proposed: –information visualization and retrieval –modeling and design –constructivist education Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques6 Interaction: the Distinguishing Factor Current applications –may involve movement through VE –may involve shooting or pointing Proposed applications –require 3D navigation and selection –require 6 DOF manipulation (object placement) –require large command spaces Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques7 How to improve VE Interaction better design of techniques systematic evaluation (formative and summative) in the context of applications and requirements Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques8 Universal Tasks: Travel Viewpoint Motion Control: The users interactive control of the position and orientation of his viewpoint Wayfinding: Cognitive process of determining a route, using landmarks, maps, etc. Navigation: VMC + Wayfinding Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques9 Universal Tasks: Selection & Manipulation Selection: Specification of one or more objects from a set –as the object of a command –to begin manipulation Manipulation: Specification of the position, orientation, and/or scale of an object Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques10 Why not natural interaction? Term VR implies replication of real world Why not use well-developed human skills to accomplish tasks in VEs? –travel: walking or driving –selection & manipulation: grasp and place These mappings are intuitive, but too limited Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques11 Interaction Techniques and Input Devices input devices are only the hardware component of an IT input device does not determine IT many ITs can be implemented with a single input device we will not design or evaluate devices we will design and evaluate ITs for common VE input devices Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques12 Problem Statement: I will... analyze universal tasks and create taxonomies of techniques design new techniques based on these formal frameworks design, implement, and conduct formal evaluations of IT performance apply the results to a complex and useful VE application Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques13 Design and Evaluation Methodology Taxonomization and Categorization Guided Design Performance Measures Range of Evaluation Methods Testbed Evaluation Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques14 Taxonomization and Categorization Introduction Methodology Travel Selection/Manipulation Application Remaining Work Task Subtask Technique Component Task analysis Consider techniques for low-level subtasks Promotes deeper understanding of task Framework for design Framework for evaluation

Doug Bowman - VE Interaction Techniques15 Guided Design Design new techniques based on taxonomy, not simply intuition Choose a component for each low-level subtask Easy to see holes in design space Introduction Methodology Travel Selection/Manipulation Application Remaining Work Task Subtask Technique Component

Doug Bowman - VE Interaction Techniques16 Evaluation Methods Range of performance metrics (quantitative and qualitative; productivity and user-centric) Range of methods (user studies, usability evaluation, formal experiments) Consideration of outside factors (characteristics of task, environment, user, system that might affect performance) Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques17 Testbed Evaluation testbed: representative set of tasks and environments evaluate techniques for overall performance in a wide range of situations vary technique components and outside factors measure several performance variables generalizable and replicable Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques18 Summary of Methodology Introduction Methodology Travel Selection/Manipulation Application Remaining Work TaxonomiesPerf. MetricsOutside Factors speed accuracy comfort... environment density users reach task difficulty... Initial Evaluation and Design

Doug Bowman - VE Interaction Techniques19 Introduction Methodology Travel Selection/Manipulation Application Remaining Work TaxonomiesPerf. MetricsOutside Factors Applications speed accuracy comfort... environment density users reach task difficulty... requirements Initial Evaluation and Design Summary of Methodology

Doug Bowman - VE Interaction Techniques20 Introduction Methodology Travel Selection/Manipulation Application Remaining Work TaxonomiesPerf. MetricsOutside Factors TESTBED EVALUATION Applications speed accuracy comfort... environment density users reach task difficulty... requirements Initial Evaluation and Design Summary of Methodology

Doug Bowman - VE Interaction Techniques21 Introduction Methodology Travel Selection/Manipulation Application Remaining Work TaxonomiesPerf. MetricsOutside Factors TESTBED EVALUATION Performance Measurements/ Models Applications speed accuracy comfort... environment density users reach task difficulty... requirements choice of techniques Initial Evaluation and Design Summary of Methodology

Doug Bowman - VE Interaction Techniques22 Informal Evaluation based on observations –default gaze-directed steering –lack of published work based on our own applications –Conceptual Design Space –Virtual GIS Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques23 Initial Taxonomy Task: Move from the current location to the desired location Introduction Methodology Travel Selection/Manipulation Application Remaining Work Viewpoint Motion Control Direction/Target Selection Velocity/Acceleration Selection Conditions of Input gaze-directed pointing physical props gesture slow in, slow out physical props start/stop buttons automatic start/stop constant movement

Doug Bowman - VE Interaction Techniques24 Performance Measures Quantitative (e.g. speed, accuracy) Qualitative (e.g. presence) User-Centric (e.g. ease of use, comfort) Introduction Methodology Travel Selection/Manipulation Application Remaining Work ITApps Quality Factors -speed -accuracy -cognitive load -presence -spatial awareness -...

Doug Bowman - VE Interaction Techniques25 Simple Experiments (Bowman, Koller, and Hodges, VRAIS 97) Absolute Motion –no difference between gaze and pointing Relative Motion –pointing superior to gaze Spatial Awareness –teleportation causes disorientation –any continuous motion does not Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques26 Expanded Framework Absolute vs. Relative Motion –same techniques, different results –highlights need to consider outside factors Consider task, user, system, and environment characteristics Introduction Methodology Travel Selection/Manipulation Application Remaining Work Task Performance absolute relative gaze-directed steering pointing

Doug Bowman - VE Interaction Techniques27 Complex Experiment Does travel IT affect cognitive load? Task: gather as much info as possible Variables: »IT: gaze, pointing, torso »Environment: 1-, 2-, or 3- dimensional »System: collision detection Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques28 Guided Design Taxonomy: tour technique –environmental target selection –gesture-based velocity selection –explicit or automatic stop inputs Intuition: travel based on manipulation –cross-task technique –still fits in taxonomy Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques29 Final Framework and Testbed Rework taxonomy to be more general –task analysis: 2 basic position-setting methods are specifying destination, specifying trajectory –distinction allows better fitting of techniques VMC Testbed –still in design stage –based on evaluation framework Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques30 Initial Taxonomy Based on metaphor, not task Arm-extension metaphor –touch and place object with virtual hand –hand may extend beyond normal range Ray-casting metaphor –point at object to select –manipulate by attaching to virtual light ray Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques31 Informal Evaluation (Bowman and Hodges, I3DG 97) Studied six techniques (4 AE, 2 RC) Simple user study (comments, observations) Eleven subjects used techniques to place furniture in a room Results –AE excels at manip., RC better at selection –selection & manipulation should be separated Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques32 HOMER Techniques Hand-Centered Object Manipulation Extending Ray-Casting Hybrid technique Select: ray-casting Manipulate: v. hand Introduction Methodology Travel Selection/Manipulation Application Remaining Work Time

Doug Bowman - VE Interaction Techniques33 HOMER Techniques Hand-Centered Object Manipulation Extending Ray-Casting Hybrid technique Select: ray-casting Manipulate: v. hand Introduction Methodology Travel Selection/Manipulation Application Remaining Work Time

Doug Bowman - VE Interaction Techniques34 HOMER Techniques Hand-Centered Object Manipulation Extending Ray-Casting Hybrid technique Select: ray-casting Manipulate: v. hand Introduction Methodology Travel Selection/Manipulation Application Remaining Work Time

Doug Bowman - VE Interaction Techniques35 HOMER Techniques Hand-Centered Object Manipulation Extending Ray-Casting Hybrid technique Select: ray-casting Manipulate: v. hand Introduction Methodology Travel Selection/Manipulation Application Remaining Work Time

Doug Bowman - VE Interaction Techniques36 Formal Taxonomy Introduction Methodology Travel Selection/Manipulation Application Remaining Work Selection Manipulation Release Indication of Object Indication to Select Attachment Positioning Orientation Indication to Release Final Object Position/Orientation touch occlude button gesture hand moves to object user scales to touch object 1-to-1 hand motion mapping match tracker orientation indirect control button gesture remain in place use physics model

Doug Bowman - VE Interaction Techniques37 Evaluation Framework Performance measures similar to travel Important outside factors: –task characteristics: DOFs to manipulate –user characteristics: reach, spatial ability –system characteristics: constraints used Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques38 Guided Design testbed implemented to allow arbitrary combinations of technique components 4608 possible combinations - reduced to 667 via dependencies and constraints Taxonomy: gaze-based HOMER with separate positioning and orientation Intuition: manipulation based on travel (cross-task technique) Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques39 Selection/Manipulation Testbed Introduction Methodology Travel Selection/Manipulation Application Remaining Work Tasks that test all important aspects of a select/manip. IT Selection variables: distance, size, density Manip. variables: distance, accuracy, DOFs required

Doug Bowman - VE Interaction Techniques40 Application Case Study: Immersive Design Verify evaluation results in a complex VE application Design system involves all universal tasks Choose ITs based on testbed results and specified application requirements Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques41 Interaction Requirements Travel –exploration and goal-based movements –spatial awareness, info gathering, ease of use Selection –accuracy at a distance, speed, comfort Manipulation –expressibility, accuracy, ease of use Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques42 Three Levels of Interaction Design Naive design –taken from CDS application (in D. Bertol, Designing Digital Space) –gaze-directed steering, ray-casting Intuitive design iteration –current implementation –pen & tablet, pointing, Go-Go technique Final, systematic design Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques43 Remaining Work complete design and evaluation framework for travel design, implement, and run travel testbed complete and run selection/manipulation testbed modify application interaction design and verify with a usability study Introduction Methodology Travel Selection/Manipulation Application Remaining Work

Doug Bowman - VE Interaction Techniques44 Contributions formal understanding of tasks/techniques testbeds for future evaluations performance results and models new interaction techniques useful and usable immersive design application