Selected Topics in 3D User Interfaces Joseph J. LaViola Jr. CS March 6, 2006

Lecture Outline 2D/3D interfaces 2D/3D interfaces Nonisomorphic interaction Nonisomorphic interaction

Lecture Outline 2D/3D interfaces 2D/3D interfaces Nonisomorphic interaction Nonisomorphic interaction

2D Interaction Advantages Advantages provides a sense of feedback provides a sense of feedback very accurate very accurate some operations that are 3D in nature are more easily done with a 2D input device (e.g. object selection) some operations that are 3D in nature are more easily done with a 2D input device (e.g. object selection) picking objects is much easier in two dimensions picking objects is much easier in two dimensions Limitations Limitations manipulating 3D objects manipulating 3D objects have to add 3rd dimension in unconventional and unnatural ways have to add 3rd dimension in unconventional and unnatural ways WIMP WIMP

3D Interaction Advantages Advantages more natural for object manipulation once the object is taken more natural for object manipulation once the object is taken take advantage of 3D hand gestures and postures take advantage of 3D hand gestures and postures stereoscopic vision stereoscopic vision Limitations Limitations very difficult to write and annotate very difficult to write and annotate difficult to pick and place objects accurately difficult to pick and place objects accurately

Bringing 2D and 3D Together Goal: Let’s take the advantages from each type of interaction and bring them together to form a more usable interface Goal: Let’s take the advantages from each type of interaction and bring them together to form a more usable interface Broaden the application space Broaden the application space

Seamless Integration Critical component Critical component Requires both physical and logical integration Requires both physical and logical integration Do not want the user to work hard to change modes Do not want the user to work hard to change modes Tools should know what interaction technique they are used for Tools should know what interaction technique they are used for a device should know whether it is used for 3D interaction or 2D interaction based on context a device should know whether it is used for 3D interaction or 2D interaction based on context

2D/3D Interface Taxonomy Based on display surface interaction Based on display surface interaction Taxonomy Taxonomy direct direct hand-held indirect hand-held indirect hand-held direct hand-held direct

Direct Display - ErgoDesk 3D modeling application 3D modeling application 2D interaction on display surface 2D interaction on display surface Based on Sketch Based on Sketch Allows users to create, edit, view and manipulate 3D models Allows users to create, edit, view and manipulate 3D models

Hand-Held Indirect (1): Virtual Notepad Tool for writing in immersive environments Tool for writing in immersive environments Allows users to take notes and annotate documents Allows users to take notes and annotate documents

Hand-Held Indirect (2): Transparent Pad Transparent prop for the Virtual Table Transparent prop for the Virtual Table tool and object palette tool and object palette window tools window tools through-the-plane tool through-the-plane tool volumetric manipulation volumetric manipulation

Hand-Held Direct Displays PDA’s in Immersive VEs PDA’s in Immersive VEs Watsen used PalmPilot in a CAVE-like device [IPT99] Watsen used PalmPilot in a CAVE-like device [IPT99] provides camera, environment, and geometry controls provides camera, environment, and geometry controls Wacom Tablet in the TAN-Cube Wacom Tablet in the TAN-Cube too heavy too heavy wires got in the way wires got in the way has potential has potential

Going Beyond the 2D/3D Taxonomy Go beyond the 2D surface and hand approach Go beyond the 2D surface and hand approach Utilize traditional 2D concepts and extend to 3D interfaces Utilize traditional 2D concepts and extend to 3D interfaces Step WIM – based on maps Step WIM – based on maps TULIP – based on 2D menus TULIP – based on 2D menus

The Step WIM Miniature version of the world placed on the floor Miniature version of the world placed on the floor Motivated by Pausch and Stoakley’s WIM Motivated by Pausch and Stoakley’s WIM Augmented roadmap Augmented roadmap Step WIM scales up around users feet Step WIM scales up around users feet Operations Operations invoking invoking navigating navigating dismissing dismissing scaling scaling

Foot-based Interface Toe and heel tapping Toe and heel tapping “no place like home” metaphor “no place like home” metaphor Developed interaction slippers Developed interaction slippers Disambiguation of navigate and dismissal Disambiguation of navigate and dismissal based on user gaze based on user gaze derived from pilot studies derived from pilot studies

Body Gesture Interface More fluid gesture/less invasive device More fluid gesture/less invasive device Use waist tracker to detect upward bouncing gestures Use waist tracker to detect upward bouncing gestures Algorithm Algorithm first get user’s initial waist height first get user’s initial waist height monitor the waist tracker’s position monitor the waist tracker’s position check to see if the waist is above a height delta for a given amount of time check to see if the waist is above a height delta for a given amount of time

Step WIM Scaling VEs may be too large to fit within user’s walking area VEs may be too large to fit within user’s walking area Scaling implicitly provides different levels of detail Scaling implicitly provides different levels of detail

Foot-based Scaling Heel click toggles Step WIM scaling mode Heel click toggles Step WIM scaling mode Center of scale is user’s initial “location” in WIM Center of scale is user’s initial “location” in WIM maintain position within the WIM maintain position within the WIM Walking forward – closer look at the world Walking forward – closer look at the world Step WIM grows larger Step WIM grows larger Walking backward – gain perspective Walking backward – gain perspective Step WIM grows smaller Step WIM grows smaller

Body Gesture Scaling Avoid cue conflict of “walking in place” Avoid cue conflict of “walking in place” Holding a crouching gesture increases Step WIM size Holding a crouching gesture increases Step WIM size Holding a bouncing gesture decreases Step WIM size Holding a bouncing gesture decreases Step WIM size Center of scaling is projection of user’s waist Center of scaling is projection of user’s waist Gestures must be held longer than the bounce time threshold Gestures must be held longer than the bounce time threshold distinguishes between scaling and activation/dismissal distinguishes between scaling and activation/dismissal

TULIP – Three Up Labels in Palm Menu system using Pinch gloves Menu system using Pinch gloves Derived from a number of iterations Derived from a number of iterations Non-dominant hand controls menus Non-dominant hand controls menus Dominant hand controls menu items Dominant hand controls menu items

TULIP – Evaluation Compared with pull-down and pen and “pen and tablet” menus Compared with pull-down and pen and “pen and tablet” menus “Pen and tablet” found to be faster “Pen and tablet” found to be faster Users preferred TULIP Users preferred TULIP TULIP had higher comfort level TULIP had higher comfort level

Lecture Outline 2D/3D interfaces 2D/3D interfaces Nonisomorphic interaction Nonisomorphic interaction

Isomorphic vs. Non-Isomorphic Philosophies Human-Machine interaction Human-Machine interaction input device input device display device display device transfer function (control to display mapping) transfer function (control to display mapping) Isomorphic – one-to-one mapping Isomorphic – one-to-one mapping Non-isomorphic – scaled linear/non-linear mapping Non-isomorphic – scaled linear/non-linear mapping

Non-Isomorphic 3D Spatial Rotation Important advantages Important advantages manual control constrained by human anatomy manual control constrained by human anatomy more effective use of limited tracking range (i.e vision-based tracking) more effective use of limited tracking range (i.e vision-based tracking) additional tools for fine tuning interaction techniques additional tools for fine tuning interaction techniques Questions Questions faster? faster? more accurate? more accurate?

Rotational Space Rotations in 3D space are a little tricky Rotations in 3D space are a little tricky do not follow laws of Euclidian geometry do not follow laws of Euclidian geometry Space of rotations is not a vector space Space of rotations is not a vector space Represented as a closed and curved surface Represented as a closed and curved surface 4D sphere or manifold 4D sphere or manifold Quaternions provide a tool for describing this surface Quaternions provide a tool for describing this surface

Quaternions Four-dimensional vector (v,w) where v is a 3D vector and w is a real number Four-dimensional vector (v,w) where v is a 3D vector and w is a real number A quaternion of unit length can be used to represent a single rotation about a unit axis A quaternion of unit length can be used to represent a single rotation about a unit axis and angle as and angle as

Linear 0 th Order 3D Rotation Let be the orientation of the input device and be the displayed orientation then Let be the orientation of the input device and be the displayed orientation then Final equations w.r.t. identity or reference orientation are Final equations w.r.t. identity or reference orientation are

Non-Linear 0 th Order 3D Rotation Consider Consider Let k be a non-linear function as in Let k be a non-linear function as in

Design Considerations Absolute mapping – taken on i-th cycle of the simulation loop Absolute mapping – taken on i-th cycle of the simulation loop Relative mapping – taken between the i-th and i- 1th cycle of the simulation loop Relative mapping – taken between the i-th and i- 1th cycle of the simulation loop

Absolute Non-Isomorphic Mapping Generally do not preserve directional compliance Generally do not preserve directional compliance Strictly preserve nulling compliance Strictly preserve nulling compliance

Relative Non-Isomorphic Mapping Always maintain directional compliance Always maintain directional compliance Do not generally preserve nulling compliance Do not generally preserve nulling compliance

Experimental Usability Study Comparison of relative non-isomorphic rotation technique with conventional technique Comparison of relative non-isomorphic rotation technique with conventional technique Hypothesis Hypothesis rotation tasks will be faster with non-isomorphic approach for large rotations rotation tasks will be faster with non-isomorphic approach for large rotations moderate amplified rotations will decrease accuracy moderate amplified rotations will decrease accuracy Results Results subjects performed 13% faster with non-isomorphic approach with no accuracy degradation subjects performed 13% faster with non-isomorphic approach with no accuracy degradation

Amplified Non-Linear Rotation for VE Navigation (1) Users expect the virtual world to exist in any direction Users expect the virtual world to exist in any direction 3-walled Cave does not allow this 3-walled Cave does not allow this adapt expected UI to work in restricted environment adapt expected UI to work in restricted environment Amplified rotation allows users to see a full 360 degrees in a 3-walled display Amplified rotation allows users to see a full 360 degrees in a 3-walled display A number of approaches were tested A number of approaches were tested important to take cybersickness into account important to take cybersickness into account

Amplified Non-Linear Rotation for VE Navigation (2) Apply a non-linear mapping function to the user’s waist orientation and his or her distance from the back of the Cave Apply a non-linear mapping function to the user’s waist orientation and his or her distance from the back of the Cave Calculate the rotation factor using a scaled 2D Gaussian function Calculate the rotation factor using a scaled 2D Gaussian function The new viewing angle is The new viewing angle is

Amplified Non-Linear Rotation for VE Navigation (3)

Non-Linear Translation for VE Navigation (1) Users lean about the waist to move small to medium distances Users lean about the waist to move small to medium distances users can lean and look in different directions users can lean and look in different directions Users can also lean to translate a floor-based interactive world in miniature (WIM) Users can also lean to translate a floor-based interactive world in miniature (WIM) Step WIM must be active Step WIM must be active user’s gaze must be 25 degrees below horizontal user’s gaze must be 25 degrees below horizontal

Non-Linear Translation for VE Navigation (2) Leaning vector is the projection of the vector between the waist and the head onto the floor Leaning vector is the projection of the vector between the waist and the head onto the floor gives direction and raw magnitude components gives direction and raw magnitude components Navigation speed is dependent on the user’s physical location Navigation speed is dependent on the user’s physical location Leaning sensitivity increases close to a boundary Leaning sensitivity increases close to a boundary Linear function - Linear function - Mapped velocity - Mapped velocity -

Non-Linear Translation for VE Navigation (3) Navigation speed is also dependent on the user’s head orientation with respect to the vertical axis Navigation speed is also dependent on the user’s head orientation with respect to the vertical axis especially useful when translating the floor-based WIM especially useful when translating the floor-based WIM Mapping is done with a scaled exponential function Mapping is done with a scaled exponential function Final leaning velocity is Final leaning velocity is

Conclusions Important ideas found in Chapters 5-8 of 3D UI book Important ideas found in Chapters 5-8 of 3D UI book