Chapter 9 System Control Rowan Gudmundsson

Human Factors Some human factors we have to consider when designing a 3D UI: Visibility Focus switching Feedback

Visibility Occlusion Legibility

Focus Switching UI decoupling Physical devices

Feedback Haptic Auditory Visual

Breadth and Depth Breadth (number of categories) Depth (depth of sub-categories)

Ergonomic Issues Control position Shape, size, and location of controls Optimal grip, pose for user

System Factors Resolution, size, and luminance Secondary visual devices Input devices Noise, device constraints

Classification Physical controllers Graphical menus Voice commands Buttons Switches Graphical menus Adapted 2D menus 1 DOF menus 3D Widgets Voice commands Gestural commands Mimic gestures Symbolic gestures Sign language Speech connected hand gestures Whole-body interaction

Classification continued ... Tools Physical tools Virtual tools Tangibles Multimodal techniques

Physical Controllers Buttons (usually to switch states) Examples: External joysticks (flight simulators) Controllers (consoles) Mice (personal computers)

Graphical Menus Adapted 2D menus 1-DOF menus 3D widgets

Adapted 2D Menus The 2D menus that we’re all used to but in 3D Projected as a billboard into 3D space

Adapted 2D Menus ... Pros: Familiar Simple to use Good for complex functionality Cons: Obvious occlusion issues Difficult to navigate for some selection techniques

1-DOF Menus Menu usually attached to the player in some way Controlled by a single motion with a button selection

1-DOF Menus Continued ... Pros: Natural for the user Attached directly to the hand Cons: Ergonomic constraints Simple functionality

3D Widgets Tied directly to the object being manipulated Allows for complex options for a specific interaction 2 Different types: Collocated (context-sensitive) Non-context-sensitive

3D Widgets Continued ... Pros: Allows for complex options for specific interactions Integration into the 3D space Cons: Discoverability issues Difficult to implement Sometimes complex interactions are needed

Design and Implementation Issues Placement Selection Representation and structure

Menu Placement Things to consider: Amount of occlusion introduced Ability for the user to access the menu Spatial reference point world-ref , object-ref, head-ref, body-ref, device-ref.

Menu Selection Things to consider: Controlling degrees of freedom Intuitive methods of selection

Menu Representation and Structure Things to consider: Size and spacing issues Functionally grouping items Control coding (textures, colors, dimensions, symbols, etc.) Environment issues (luminance, contrast, etc.)

Voice Commands Can be very effective in applications which require the use of both hands Very low focus shift Graphically invisible to the user Downsides: Very robust speech recognition engine for seamless integration Influenced by background noise Can be tiring for the user Requires confirmation to verify intent

Gestural Commands Broken down into 2 categories: Postures (static poses) Gestures (full motion metaphor)

Gestural Commands Continued ... Mimic gestures Not connected to speech but instead convey a concept Symbolic gestures Gestures as used in daily life to express things such as insults or praise Sweeping Gestures for navigation like scrolling Sign language Set of postures or gestures to convey particular words or numbers Speech connected hand gestures Gestures people use when speaking to further convey emotion or to emphasise a point Surface based gestures Gestures made on 2D surfaces (eg. pinch-zoom) Whole-body interaction Mimic or symbolic gestures using more than just hands and arms

Gestural Commands Continued ... Pros: Natural to the user Can be used to convey complex interactions Cons: Calibration may be required Requires a machine learning or heuristic back-end Like speech, a confirmation is usually needed to verify intent

Tools Physical tools Tangibles Virtual tools Advantages of using tools: Users can operate them by touch which frees up visual attention May feel more natural to the user More defined behaviour

Physical Tools Real world “props” which have a specific purpose Largely used in public installations like theme parks or museums Very easy to use but have limited functionality Pirates of the caribbean - Disney quest

Tangibles A superset of physical tools More abstract than physical tools Tool may have buttons or gestures associated with it to allow for more functionality

Virtual Tools Tools which have no physical manifestation Representation of a specific action set in the virtual environment Allows developer to group actions

Multimodal Techniques Combines previous techniques to give a more complete UX.