Interaction in the Virtual World: Overview User interface Navigation Wayfinding Travel Interaction and communication with others Sharing Collaboration Manipulation Methods Properties Selection Operations

Broad Classification of Interactions Reality-based Imagination-based (magic)

Navigation Types of navigation goals: explore rate or position controlled), search (target) and maneuver (objects, environment) Methods: Wayfinding Travel

Navigation: Wayfinding (knowing where you are and where you’re going) Need a mental or cognitive map Relationship to the world, to the destination Situational awareness eg. of GPS system maps Great differences in people, sometimes gender related Problem of becoming lost, metaphors

Navigation: Styles Relative motion (steps) Absolute motion (point to a location and go) Teleport Move the world instead of the user

Navigation: Wayfinding (con’t) Mental map: Divide and conquer: small regions, then piece together Global network: use landmarks Memorize a map of the space Remember a story about the space

Navigation: Wayfinding (con’t) Wayfinding aids Path following: color, lines, labels Maps (exo or ego): scale, icons, you-are-here Landmarks: objects, audio Placenames Breadcrumbs (trail) Compass Instrument guidance: could be multimodal Shift to exocentric view Display of coordinates or grids, names Constrain travel

Navigation: Travel Properties Manipulation method: physical (steering wheels, flight, dashboard) and virtual controls Constraints: terrain following, no z-axis, no lateral Frame of reference: relative motion Movement: gain, velocity, acc; voice

Navigation: Types of Travel Physical locomotion: direct tracking, issue of working volume Vehicle simulation Magic techniques, including steering, target-based travel Combination of natural and magic: manipulation-based travel Ride along or towrope: pre-set path

Navigation: Types of Travel (con’t) Fly-through, walkthrough, pilot-through: pointer directed, gaze-directed, torso-directed, dual-handed Move the world Scale the world Put me there, jump to destination Time travel

Navigation and Travel Physical walking Vehicle simulation Magic techniques, including steering, target-based travel Combination of natural and magic: manipulation-based

Interaction: Communication Shared experience: same world, may be competing or co-existing Collaborative experience: need to work together in some way

Shared Experience Different ways of sharing in different mediums What can be shared: ideas, world, speech Ways to share viewpoint Full multipresence: may have same equipment (such as HMDs), or be in same location (cockpit) Some viewers have partial presence: one immersed participant with onlookers, open display (possibly projection)

Shared Experience (con’t) Purpose of sharing: marketing (onlookers become interested), inc. throughput (onlookers get some experience even if not the full experience)

Collaborative Experience Communication: aural, visual, gestures, teleconferencing, marks or notes, physical proximity Synchronous communication: live conversations Asynchronous communication: change a world but then not be present, present in another part of the world – can leave marks or messages Can have both synchronous and asynchronous in the same world

Collaborative Experience (con’t) Messages or annotations: recipient, time (when needed), purpose How do you know it’s there: voice, text, gestures, pictures Location Point of view Time placeholders

Collaborative Experience (con’t) Who’s in charge (floor control and permissions): no one, moderated, permissions, formal, hierarchical, first come first served

Metacommands Manipulation of environment by someone other than participant (eg. of VR therapy) User can cause different worlds to load

Manipulation: methods Direct user control: gestures mimic real world Physical control devices: wands, buttons, joysticks, mouse Virtual controls: interfaces on the screen such as buttons, sliders, control panel, arrows, etc. Agent controls: intelligent intermediary

Manipulation: properties Feedback: user needs to know when something has happened- haptic (can feel the contact), aural (sound), visual (change color) Ratcheting: repeat input – eg. pick up mouse Constraints: restrict a DOF, snap to grid, lock to surface, keep on floor

Manipulation: properties (con’t) Distance: manipulate objects beyond reach Pointer beam scope: shape of beam; eg. laserbeam, spotlight, conelike

Manipulation: properties (con’t) Frame of reference: world representation, local representation– inside or outside the world- telepresence and teleoperation-egocentric or exocentric (god’s eye)- Coordinate systems: world, local, parent- vocabulary of DOF: vertical, lateral, longitudinal; Euler angles; yaw, pitch, roll

Manipulation: properties (con’t) Bimanual interface: sometimes with haptics, keypress with joystick (difficult), medical applications; must be well designed Control location: omnipresent or appear, placement (interfere with world or not): part of the world (door handle), attached to hand, front of the view (helmet, windshield), on the display, on a 2D panel of controls, physical devices (tablets)

Manipulation: properties (con’t) Control visibility: visible and invisible; how made visible Movement formula: gain (how much movement in one action), derivatives (velocity, acceleration)

Object Manipulation Manipulation of the environment: could also be considered viewpoint navigation Selection of objects: pointing, picking, grasping System control: usually through 2D interfaces, buttons

Manipulation: selection Can be direction, item or value Direction Pointer-directed: gesture Gaze or eye tracking Crosshair: pointer and gaze Torso for direction of travel Joystick, mouse, trackball- what about angles Coordinates: possibly by voice Landmarks

Manipulation: selection (con’t) Item selection Contact between avatar and item – choose part of body; need feedback Point to select (prop, gesture) 3D cursor select; may need button; question of what’s in range and out of range; change in symbol Pinching or grasping Naming Menu; words or pictures Select in a small world

Manipulation: selection (con’t) Value selection: pen devices, keyboard, tablet, physical sliders or controls, agents

Manipulation: operations Positioning and sizing objects Exerting force: pushing, hitting, supporting- does it move? Modifying attributes of objects: color, light, transparency, weight, shape, firmness, density, etc. Modifying global attributes: sounds, time of day, form of rendering

Interactions Reality-based interaction: Head tracking, locomotion, 3D pointing Direct manipulation of objects, gloves, haptics Indirect manipulation through input devices Objects for manipulation, tools (eg. wands), tangible input devices, sliders, joysticks, touch sensors Imagination-based interaction: Suspension of physics Scaling of geometry, zooming, no clear standards yet Scaling of motion Automation of tasks, motion, etc. Magic spells: dynamically assign meaning and tasks Mode changes: eg from selecting to manipulation

Sources Building on Realism and Magic for Designing 3D Interaction Techniques by Kulik, IEEE Computer Graphics and Applications, Nov/Dec 2009 Questioning Naturalism in 3D User Interfaces, Bowman, McMahan, and Ragan, Comm. Of the ACM, 2012 Understanding Virtual Reality by Sherman & Craig, Morgan Kaufman, 2003 Computer Graphics and Virtual Environments by Slater et al