1. Interaction in the Virtual World: Overview
User interface
Navigation
Wayfinding
Travel
Interaction and communication with others
Sharing
Collaboration
Manipulation
Methods
Properties
Selection
Operations

2. Broad Classification of Interactions
Reality-based
Imagination-based (magic)

3. Navigation
Types of navigation goals: explore rate or position controlled), search (target) and maneuver (objects, environment)
Methods:
Wayfinding
Travel

4. Navigation: Wayfinding (knowing where you are and where you’re going)
Need a mental or cognitive map
Relationship to the world, to the destination
Situational awareness
eg. of GPS system maps
Great differences in people, sometimes gender related
Problem of becoming lost, metaphors

5. Navigation: Styles Relative motion (steps)
Absolute motion (point to a location and go)
Teleport
Move the world instead of the user

6. Navigation: Wayfinding (con’t)
Mental map:
Divide and conquer: small regions, then piece together
Global network: use landmarks
Memorize a map of the space
Remember a story about the space

7. Navigation: Wayfinding (con’t)
Wayfinding aids
Path following: color, lines, labels
Maps (exo or ego): scale, icons, you-are-here
Landmarks: objects, audio
Placenames
Breadcrumbs (trail)
Compass
Instrument guidance: could be multimodal
Shift to exocentric view
Display of coordinates or grids, names
Constrain travel

8. Navigation: Travel Properties
Manipulation method: physical (steering wheels, flight, dashboard) and virtual controls
Constraints: terrain following, no z-axis, no lateral
Frame of reference: relative motion
Movement: gain, velocity, acc; voice

9. Navigation: Types of Travel
Physical locomotion: direct tracking, issue of working volume
Vehicle simulation
Magic techniques, including steering, target-based travel
Combination of natural and magic: manipulation-based travel
Ride along or towrope: pre-set path

10. Navigation: Types of Travel (con’t)
Fly-through, walkthrough, pilot-through: pointer directed, gaze-directed, torso-directed, dual-handed
Move the world
Scale the world
Put me there, jump to destination
Time travel

11. Navigation and Travel Physical walking Vehicle simulation
Magic techniques, including steering, target-based travel
Combination of natural and magic: manipulation-based

12. Interaction: Communication
Shared experience: same world, may be competing or co-existing
Collaborative experience: need to work together in some way

13. Shared Experience Different ways of sharing in different mediums
What can be shared: ideas, world, speech
Ways to share viewpoint
Full multipresence: may have same equipment (such as HMDs), or be in same location (cockpit)
Some viewers have partial presence: one immersed participant with onlookers, open display (possibly projection)

14. Shared Experience (con’t)
Purpose of sharing: marketing (onlookers become interested), inc. throughput (onlookers get some experience even if not the full experience)

15. Collaborative Experience
Communication: aural, visual, gestures, teleconferencing, marks or notes, physical proximity
Synchronous communication: live conversations
Asynchronous communication: change a world but then not be present, present in another part of the world – can leave marks or messages
Can have both synchronous and asynchronous in the same world

16. Collaborative Experience (con’t)
Messages or annotations: recipient, time (when needed), purpose
How do you know it’s there: voice, text, gestures, pictures
Location
Point of view
Time
placeholders

17. Collaborative Experience (con’t)
Who’s in charge (floor control and permissions): no one, moderated, permissions, formal, hierarchical, first come first served

18. Metacommands
Manipulation of environment by someone other than participant (eg. of VR therapy)
User can cause different worlds to load

19. Manipulation: methods
Direct user control: gestures mimic real world
Physical control devices: wands, buttons, joysticks, mouse
Virtual controls: interfaces on the screen such as buttons, sliders, control panel, arrows, etc.
Agent controls: intelligent intermediary

20. Manipulation: properties
Feedback: user needs to know when something has happened- haptic (can feel the contact), aural (sound), visual (change color)
Ratcheting: repeat input – eg. pick up mouse
Constraints: restrict a DOF, snap to grid, lock to surface, keep on floor

21. Manipulation: properties (con’t)
Distance: manipulate objects beyond reach
Pointer beam scope: shape of beam; eg. laserbeam, spotlight, conelike

22. Manipulation: properties (con’t)
Frame of reference: world representation, local representation– inside or outside the world- telepresence and teleoperation-egocentric or exocentric (god’s eye)-
Coordinate systems: world, local, parent- vocabulary of DOF: vertical, lateral, longitudinal; Euler angles; yaw, pitch, roll

23. Manipulation: properties (con’t)
Bimanual interface: sometimes with haptics, keypress with joystick (difficult), medical applications; must be well designed
Control location: omnipresent or appear, placement (interfere with world or not): part of the world (door handle), attached to hand, front of the view (helmet, windshield), on the display, on a 2D panel of controls, physical devices (tablets)

24. Manipulation: properties (con’t)
Control visibility: visible and invisible; how made visible
Movement formula: gain (how much movement in one action), derivatives (velocity, acceleration)

25. Object Manipulation
Manipulation of the environment: could also be considered viewpoint navigation
Selection of objects: pointing, picking, grasping
System control: usually through 2D interfaces, buttons

26. Manipulation: selection
Can be direction, item or value
Direction
Pointer-directed: gesture
Gaze or eye tracking
Crosshair: pointer and gaze
Torso for direction of travel
Joystick, mouse, trackball- what about angles
Coordinates: possibly by voice
Landmarks

27. Manipulation: selection (con’t)
Item selection
Contact between avatar and item – choose part of body; need feedback
Point to select (prop, gesture)
3D cursor select; may need button; question of what’s in range and out of range; change in symbol
Pinching or grasping
Naming
Menu; words or pictures
Select in a small world

28. Manipulation: selection (con’t)
Value selection: pen devices, keyboard, tablet, physical sliders or controls, agents

29. Manipulation: operations
Positioning and sizing objects
Exerting force: pushing, hitting, supporting- does it move?
Modifying attributes of objects: color, light, transparency, weight, shape, firmness, density, etc.
Modifying global attributes: sounds, time of day, form of rendering

30. Interactions Reality-based interaction:
Head tracking, locomotion, 3D pointing
Direct manipulation of objects, gloves, haptics
Indirect manipulation through input devices
Objects for manipulation, tools (eg. wands), tangible input devices, sliders, joysticks, touch sensors
Imagination-based interaction:
Suspension of physics
Scaling of geometry, zooming, no clear standards yet
Scaling of motion
Automation of tasks, motion, etc.
Magic spells: dynamically assign meaning and tasks
Mode changes: eg from selecting to manipulation

31. Sources
Building on Realism and Magic for Designing 3D Interaction Techniques by Kulik, IEEE Computer Graphics and Applications, Nov/Dec 2009
Questioning Naturalism in 3D User Interfaces, Bowman, McMahan, and Ragan, Comm. Of the ACM, 2012
Understanding Virtual Reality by Sherman & Craig, Morgan Kaufman, 2003
Computer Graphics and Virtual Environments by Slater et al