1. Leveraging Human Capabilities in Advanced User Interfaces
Mary Czerwinski
Sr. Researcher
Microsoft Research

2. Thanks to my Colleagues
Desney Tan
George Robertson
Greg Smith
Patrick Baudisch
Brian Meyers
11/9/2018
IEEE HCC Keynote

3. Introduction Overview of user-centered design and process
Exploration of 3D UI designs that leverage human capabilities
Examination of animation in UI
Exploration of large displays and gender differences in 3D navigation
Conclusion
11/9/2018
IEEE HCC Keynote

4. User-Centered Design Start from existing user problem
Ethnographic work, lab studies or literature
Design innovation and brainstorming
Psychological and human-computer interaction (hci) principles-driven!
Prototype -> show to users -> iterate
Extend principles, publish findings to hci community
11/9/2018
IEEE HCC Keynote

5. Progress Chart Overview of user-centered design and process
Exploration of 3D UI designs that leverage human capabilities
Examination of animation in UI
Exploration of large displays and gender differences in 3D navigation
Conclusion
11/9/2018
IEEE HCC Keynote

6. 3D UI Projects Data Mountain (UIST 98) Task Gallery (CHI 2000)
Polyarchy visualization (shipped June 2003)
Scalable Fabric
Large displays, optical flow and wider fields of view (CHI 2001, 2002, 2003)
11/9/2018
IEEE HCC Keynote

7. Data Mountain User problem—can’t use IE Favorites
Leverages spatial memory and visual recognition
“Strongest cue-
relative size”
Document management -- IE Favorites, window management
Pages of interest are placed on a mountain side (a tilted plane in prototype)
Act of placing page makes it easier to remember where it is
Usability test: Storage & retrieval test for 100 pages; ~26% faster
By exploiting 3D perception and spatial memory
User can organize documents spatially
Can get more info in same space with no additional cognitive load
Can see multiple pages at a time
Can see patterns of related documents
Advantages of 3D with 2D interaction technique
Here is a sample user layout of 100 pages
Markings are landmarks with no intrinsic meaning
May look random, but in fact makes a lot of sense to this user
Typical comments
Strongest cue is the relative size
“I know where that is … “
Subject Layout of 100 Pages
11/9/2018
IEEE HCC Keynote

8. Data Mountain Usability
Study #1 (Compare with IE4 Favorites)
Reliably faster (26%)
Study #2 (Longevity and Thumbnails)
After 6 months, no performance change
Images help, but are not required
Studies #3 & 4 (Implicit Query)
Faster retrieval if similar pages highlighted
100 Pages taken from:
PC Magazine list of top web sites
Yahoo Web Roulette
Store the 100 pages
Pages presented sequentially in Random order
“Organize as you do at home/work”
Allowed to Reorganize any time
Organize
<short break>
Retrieve
Random order
Always a page seen in storage phase
CUES: Title, Summary, Thumbnail, AllThree
11/9/2018
IEEE HCC Keynote

9. Task Gallery User problem: task switching Task management
Simple, forward-back navigation
Tasks laid out spatially on floor, ceiling, walls
Simple task switch
Leverages spatial memory, visual attention and recognition
11/9/2018
IEEE HCC Keynote

10. Task Gallery Simultaneous viewing of multiple windows
Simple shift select
Smart arrangement
Use 3D to provide uniform scaling
Saves user from having to manage layouts
11/9/2018
IEEE HCC Keynote

11. Progress Chart Overview of user-centered design and process
Exploration of 3D UI designs that leverage human capabilities
Examination of animation in UI
Exploration of large displays and gender differences in 3D navigation
Conclusion
11/9/2018
IEEE HCC Keynote

12. Animation Effectiveness
Tversky et al. (2001)
Animation not always useful
Info Vis Community
Robertson, Card & Mackinlary (‘91): Cone Tree node transitions involved rotations for maintaining context
Bartram (‘98): animation evoked an emergent property of grouping when multiple, similar motions occur
Bederson & Boltman (‘98): 1 sec. animation significantly reduced errors and task times
. In the Cone Tree, animation was used during transitions to newly selected nodes. The transformation was complex, possibly involving several simultaneous rotations of sub-cones (some clockwise and some counter-clockwise), bringing the selected node and the path to the root to the front of the display. It was observed that without animation users took extra time to re-assimilate the information structure with which they were interacting. With a one second animation, the perceptual system tracked the transformation, and users spent much less time in re-assimilation. The speculation was that the visual perceptual tracking system was at work, allowing the user to pre-attentively perceive that it was the same structure undergoing transformation. While no user studies were done to verify that speculation, a simple demonstration comparing the transformation with and without animation was very compelling. We believe the same mechanism is at work in the Visual Pivot animations.
). In order to support synthesis and awareness across large information displays, Bartram argues for the use of animation to improve information bandwidth. Based on a series of studies, Bartram concludes that one strong argument for using animation comes from its ability to evoke an emergent property of grouping when multiple, similar motions occur across a dense data display. This allows the user to immediately recognize associated elements which may be widely dispersed. In Visual Pivot, the nodes in each view are grouped with the aid of animation in just the way Bartram describes. Bartram also demonstrates that combining movements of separate elements can elicit an immediate understanding as to how they are related. These findings support the claim that Visual Pivot helps the user see the relationships between views.
one second animation did not slow down participant response times, and significantly reduced the number of errors and time taken in the reconstruction of the tree from memory. In addition, 12 out of 20 participants thought that the animated version of the tree provided both an improved understanding of relationships and the ability to better see an overall view. The authors believe that the animation directly aided the learning of spatial relationships through spatial memory, which helped in the navigation and reconstruction tasks. A preliminary design guideline offered by the authors is that if a task requires users to know something about objects’ spatial positioning, and the viewpoint is changed, then animating that change in viewpoint appears to benefit users in the task.
11/9/2018
IEEE HCC Keynote

13. Polyarchy Visualization
Multiple Intersecting Hierarchies
Solves user problems of:
Shows multiple foci at once
Shows item relationships in context
Manages viewing multiple hierarchies
Key concept: visual pivot
Shipped June 2003
11/9/2018
IEEE HCC Keynote

14. Two Styles of Visual Pivot
Rotating
Sliding
Notice in all of these displays, we are managing an enormous database by
showing the minimum set of information that meets the user’s needs
Allowing the user to easily change the information shown
11/9/2018
IEEE HCC Keynote

15. Rotation around Horizontal Axis
The results of the third study indicated that two particular pivot styles should be further evaluated
Initial pivots had a final step that moved to a preferred position and scale
The study showed that people had trouble understanding that because the new information was not always readily visible
The revised animations do that movement first rather than last
The study also showed that users had difficulty reading text while rotating around the vertical axis
Revised pivot is around the horizontal axis
11/9/2018
IEEE HCC Keynote

16. Sliding Animation 11/9/2018 IEEE HCC Keynote
A new pivot was added that slides the new information from right to left, then up a small amount
This allows reading the text of old and new side by side
Here we use transparency, 3D, animation (translation)
Note the use of a frame of reference around the new hierarchy (darker border)---used in all animations
11/9/2018
IEEE HCC Keynote

17. Proffitt and Kaiser (’93)
Users analyze animations into relative (rotation) and common (translation) motion components
Secondly, rotation and translation motions have different perceptual significance
Rotations define 3D form, while translations define observer-relative displacements
Suggests sliding pivot perceived as observer-relative and rotating perceived as defining 3D form (less useful for our tasks?)
11/9/2018
IEEE HCC Keynote

18. Polyarchy Visualization User Studies
Study 3: Animation Styles and Speeds
Six animation styles: Picked two best
Twice as fast as study 2: Still too slow
Study 4: Prototype: 2D vs 3D
Identified most effective animation style
Identified best speed range—0.5 sec.
Study 5: Examined complexity of query and sliding v. stacked animations
The third study was specifically designed to evaluate six alternative animation styles and to identify the appropriate speed range
The most recent study was another performance study, similar to Study 2 except that the usability issues were resolved
11/9/2018
IEEE HCC Keynote

19. Study 4: Animation Styles Sliding versus Rotating
The fourth study tested these styles
Looking at performance data, there is a reliable advantage to the sliding pivot
Satisfaction data was mixed, with no clear preference
11/9/2018
IEEE HCC Keynote

20. Study 4: Animation Styles Learning Effects
Performance data shows a reliable learning effect,
with the sliding pivot easier to learn
11/9/2018
IEEE HCC Keynote

21. Study 4: Animation Timing
Pivot times are broken into three phases:
Reposition
Pause after new information is shown
Pivot
Total Times tested: 0 sec, ½ sec, 1 sec, 2 sec
User task completion time was reliably slower with the slow animation
But, the other animation speeds, as well as no animation, were not reliably different
Can conclude that 3D animation does not incur any performance penalty
Preference data indicated a reliable preference for the “fast” animation
We are continuing the study of these visual pivot techniques
11/9/2018
IEEE HCC Keynote

22. Progress Chart Overview of user-centered design and process
Exploration of 3D UI designs that leverage human capabilities
Examination of animation in UI
Exploration of large displays and gender differences in 3D navigation
Conclusion
11/9/2018
IEEE HCC Keynote

23. Ignore Science Fiction at Our Peril
Workstation in the world of the Matrix
11/9/2018
IEEE HCC Keynote

24. Large Display Surfaces are Here
Workstation in the real world
11/9/2018
IEEE HCC Keynote

25. Why A Larger Display Surface?
Productivity benefits 15-30% (despite OS issues)
Users prefer more display surface
Prices dropping fast
Footprints getting smaller
11/9/2018
IEEE HCC Keynote

26. Multimon Trend is Growing
No Multimon
30%
Plan to Use
Multimon
38%
Use Multimon
32%
(Jon Peddie Research
Dec, 2002 N=6652)
11/9/2018
IEEE HCC Keynote

27. 2004 Large Monitor ASP Projections
16:9 x 22” Diagonal
20”Diagonal
2 x 17” (30” Diagonal)
Relative Pricing
$1000
Point here is that, for the same amount of money (and this estimate is already too conservative), the user will be able to gain more diagonal display inches by purchasing 2 smaller LCDs than a single, larger one, for the same price.
2 x 15” (26” Diagonal)
17”Diagonal
15”Diagonal
Single
Multiple
Wide
11/9/2018
IEEE HCC Keynote
Note: All Prices are for Liquid Crystal Displays
Source for Single Panel Pricing: IDC and Display Search

28. Large Display User Experience, MSR
Large display surfaces fundamentally change user interaction
Focus on input, visualization and windows management
Large display surfaces provide non-linear productivity increases
Additional space has different utility
E.g. Focal/peripheral displays provide different cues
11/9/2018
IEEE HCC Keynote

29. Windows and Task Management Issues Emerge
Larger displays = more open windows
Multimon users arrange windows spatially
TaskBar does not scale:
Aggregation model not task-based
Users can’t operate on groups of related windows
11/9/2018
IEEE HCC Keynote

30. INPUT: Drag ‘n Pop Problems:
Large displays create long distance mouse movement
Drag ‘n Pop brings proxies of targets to the user from across display surfaces
11/9/2018
IEEE HCC Keynote

31. Scalable Fabric (for Large Displays)
Beyond Minimization
Large display users keep more windows open
With so much screen real estate, why minimize?
Manage tasks using visual recognition and spatial memory
Central focus area
Periphery: windows scaled
Cluster of windows = task
11/9/2018
IEEE HCC Keynote

32. Women Take a Wider View (CHI 2002)
Grew from work designing and evaluating 3D virtual navigation techniques
On regular desktop display:
Men performed significantly better than women
On exploratory widescreen display:
Overall improvement for all users
Surprising finding:
Gender gap disappeared - Males and females performed equally on widescreen display
11/9/2018
IEEE HCC Keynote

33. Related Work
Formation of cognitive maps while navigating 3D virtual worlds
Spatial abilities
Artifacts (maps, landmarks,…)
Gender differences in spatial ability and navigation strategies
Most report male advantages, especially in virtual environments
11/9/2018
IEEE HCC Keynote

34. Related Work: Optical Flow
Changing retinal image as we move through the environment
Aids perception of environmental structure
11/9/2018
IEEE HCC Keynote

35. Related Work: Optical Flow
Changing retinal image as we move through the environment
Aids perception of environmental structure
11/9/2018
IEEE HCC Keynote

36. What we know about Optical Flow
Optical flow benefits heading perception in active navigation
Shown for fields of view up to 90 degrees
Hypothesized that effectiveness of optical flow depends on spatial ability
[Cutmore et al. 2000]
Gender unexplored
11/9/2018
IEEE HCC Keynote

37. Our Hypotheses
Optical flow cues help all users form better cognitive maps when navigating 3D virtual environments
Better optical flow cues help women more than men in cognitive map formation
Wider displays offer even better optical flow cues
We’ve seen these effects for up to about 90 degree FOV
11/9/2018
IEEE HCC Keynote

38. Dsharp Display
43"
11"
11/9/2018
IEEE HCC Keynote

39. Task: General Description
Learning: User controls movement along path through virtual 3D maze
Testing: Remember path traveled
11/9/2018
IEEE HCC Keynote

40. Virtual Maps 14 rooms (6 straight ahead, 8 turns)
Some paths go through same room twice
For example:
11/9/2018
IEEE HCC Keynote

41. Cognitive Map Learning
Use arrow keys to go through green door
Determine if path crosses itself
Remember full path
With flow and without flow
11/9/2018
IEEE HCC Keynote

42. Cognitive Map Memory Test
Tested on memory for maze
Forward test and backward test
Measured task time & number of correct doors opened on first attempt
Same controls as in learning phase, but without green door guides
Given feedback
11/9/2018
IEEE HCC Keynote

43. Experimental Design Female Male Small FOV: 100 degrees
Large FOV: 120 degrees
Optical Flow
Absent
Optical Flow
Present
11/9/2018
IEEE HCC Keynote

44. Experimental Procedure
Paper folding test of spatial ability
1 practice trial + 4 test trials
Satisfaction questionnaire
11/9/2018
IEEE HCC Keynote

45. Benefits of Optical Flow
11/9/2018
IEEE HCC Keynote

46. Optical Flow Helps All Users in Forward Test
Backward
11/9/2018
IEEE HCC Keynote

47. Optical Flow Benefits Females More in the Forward Test
11/9/2018
IEEE HCC Keynote

48. Other Results No effects for field of view
No effects for spatial ability measure
Satisfaction ratings matched performance results
11/9/2018
IEEE HCC Keynote

49. Conclusion
Optical flow cues help all users form better cognitive maps when navigating 3D virtual environments
Better optical flow cues help women more than men in cognitive map formation
Unexplained by biases in spatial ability
Wider displays offer even better optical flow cues
100 degree field of view seems sufficient
Spatial ability held constant
11/9/2018
IEEE HCC Keynote

50. Information Voyeurism: Social Impact of Large Displays
Exploit social cues induced by physical size:
Help people communicate
Increase productivity on individual tasks
Must quantify in order to exploit
Information on large displays more public
Ask user? Cannot guarantee accuracy
Video? Cannot disambiguate glance from reading
Exploit social cues induced by physical size:
Help people communicate
Increase productivity on individual tasks
Must quantify in order to exploit
Information on large displays more public
Ask user? Cannot guarantee accuracy
Video? Cannot disambiguate glance from reading
11/9/2018
IEEE HCC Keynote

51. Measuring ‘Peeking’ Implicit memory priming paradigm
Expose user to stimulus
Test user implicitly on how much they’ve processed stimulus
Word stem completion
Eg. Mon_____
Priming measured by faster response or higher frequency of stimulus
Monkey, Money, Monster, Monday, Monopoly, …
Repetition priming effects: facilitation in processing of a stimulus as a function of recent encounter with the same stimulus
Without reference to the prior stimulus…
First step in this experimental paradigm:
Let user see the stimulus (in this case, a monkey)
User can be asked to perform some action/decision on this object (just to make sure it’s being processed). Not necessary
Then:
Test user implicitly on how much they’ve processed stimulus.
Without reference to the prior stimulus, asked to perform a task which is influenced by processing of stimulus
In this case, user likely to answer with “monkey”
Measure:
Faster
More frequent
11/9/2018
IEEE HCC Keynote

52. Experiment Materials Stimulus: 30 words embedded in:
7 subject lines
2 messages
Place where it can be seen by user
Priming test to see if they’ve read it
Word stem completion
We used a textual version of this priming paradigm:
We embedded 30 carefully selected words in 7 subject lines and 2 msgs
The priming test we used was word stem completion.
Given the first 3 letters of a word and asked to complete the word as quickly as possible
Measure the frequency that primed words appeared as solutions
If the user had read the subject lines/ msgs, they would be process the primed words and be more likely to complete the stems with these words
Actually
Controlled and didn’t control…
New paradigm
11/9/2018
IEEE HCC Keynote

53. Experimental Setup Large Projection Screen Small Desktop Monitor
156″
38″
16″
27.5″
66″
114″
Experimenter
Participant
Large Projection
Screen
Small
Desktop Monitor
11/9/2018
IEEE HCC Keynote

54. Implicit Memory Results
N=12
Average # of Target Words
N=12
Small Display
Large Display
11/9/2018
IEEE HCC Keynote

55. Other Converging Data More users admitted reading text on:
Large Screen (7/12) vs.
Small Screen (3/12)
Comments indicated reading someone else’s more acceptable on large screen
Video shows users glanced more at:
Large Screen (M=19 seconds) vs.
Small Screen (M=14 seconds)
11/9/2018
IEEE HCC Keynote

56. Design Implications Protect private information from prying eyes
Private information never placed on public screens
Interface conventions that convey level of privacy
Facilitate ad hoc collaboration
Display systems that make people interact more
11/9/2018
IEEE HCC Keynote

57. Progress Chart Overview of user-centered design and process
Exploration of 3D UI designs that leverage human capabilities
Examination of animation in UI
Exploration of large displays and gender differences in 3D navigation
Conclusion
11/9/2018
IEEE HCC Keynote

58. Conclusion
Successful, advanced user interface design requires knowing the user problem to be solved
As opposed to being technology-driven
Leveraging human capabilities ensures improvements over existing techniques
Usable designs
Contributions to science
Principles derived for human-computer interaction discipline
11/9/2018
IEEE HCC Keynote

59. Thank you…

60. Large Display Surfaces are Here
Trend towards multiple display systems that enlarge display space
Workstation in the real world (InfoCockpit--CMU)
11/9/2018
IEEE HCC Keynote

61. User Views of Maze Narrow field of view (100 degrees)
Wide field of view (120 degrees)
11/9/2018
IEEE HCC Keynote

62. Visual Pivot (Rotation around Vertical Axis)B C D E F G H
The user can pivot to any other dimension that the selected node participates in
This shows the original visual pivot animation around a vertical axis
First pivoting to the business unit hierarchy
Using animation and transparency, 3D to emphasize what’s happening between the two hierarchies around the pivot point
11/9/2018
IEEE HCC Keynote

63. Schematic of Visual Pivot (horizontal rotation)
Hierarchy 1
Hierarchy 2
Axis of rotation
11/9/2018
IEEE HCC Keynote