Leveraging Human Capabilities in Advanced User Interfaces Mary Czerwinski Sr. Researcher Microsoft Research

Thanks to my Colleagues Desney Tan George Robertson Greg Smith Patrick Baudisch Brian Meyers 11/9/2018 IEEE HCC Keynote

Introduction Overview of user-centered design and process Exploration of 3D UI designs that leverage human capabilities Examination of animation in UI Exploration of large displays and gender differences in 3D navigation Conclusion 11/9/2018 IEEE HCC Keynote

User-Centered Design Start from existing user problem Ethnographic work, lab studies or literature Design innovation and brainstorming Psychological and human-computer interaction (hci) principles-driven! Prototype -> show to users -> iterate Extend principles, publish findings to hci community 11/9/2018 IEEE HCC Keynote

Progress Chart Overview of user-centered design and process Exploration of 3D UI designs that leverage human capabilities Examination of animation in UI Exploration of large displays and gender differences in 3D navigation Conclusion 11/9/2018 IEEE HCC Keynote

3D UI Projects Data Mountain (UIST 98) Task Gallery (CHI 2000) Polyarchy visualization (shipped June 2003) Scalable Fabric Large displays, optical flow and wider fields of view (CHI 2001, 2002, 2003) 11/9/2018 IEEE HCC Keynote

Data Mountain User problem—can’t use IE Favorites Leverages spatial memory and visual recognition “Strongest cue- relative size” Document management -- IE Favorites, window management Pages of interest are placed on a mountain side (a tilted plane in prototype) Act of placing page makes it easier to remember where it is Usability test: Storage & retrieval test for 100 pages; ~26% faster By exploiting 3D perception and spatial memory User can organize documents spatially Can get more info in same space with no additional cognitive load Can see multiple pages at a time Can see patterns of related documents Advantages of 3D with 2D interaction technique Here is a sample user layout of 100 pages Markings are landmarks with no intrinsic meaning May look random, but in fact makes a lot of sense to this user Typical comments Strongest cue is the relative size “I know where that is … “ Subject Layout of 100 Pages 11/9/2018 IEEE HCC Keynote

Data Mountain Usability Study #1 (Compare with IE4 Favorites) Reliably faster (26%) Study #2 (Longevity and Thumbnails) After 6 months, no performance change Images help, but are not required Studies #3 & 4 (Implicit Query) Faster retrieval if similar pages highlighted 100 Pages taken from: PC Magazine list of top web sites Yahoo Web Roulette Store the 100 pages Pages presented sequentially in Random order “Organize as you do at home/work” Allowed to Reorganize any time Organize <short break> Retrieve Random order Always a page seen in storage phase CUES: Title, Summary, Thumbnail, AllThree 11/9/2018 IEEE HCC Keynote

Task Gallery User problem: task switching Task management Simple, forward-back navigation Tasks laid out spatially on floor, ceiling, walls Simple task switch Leverages spatial memory, visual attention and recognition 11/9/2018 IEEE HCC Keynote

Task Gallery Simultaneous viewing of multiple windows Simple shift select Smart arrangement Use 3D to provide uniform scaling Saves user from having to manage layouts 11/9/2018 IEEE HCC Keynote

Progress Chart Overview of user-centered design and process Exploration of 3D UI designs that leverage human capabilities Examination of animation in UI Exploration of large displays and gender differences in 3D navigation Conclusion 11/9/2018 IEEE HCC Keynote

Animation Effectiveness Tversky et al. (2001) Animation not always useful Info Vis Community Robertson, Card & Mackinlary (‘91): Cone Tree node transitions involved rotations for maintaining context Bartram (‘98): animation evoked an emergent property of grouping when multiple, similar motions occur Bederson & Boltman (‘98): 1 sec. animation significantly reduced errors and task times . In the Cone Tree, animation was used during transitions to newly selected nodes. The transformation was complex, possibly involving several simultaneous rotations of sub-cones (some clockwise and some counter-clockwise), bringing the selected node and the path to the root to the front of the display. It was observed that without animation users took extra time to re-assimilate the information structure with which they were interacting. With a one second animation, the perceptual system tracked the transformation, and users spent much less time in re-assimilation. The speculation was that the visual perceptual tracking system was at work, allowing the user to pre-attentively perceive that it was the same structure undergoing transformation. While no user studies were done to verify that speculation, a simple demonstration comparing the transformation with and without animation was very compelling. We believe the same mechanism is at work in the Visual Pivot animations. ). In order to support synthesis and awareness across large information displays, Bartram argues for the use of animation to improve information bandwidth. Based on a series of studies, Bartram concludes that one strong argument for using animation comes from its ability to evoke an emergent property of grouping when multiple, similar motions occur across a dense data display. This allows the user to immediately recognize associated elements which may be widely dispersed. In Visual Pivot, the nodes in each view are grouped with the aid of animation in just the way Bartram describes. Bartram also demonstrates that combining movements of separate elements can elicit an immediate understanding as to how they are related. These findings support the claim that Visual Pivot helps the user see the relationships between views. one second animation did not slow down participant response times, and significantly reduced the number of errors and time taken in the reconstruction of the tree from memory. In addition, 12 out of 20 participants thought that the animated version of the tree provided both an improved understanding of relationships and the ability to better see an overall view. The authors believe that the animation directly aided the learning of spatial relationships through spatial memory, which helped in the navigation and reconstruction tasks. A preliminary design guideline offered by the authors is that if a task requires users to know something about objects’ spatial positioning, and the viewpoint is changed, then animating that change in viewpoint appears to benefit users in the task. 11/9/2018 IEEE HCC Keynote

Polyarchy Visualization Multiple Intersecting Hierarchies Solves user problems of: Shows multiple foci at once Shows item relationships in context Manages viewing multiple hierarchies Key concept: visual pivot Shipped June 2003 11/9/2018 IEEE HCC Keynote

Two Styles of Visual Pivot Rotating Sliding Notice in all of these displays, we are managing an enormous database by showing the minimum set of information that meets the user’s needs Allowing the user to easily change the information shown 11/9/2018 IEEE HCC Keynote

Rotation around Horizontal Axis The results of the third study indicated that two particular pivot styles should be further evaluated Initial pivots had a final step that moved to a preferred position and scale The study showed that people had trouble understanding that because the new information was not always readily visible The revised animations do that movement first rather than last The study also showed that users had difficulty reading text while rotating around the vertical axis Revised pivot is around the horizontal axis 11/9/2018 IEEE HCC Keynote

Sliding Animation 11/9/2018 IEEE HCC Keynote A new pivot was added that slides the new information from right to left, then up a small amount This allows reading the text of old and new side by side Here we use transparency, 3D, animation (translation) Note the use of a frame of reference around the new hierarchy (darker border)---used in all animations 11/9/2018 IEEE HCC Keynote

Proffitt and Kaiser (’93) Users analyze animations into relative (rotation) and common (translation) motion components Secondly, rotation and translation motions have different perceptual significance Rotations define 3D form, while translations define observer-relative displacements Suggests sliding pivot perceived as observer-relative and rotating perceived as defining 3D form (less useful for our tasks?) 11/9/2018 IEEE HCC Keynote

Polyarchy Visualization User Studies Study 3: Animation Styles and Speeds Six animation styles: Picked two best Twice as fast as study 2: Still too slow Study 4: Prototype: 2D vs 3D Identified most effective animation style Identified best speed range—0.5 sec. Study 5: Examined complexity of query and sliding v. stacked animations The third study was specifically designed to evaluate six alternative animation styles and to identify the appropriate speed range The most recent study was another performance study, similar to Study 2 except that the usability issues were resolved 11/9/2018 IEEE HCC Keynote

Study 4: Animation Styles Sliding versus Rotating The fourth study tested these styles Looking at performance data, there is a reliable advantage to the sliding pivot Satisfaction data was mixed, with no clear preference 11/9/2018 IEEE HCC Keynote

Study 4: Animation Styles Learning Effects Performance data shows a reliable learning effect, with the sliding pivot easier to learn 11/9/2018 IEEE HCC Keynote

Study 4: Animation Timing Pivot times are broken into three phases: Reposition Pause after new information is shown Pivot Total Times tested: 0 sec, ½ sec, 1 sec, 2 sec User task completion time was reliably slower with the slow animation But, the other animation speeds, as well as no animation, were not reliably different Can conclude that 3D animation does not incur any performance penalty Preference data indicated a reliable preference for the “fast” animation We are continuing the study of these visual pivot techniques 11/9/2018 IEEE HCC Keynote

Progress Chart Overview of user-centered design and process Exploration of 3D UI designs that leverage human capabilities Examination of animation in UI Exploration of large displays and gender differences in 3D navigation Conclusion 11/9/2018 IEEE HCC Keynote

Ignore Science Fiction at Our Peril Workstation in the world of the Matrix 11/9/2018 IEEE HCC Keynote

Large Display Surfaces are Here Workstation in the real world 11/9/2018 IEEE HCC Keynote

Why A Larger Display Surface? Productivity benefits 15-30% (despite OS issues) Users prefer more display surface Prices dropping fast Footprints getting smaller 11/9/2018 IEEE HCC Keynote

Multimon Trend is Growing No Multimon 30% Plan to Use Multimon 38% Use Multimon 32% (Jon Peddie Research Dec, 2002 N=6652) 11/9/2018 IEEE HCC Keynote

2004 Large Monitor ASP Projections 16:9 x 22” Diagonal 20”Diagonal 2 x 17” (30” Diagonal) Relative Pricing $1000 Point here is that, for the same amount of money (and this estimate is already too conservative), the user will be able to gain more diagonal display inches by purchasing 2 smaller LCDs than a single, larger one, for the same price. 2 x 15” (26” Diagonal) 17”Diagonal 15”Diagonal Single Multiple Wide 11/9/2018 IEEE HCC Keynote Note: All Prices are for Liquid Crystal Displays Source for Single Panel Pricing: IDC and Display Search

Large Display User Experience, MSR Large display surfaces fundamentally change user interaction Focus on input, visualization and windows management Large display surfaces provide non-linear productivity increases Additional space has different utility E.g. Focal/peripheral displays provide different cues 11/9/2018 IEEE HCC Keynote

Windows and Task Management Issues Emerge Larger displays = more open windows Multimon users arrange windows spatially TaskBar does not scale: Aggregation model not task-based Users can’t operate on groups of related windows 11/9/2018 IEEE HCC Keynote

INPUT: Drag ‘n Pop Problems: Large displays create long distance mouse movement Drag ‘n Pop brings proxies of targets to the user from across display surfaces 11/9/2018 IEEE HCC Keynote

Scalable Fabric (for Large Displays) Beyond Minimization Large display users keep more windows open With so much screen real estate, why minimize? Manage tasks using visual recognition and spatial memory Central focus area Periphery: windows scaled Cluster of windows = task 11/9/2018 IEEE HCC Keynote

Women Take a Wider View (CHI 2002) Grew from work designing and evaluating 3D virtual navigation techniques On regular desktop display: Men performed significantly better than women On exploratory widescreen display: Overall improvement for all users Surprising finding: Gender gap disappeared - Males and females performed equally on widescreen display 11/9/2018 IEEE HCC Keynote

Related Work Formation of cognitive maps while navigating 3D virtual worlds Spatial abilities Artifacts (maps, landmarks,…) Gender differences in spatial ability and navigation strategies Most report male advantages, especially in virtual environments 11/9/2018 IEEE HCC Keynote

Related Work: Optical Flow Changing retinal image as we move through the environment Aids perception of environmental structure 11/9/2018 IEEE HCC Keynote

Related Work: Optical Flow Changing retinal image as we move through the environment Aids perception of environmental structure 11/9/2018 IEEE HCC Keynote

What we know about Optical Flow Optical flow benefits heading perception in active navigation Shown for fields of view up to 90 degrees Hypothesized that effectiveness of optical flow depends on spatial ability [Cutmore et al. 2000] Gender unexplored 11/9/2018 IEEE HCC Keynote

Our Hypotheses Optical flow cues help all users form better cognitive maps when navigating 3D virtual environments Better optical flow cues help women more than men in cognitive map formation Wider displays offer even better optical flow cues We’ve seen these effects for up to about 90 degree FOV 11/9/2018 IEEE HCC Keynote

Dsharp Display 43" 11" 11/9/2018 IEEE HCC Keynote

Task: General Description Learning: User controls movement along path through virtual 3D maze Testing: Remember path traveled 11/9/2018 IEEE HCC Keynote

Virtual Maps 14 rooms (6 straight ahead, 8 turns) Some paths go through same room twice For example: 11/9/2018 IEEE HCC Keynote

Cognitive Map Learning Use arrow keys to go through green door Determine if path crosses itself Remember full path With flow and without flow 11/9/2018 IEEE HCC Keynote

Cognitive Map Memory Test Tested on memory for maze Forward test and backward test Measured task time & number of correct doors opened on first attempt Same controls as in learning phase, but without green door guides Given feedback 11/9/2018 IEEE HCC Keynote

Experimental Design Female Male Small FOV: 100 degrees Large FOV: 120 degrees Optical Flow Absent Optical Flow Present 11/9/2018 IEEE HCC Keynote

Experimental Procedure Paper folding test of spatial ability 1 practice trial + 4 test trials Satisfaction questionnaire 11/9/2018 IEEE HCC Keynote

Benefits of Optical Flow 11/9/2018 IEEE HCC Keynote

Optical Flow Helps All Users in Forward Test Backward 11/9/2018 IEEE HCC Keynote

Optical Flow Benefits Females More in the Forward Test 11/9/2018 IEEE HCC Keynote

Other Results No effects for field of view No effects for spatial ability measure Satisfaction ratings matched performance results 11/9/2018 IEEE HCC Keynote

Conclusion Optical flow cues help all users form better cognitive maps when navigating 3D virtual environments Better optical flow cues help women more than men in cognitive map formation Unexplained by biases in spatial ability Wider displays offer even better optical flow cues 100 degree field of view seems sufficient Spatial ability held constant 11/9/2018 IEEE HCC Keynote

Information Voyeurism: Social Impact of Large Displays Exploit social cues induced by physical size: Help people communicate Increase productivity on individual tasks Must quantify in order to exploit Information on large displays more public Ask user? Cannot guarantee accuracy Video? Cannot disambiguate glance from reading Exploit social cues induced by physical size: Help people communicate Increase productivity on individual tasks Must quantify in order to exploit Information on large displays more public Ask user? Cannot guarantee accuracy Video? Cannot disambiguate glance from reading 11/9/2018 IEEE HCC Keynote

Measuring ‘Peeking’ Implicit memory priming paradigm Expose user to stimulus Test user implicitly on how much they’ve processed stimulus Word stem completion Eg. Mon_____ Priming measured by faster response or higher frequency of stimulus Monkey, Money, Monster, Monday, Monopoly, … Repetition priming effects: facilitation in processing of a stimulus as a function of recent encounter with the same stimulus Without reference to the prior stimulus… First step in this experimental paradigm: Let user see the stimulus (in this case, a monkey) User can be asked to perform some action/decision on this object (just to make sure it’s being processed). Not necessary Then: Test user implicitly on how much they’ve processed stimulus. Without reference to the prior stimulus, asked to perform a task which is influenced by processing of stimulus In this case, user likely to answer with “monkey” Measure: Faster More frequent 11/9/2018 IEEE HCC Keynote

Experiment Materials Stimulus: 30 words embedded in: 7 e-mail subject lines 2 e-mail messages Place e-mail where it can be seen by user Priming test to see if they’ve read it Word stem completion We used a textual version of this priming paradigm: We embedded 30 carefully selected words in 7 e-mail subject lines and 2 e-mail msgs The priming test we used was word stem completion. Given the first 3 letters of a word and asked to complete the word as quickly as possible Measure the frequency that primed words appeared as solutions If the user had read the subject lines/e-mail msgs, they would be process the primed words and be more likely to complete the stems with these words Actually e-mail Controlled and didn’t control… New paradigm 11/9/2018 IEEE HCC Keynote

Experimental Setup Large Projection Screen Small Desktop Monitor 156″ 38″ 16″ 27.5″ 66″ 114″ Experimenter Participant Large Projection Screen Small Desktop Monitor 11/9/2018 IEEE HCC Keynote

Implicit Memory Results N=12 Average # of Target Words N=12 Small Display Large Display 11/9/2018 IEEE HCC Keynote

Other Converging Data More users admitted reading text on: Large Screen (7/12) vs. Small Screen (3/12) Comments indicated reading someone else’s e-mail more acceptable on large screen Video shows users glanced more at: Large Screen (M=19 seconds) vs. Small Screen (M=14 seconds) 11/9/2018 IEEE HCC Keynote

Design Implications Protect private information from prying eyes Private information never placed on public screens Interface conventions that convey level of privacy Facilitate ad hoc collaboration Display systems that make people interact more 11/9/2018 IEEE HCC Keynote

Progress Chart Overview of user-centered design and process Exploration of 3D UI designs that leverage human capabilities Examination of animation in UI Exploration of large displays and gender differences in 3D navigation Conclusion 11/9/2018 IEEE HCC Keynote

Conclusion Successful, advanced user interface design requires knowing the user problem to be solved As opposed to being technology-driven Leveraging human capabilities ensures improvements over existing techniques Usable designs Contributions to science Principles derived for human-computer interaction discipline 11/9/2018 IEEE HCC Keynote

Thank you…

Large Display Surfaces are Here Trend towards multiple display systems that enlarge display space Workstation in the real world (InfoCockpit--CMU) 11/9/2018 IEEE HCC Keynote

User Views of Maze Narrow field of view (100 degrees) Wide field of view (120 degrees) 11/9/2018 IEEE HCC Keynote

Visual Pivot (Rotation around Vertical Axis) B C D E F G H The user can pivot to any other dimension that the selected node participates in This shows the original visual pivot animation around a vertical axis First pivoting to the business unit hierarchy Using animation and transparency, 3D to emphasize what’s happening between the two hierarchies around the pivot point 11/9/2018 IEEE HCC Keynote

Schematic of Visual Pivot (horizontal rotation) Hierarchy 1 Hierarchy 2 Axis of rotation 11/9/2018 IEEE HCC Keynote