Novel Visualization and Interaction for Small to Jumbo Displays Mary Czerwinski Microsoft Research Mary Czerwinski Microsoft Research

Who Contributed: VIBE Team

Large Display Surfaces Are Here Workstation in the real world

Overview Initial large display research Prototypes around usability issues observed Visualization and interaction New user experiences have to scale the wide continuum of displays Future directions Initial large display research Prototypes around usability issues observed Visualization and interaction New user experiences have to scale the wide continuum of displays Future directions

Harris Poll Responses (7/02, N=1197)

no multimon 30% plan to use multimon 38% use multimon 32% Peddie Research, 2001 (N>6000) Multimon Usage Trends

Why A Larger Display Surface? Productivity benefits % (despite sw usability issues) Users prefer more display surface Prices dropping Footprints getting smaller Productivity benefits % (despite sw usability issues) Users prefer more display surface Prices dropping Footprints getting smaller

1 st Prototype--dSharp Display Triple projection matrox parhelia card 3028 x764 resolution 42 in. across Slightly curved 120 degree FOV Triple projection matrox parhelia card 3028 x764 resolution 42 in. across Slightly curved 120 degree FOV

Task Times – Significant

User Satisfaction - Significant

Windows Layout - Significant

Tenets--Large Display UX User studies show large display surfaces fundamentally change user interaction Designed tools to better understand/ complement how work practice changes Large display surfaces provide non-linear productivity increases Additional space has different utility e.g. focal/peripheral displays provide different cues User studies show large display surfaces fundamentally change user interaction Designed tools to better understand/ complement how work practice changes Large display surfaces provide non-linear productivity increases Additional space has different utility e.g. focal/peripheral displays provide different cues

But…Usability Issues Why click to bring a clearly visible window into focus? caused many errors Where is my cursor? Where is my start button? Where is my taskbar? Where are my dialogs? The software doesn’t know where the bezel is… Why click to bring a clearly visible window into focus? caused many errors Where is my cursor? Where is my start button? Where is my taskbar? Where are my dialogs? The software doesn’t know where the bezel is…

Vibelogger 1st activity repository for studying windows usage in aggregate can profile users based on display size can be extended to visualize workflow and capture context single user: capture task contexts to surface pertinent ui or provide reminders 1st activity repository for studying windows usage in aggregate can profile users based on display size can be extended to visualize workflow and capture context single user: capture task contexts to surface pertinent ui or provide reminders

Multitask Visualization colored block for each time point and app amount of shading indicates percentage of visibility of the window tasks subtasks

Task Switching Visualization switching tasks (red to blue) how are windows arranged and used? compare to... switching tasks (red to blue) how are windows arranged and used? compare to...

Windows and Task Management Issues Emerge Larger displays = more open windows Multimon users arrange windows spatially Taskbar does not scale: aggregation model not task-based users can’t operate on groups of related windows Larger displays = more open windows Multimon users arrange windows spatially Taskbar does not scale: aggregation model not task-based users can’t operate on groups of related windows

Changes in Window Access Patterns

Input: Drag-and-Pop problem large displays create long distance mouse movement touch & pen input has problems moving between screen units solution drag-and-pop brings proxies of targets to the user from across display surfaces the user can complete drag interactions locally—no need to deal with distances or to cross display borders problem large displays create long distance mouse movement touch & pen input has problems moving between screen units solution drag-and-pop brings proxies of targets to the user from across display surfaces the user can complete drag interactions locally—no need to deal with distances or to cross display borders

Table Cloth Problem: User wants to access content physically far away Solution: Pan the desktop to user Compress content to the right of focus Grab content you need and snap back Problem: User wants to access content physically far away Solution: Pan the desktop to user Compress content to the right of focus Grab content you need and snap back

Multitasking Support Projectbar, layoutbar, groupbar Scalable fabric Task flasher Secret passwords Table cloth Wincuts Projectbar, layoutbar, groupbar Scalable fabric Task flasher Secret passwords Table cloth Wincuts

Task Management: Groupbar Taskbar for lightweight grouping of windows Allows for multiple bars, spatial placement of bars Desktop snapshotting; task snapshots Taskbar for lightweight grouping of windows Allows for multiple bars, spatial placement of bars Desktop snapshotting; task snapshots

Task management: Scalable Fabric Configurable central focus + periphery Easy task switch from periphery to focus Leverages human spatial memory Configurable central focus + periphery Easy task switch from periphery to focus Leverages human spatial memory

Task Flasher A more visual alt + tab Uses 3d scaling and selection animation Windows stay on the monitor on which they are positioned A more visual alt + tab Uses 3d scaling and selection animation Windows stay on the monitor on which they are positioned

Secret Passwords Problem: many touchscreen systems (eg. SmartBoard, TabletPC) have no keyboard Use software virtual keyboard Hard to hide password on a virtual keyboard Previous research showed users think anything on large display is considered public…they watch (Tan et al, CHI ‘03) Problem: many touchscreen systems (eg. SmartBoard, TabletPC) have no keyboard Use software virtual keyboard Hard to hide password on a virtual keyboard Previous research showed users think anything on large display is considered public…they watch (Tan et al, CHI ‘03)

Solution: Secret Passwords

Meeting Support: WincutsWincuts Meeting Support: WincutsWincuts What about co-located collaborative work? People bring different expertise and information on personal devices to meetings and need to share today’s model is broken—only one person gets to display entire desktop at a time (for better or worse) or, must share applications and files with others what about private information? What about UI that should be scaled for the task at hand? What about co-located collaborative work? People bring different expertise and information on personal devices to meetings and need to share today’s model is broken—only one person gets to display entire desktop at a time (for better or worse) or, must share applications and files with others what about private information? What about UI that should be scaled for the task at hand?

Visualization Research DateLens with Ben Bederson Facetmap Team Tracks (SW Vis) DateLens with Ben Bederson Facetmap Team Tracks (SW Vis)

Datelens with Ben Bederson Fisheye representation of dates Compact overviews User control Integrated search Integrated with outlook Pen-enabled Fisheye representation of dates Compact overviews User control Integrated search Integrated with outlook Pen-enabled

FacetMap

Scalable Visualization for “all of your stuff” Queries MyLifeBits SQL database interactively Domain includes web pages, digital images, documents, , SenseCam physical activity data, VibeLog PC activity data Heterogeneous data, metadata, annotations, and relationships organized into “facets” Browsing and searching accomplished by iterative selection from among available item attributes (and/or full text search) Uses Piccolo.Net (on top of GDI+) for graphics & animation Scalable Visualization for “all of your stuff” Queries MyLifeBits SQL database interactively Domain includes web pages, digital images, documents, , SenseCam physical activity data, VibeLog PC activity data Heterogeneous data, metadata, annotations, and relationships organized into “facets” Browsing and searching accomplished by iterative selection from among available item attributes (and/or full text search) Uses Piccolo.Net (on top of GDI+) for graphics & animation

Scalability Scales in three ways: Space: Recursive, space-filling algorithm generates useful views for any size screen Items: Large numbers of items aggregated/abstracted into groups and counts at multiple levels Facets: New item attributes can be added to the visualization dynamically Fixed minimum node size to guarantee readability Consistent information density Greater levels of detail are collapsed until they are feasible to present, given display constraints Scales in three ways: Space: Recursive, space-filling algorithm generates useful views for any size screen Items: Large numbers of items aggregated/abstracted into groups and counts at multiple levels Facets: New item attributes can be added to the visualization dynamically Fixed minimum node size to guarantee readability Consistent information density Greater levels of detail are collapsed until they are feasible to present, given display constraints

Medium Size (No Filters)

Medium Size (Filters Active)

Small Size

Large Size (Wall Display)(Wall Display) Large Size (Wall Display)(Wall Display)

FacetMap for Mobile Phones Quick searching through structured data Scalable UI Half visual and half text list Prototyping stage (desktop) Multiple visualizations (pick the best for a given results set) Quick searching through structured data Scalable UI Half visual and half text list Prototyping stage (desktop) Multiple visualizations (pick the best for a given results set)

SW Visualization: Team Tracks Goal: help developers new to a code base familiarize themselves quickly Based on logs of teams using the code base Assumption: most frequently visited areas of the code are more important We empirically verified this Show developers most related areas of code based on where they currently are Also give them previews of that code Goal: help developers new to a code base familiarize themselves quickly Based on logs of teams using the code base Assumption: most frequently visited areas of the code are more important We empirically verified this Show developers most related areas of code based on where they currently are Also give them previews of that code

Screenshot of Team Tracks

Evaluation Results Better task completion rates 9 / 9 completed tasks 1 and 2 (same) 3 / 9 completed task 3 (versus 1 / 7) dominated by algorithmic detail 7 / 9 completed task 4 (versus 1 / 7) dominated by finding relevant code fragments Better quiz scores (t(16)=-2.04, p<.03, one-tailed) Same importance ratings (r=0.45, p=.02) Better task completion rates 9 / 9 completed tasks 1 and 2 (same) 3 / 9 completed task 3 (versus 1 / 7) dominated by algorithmic detail 7 / 9 completed task 4 (versus 1 / 7) dominated by finding relevant code fragments Better quiz scores (t(16)=-2.04, p<.03, one-tailed) Same importance ratings (r=0.45, p=.02)

Future VIBE Directions Novel interaction and visualization techniques that scale from small to very large displays Continued evaluation and iteration of designs from a user-centered perspective Automatic task identification For more information: Novel interaction and visualization techniques that scale from small to very large displays Continued evaluation and iteration of designs from a user-centered perspective Automatic task identification For more information:

Thank You! /research/vibe /research/vibe Thank You! /research/vibe /research/vibe