Large Display Research Overview Mary Czerwinski, George Robertson, Brian Meyers, Greg Smith, Daniel Robbins & Desney Tan

Microsoft Research2 Introduction  The increasing graphical processing power of the PC has fueled a powerful demand for larger displays  Despite the increasing affordability and availability of larger displays, most users’ display space represents less than 10% of their physical workspace area

Microsoft Research3 Introduction continued  Current interfaces are designed around the assumption of a relatively small display providing access to a larger virtual world  How might users cope with and benefit from display devices that provide 25% to 35% of their physical desk area or perhaps one day cover entire office walls?  We evaluated usability issues for large displays and developed a series of research prototypes that address various issues we discovered  Also, large displays should and can present beautiful visualizations

Microsoft Research4 Harris Poll responses (7/02, N=1197)

Microsoft Research5 Why a Larger Display Surface?  Productivity benefits 10-30% (despite sw usability issues)  Users prefer more display surface  Prices dropping fast  Footprints getting smaller

Microsoft Research6 Productivity Study w/dSharp Display  Triple projection  Matrox parhelia card  3028 x764 resolution  42 in. across  Slightly curved  120 degree FOV

Microsoft Research7 Task Times – Significant Benefits

Microsoft Research8 User Satisfaction - Significant

Microsoft Research9 Windows Layout - Significant

Microsoft Research10 Cognitive Benefits of Large Displays  Czerwinski et al. document results showing that larger displays lead to improved recognition memory and peripheral awareness  Tan et al. demonstrate the advantages of large displays on 3D navigation in virtual worlds. Wider fields of view lead to increased ability to process optical flow cues during navigation, cues that females are more reliant upon than males Tan et al. also found that large displays provide for a more immersive experience when performing spatial tasks, building better cognitive maps of the virtual world

Microsoft Research11 But…Usability Issues  Why click to bring a clearly visible window into focus? caused many errors  Where is my cursor?  Where is my start button?  Where is my taskbar?  Where are my dialogs?  The software doesn’t know where the bezel is…

Microsoft Research12 Vibelog: How Users Interact with Displays  1st activity repository for studying windows usage in aggregate can’t fix what you can’t measure can profile users can be extended  Single user: capture task contexts to surface pertinent ui or provide reminders

Microsoft Research13 Multitasking Visualization  Colored block for each time point and app  Amount of shading indicates percentage of visibility of the window  Tasks  Subtasks

Microsoft Research14 Task Switching Visualization  Switching tasks (red to blue)  How are windows arranged and used?  compare to...

Microsoft Research15 Windows and Task Management Issues Emerge  More open windows  Users arrange windows spatially  Taskbar does not scale: aggregation model not task-based users can’t operate on groups of related windows

Microsoft Research16 Changes in Window Access Patterns

Microsoft Research17 Basic Usability Issues Seven broad categories: 1. Input: Losing track of the cursor 2. Input: Distal access to information 3. Window management problems 4. Task management problems 5. Configuration problems 6. Failure to leverage the periphery 7. Failure to use displays artistically In this overview, research prototypes will be described that address many of these problems across the research community

Microsoft Research18 Input: High Density Cursor (Baudisch et al.)

Microsoft Research19 Input: Drag-and-Pop (Baudisch et al.) Problem  Large displays create long distance mouse movement  Touch & pen input has problems moving between screen units Solution  Drag-and-pop brings proxies of targets to the user from across display surfaces  The user can complete drag interactions locally—no need to deal with distances or to cross display borders

Microsoft Research20 Input: Continued 1  Vacuum (Bezerianos & Balakrishnan)  Vision-tracked multi- finger gestural input (Malik, Ranjan & Balakrishnan)  Handheld projector (Forlines et al.)

Microsoft Research21 Input: Continued 2  Vogel & Balakrishnan--Distant freehand pointing and clicking Hand controls pointer position and makes click selection with finger or thumb

Microsoft Research22 Input: Continued 3  Kahn et al.—”Frisbee”, a remote control UI  Grossman et al.—3D modeling techniques

Microsoft Research23 FlowMenus and Zoomscapes  Gruimbretiere et al. Zoomscapes allowed currently unused windows to hang around, but at 25% of their normal size Flowmenus were pen-based, fluid interaction techniques for invoking commands at the user’s point of interest

Microsoft Research24 Input: TableTop Interaction  MERL’s DiamondTouch system and two- handed touch gestures  Hinrichs et al. “Interface Currents” for collaboration

Microsoft Research25 Window & Task Management Support  Table Cloth  Task flasher  LiveBoard  Task Zones  GroupBar  Scalable Fabric  Kimura  WinCuts

Microsoft Research26 Window Management Challenges  What to do when you have your information spread out in physically large display surfaces?  Real display at Georgia Tech (Hutchings et al.)

Microsoft Research27 Window Management: Task Flasher  A more visual alt + tab  Uses 3d scaling and zooming animation to show selected window  Windows stay on the monitor on which they are positioned

Microsoft Research28 Window Management: Table Cloth  Problem: User wants to access content physically far away  Solution: Pan the desktop to user Compress content to the right of focus Grab content you need and snap back

Microsoft Research29 Task Management: LiveBoard (Elrod et al.)  Pen-based group interaction around a large surface  Supported drawing, pop- up menus, selection and annotations  Boardwalk software provides “planks” or tasks, from which the user would choose  A plank automatically opened up a set of applications (e.g., meeting, scoreboard, slideshow, games, etc.)

Microsoft Research30 TaskZones: Virtual Multimon Desktops (Hutchings et al.)  Problem: user has multiple desktops wall of monitors how to switch focus? might be using a phone or remote control  Solution: TaskZones

Microsoft Research31 Task Management: GroupBar  Taskbar for lightweight grouping of windows into tasks  Can have multiple bars for large displays  Download at /downloads/default.aspx (search for GroupBar) /downloads/default.aspx  ~40,000 downloads  Desktop snapshots

Microsoft Research32 GroupBar Usage Study

Microsoft Research33 Task Management: Scalable Fabric  Scaled down versions of grouped windows in periphery  Supports task switching and task reacquisition  fault.aspx (search for fabric); over 20,000 downloads fault.aspx

Microsoft Research34 Scalable Fabric Usage Study Figure 9: Average task times +/- one standard error of the mean for TaskBar and Scalable Fabric. Survey Question (1=Disagree, 5=Agree) TaskBarScalable Fabric Task switching was easy to perform using the… It was hard to go back and forth between my various windows and applications using… I was satisfied with the functionality of the … The TaskBar/Scalable Fabric is an attractive innovation for Windows Table 1: Average satisfaction ratings for the TaskBar and Scalable Fabric. All ratings were significantly in favour of Scalable Fabric at the p<.05 level.

Microsoft Research35 Task Management: Kimura (MacIntyre et al.)  Supported multitasking and background awareness using interactive peripheral displays  “Montages” or activities based on desktop interaction

Microsoft Research36 WinCuts: Initial Motivation  Problems: Sharing live windows/information is hard Screen space is scarce and laying out information optimally is hard

Microsoft Research37 WinCuts VideoVideo

Microsoft Research38 Specify Region of Interest Animated advertisement Seldom used interface buttons Scrolling ticker Region of interest

Microsoft Research39 Organize Content Relevant content Rescale WinCut so graph scales are comparable

Microsoft Research40 Reconfigure Interfaces Relevant interface elements Relevant content

Microsoft Research41 Sharing WinCuts across Machines 1. Click on “Share” 2. Specify destination (also running WinCuts) 3. WinCut appears on destination machine  Remote WinCuts work just like local WinCuts Except input redirection disabled

Microsoft Research42 Share Content when Collaborating

Microsoft Research43 Desktop Laptop or TabletPC PDA Cell Phone Now we’re Thinking…  With remote input redirection working: Create ad hoc remote controls and interfaces Work across displays and devices

Microsoft Research44 Huang & Mynatt’s Design Space for Peripheral Awareness Display Research

Microsoft Research45 Interaction based on Distance  Vogel & Balakrishnan’s notion of an interactive, ambient, public display  Different functionality based on distance Ambient, Implicit, Subtle and Personal spaces

Microsoft Research46 Large Displays as Peripheral Awareness Surfaces  Brignull and Rogers’ Opinionizer  McCarthy’s et al.’s Unicast, Outcast & GroupCast  Izadi et al’s Dynamo  ….etc.

Microsoft Research47 And, Large Displays as Art and Info  Blinkenlights 2.0 in Berlin  Interactive waterfall display in children’s hospital  Weather patterns window in an art gallery at night  Etc….

Microsoft Research48 Conclusions  There is a clear trend toward larger displays  Large displays increase user productivity, aid user recognition memory, and in some cases can eliminate gender bias  User studies have identified numerous usability problems  Research prototypes were presented that outline techniques for solving many of these problems  The work of integrating these prototype solutions into one system remains to be done  Correcting these problems significantly improves the user experience on large displays  Large displays also useful tools for peripheral awareness and can be aesthetically pleasing

Microsoft Research49 Acknowledgements  Gary Starkweather  Patrick Baudisch  Ed Cutrell  Eric Horvitz  Jonathan Grudin  All of our colleagues doing large display research and kindly granted me permission to show their work

Microsoft Research50 Thanks for your Attention!  Questions?  More information at:

Microsoft Research51 References  Ballagas, R., Rohs, M. & Sheridan, J. (2005). Sweep and Point & Shoot: Phonecam-based interactions for large, public displays. In Proceedings of CHI 2005 – the ACM Conference on Human Factors in Computing Systems, pp  Baudisch, P., Cutrell, E., Robbins, D., Czerwinski, M., Tandler, P., Bederson, B., and Zierlinger, A. Drag- and-Pop and Drag-and-Pick: techniques for accessing remote screen content on touch- and pen-operated systems. In Proceedings of Interact 2003, pp  Baudisch, P. Cutrell, E, and Robertson, G. High-density cursor: a visualization technique that helps users keep track of fast-moving mouse cursors. In Proceedings of Interact 2003, pp  Bezerianos, A.& Balakrishan, R. (2005). Canvas Portals: View and space management on large displays. IEEE Computer Graphics and Applications, 25(4). pp Canvas Portals: View and space management on large displays  Bezerianos, A. & Balakrishnan, R. (2005). The Vacuum: Facilitating the manipulation of distant objects. Proceedings of CHI 2005 – the ACM Conference on Human Factors in Computing Systems. pp The Vacuum: Facilitating the manipulation of distant objects  Biehl, J.T. & Bailey, B.P. (2004). ARIS: An interface for application relocation in an interactive space. In Proceedings of Graphics Interface 2004, pp  Brignull, H. & Rogers, Y. (2003). Enticing people to interact with large public displays in public places. In Proceedings of Interact 2003, pp  Buxton, W., Fitzmaurice, G., Balakrishnan, R., Kurtenbach, G. (2000). Large displays in automotive design. IEEE Computer Graphics & Applications, July  Cao, X. & Blakrishnan, R. (2003). VisionWand: Interaction techniques for large displays using a passive wand tracked in 3D. Proceedings of UIST 2003 – the ACM Symposium on User Interface Software and Technology. pp  Czerwinski, M., Smith, G., Regan, T., Meyers, B., Robertson, G. and Starkweather, G. (2003). Toward characterizing the productivity benefits of very large displays. In Proceedings of Interact 2003, pp  Elrod, S., Bruce, R., Gold, R., Goldberg, D., Halasz, F., Janssen, W., Lee, D., McCall, K., Pedersen, E., Pier, K., Tang, J. & Welch, B. (1992). LiveBoard: A large interactive display supporting group meetings, presentations and remote collaboration. In Proceedings of CHI 1992 – the ACM Conference on Human Factors in Computing Systems, pp

Microsoft Research52 References Continued 2  Grossman, T., Balakrishnan, R., Kurtenbach, G., Fitzmaurice, G., Khan, A. & Buxton, W.. (2001). Interaction techniques for 3D modeling on large displays. Proceedings of the ACM Symposium on Interactive 3D Graphics (I3DG2001), pp New York: ACM. Interaction techniques for 3D modeling on large displays  Grudin, J. (2002). Partitioning digital worlds: Focal and peripheral awareness in multiple monitor use. In Proceedings of CHI 2002 – the ACM Conference on Human Factors in Computing Systems, pp  Guimbretiere, F., Stone, M., and Winograd, T. (2001). Fluid interaction with high-resolution wall-size displays. In Proceedings of UIST 2001, pp  Huang, E.M., Russell, D.M. & Sue, A.E. (2004). IM Here: Public instant messaging on large, shared displays for workgroup interactions. In Proceedings of CHI 2004 – the ACM Conference on Human Factors in Computing Systems, pp  Hutchings, D., Czerwinski, M., Smith, G., Meyers, B., Robertson, G Display space usage and window management operation comparisons between single monitor and multiple monitor users. In Proceedings of AVI 2004, pp  Khan, A., Fitzmaurice, G., Almeida, D., Burtnyk, N. & Kurtenbach, G. (2003). A remote control interface for large displays. ACM UIST 2003 Symposium on User Interface Software & Technology, pp  Khan, A. Matejka, J. Fitzmaurice, G. & Kurtenbach, G. (2005). Spotlight: Directing users' visual attention on large displays. Proceedings of CHI 2005 – the ACM Conference on Human Factors in Computing Systems pp. 791 – 798.  MacIntyre, B., Mynatt, E., Voida, S., Hansen, K., Tullio, J., Corso, G. (2001). Support for multitasking and background awareness using interactive peripheral displays. In ACM UIST 2001 Symposium on User Interface Software & Technology, pp  Malik, S., Ranjan, A. & Balakrishnan, R. (2005). Interacting with large displays from a distance with vision-tracked multi- finger gestural input. Proceedings of UIST the ACM Symposium on User Interface Software and Technology. pp Interacting with large displays from a distance with vision-tracked multi- finger gestural input  McCarthy, J.F., Costa, T.J. & Liongosari, E.S. (2001). UniCast, OutCast & GroupCast: Three steps toward ubiquitous, peripheral displays. In Lecture Notes in Computer Science, Vol. 2201, pp. 332-*, Germany: Springer-Verlag.  Mynatt, E., Igarashi, T., Edwards, W., and LaMarca, A. (1999). Flatland: new dimensions in office whiteboards. In Proceedings of CHI 1999 – the ACM Conference on Human Factors in Computing Systems, pp  Paradiso, J.A., Leo, C.K., Checka, N. & Hsiao, K. (2002). Passive acoustic sensing for tracking knocks atop large interactive displays. In Proceedings of the IEEE Sensors 2002 Conference, pp  Pedersen, E., McCall, K., Moran, T., Halasz, F.G. (1993). Tivoli: An electronic whiteboard for informal workgroup meetings. In Proceedings of CHI 1993 – the ACM Conference on Human Factors in Computing Systems, pp  Robertson, G.G., Czerwinski, M., Baudisch, P., Meyers, B., Robbins, D., Smith, G., and Tan, D. (2005). Large display user experience. In IEEE CG&A special issue on large displays, 25(4), pp Large display user experience  Robertson, G., Horvitz, E., Czerwinski, M., Baudisch, P., Hutchings, D., Meyers, B., Robbins, D., and Smith, G. (2004), Scalable Fabric: Flexible Task Management. In Proceedings of AVI 2004 pp

Microsoft Research53 References Continued 3  Russell, D. M., Drews, C. & Sue, A. (2002). Social aspects of using large interactive displays for collaboration. In Proceedings of UbiComp 2002: Ubiquitous Computing: 4th International Conference, pp  Smith, G., Baudisch, P., Robertson, G., Czerwinski, M., Meyers, B., Robbins, D., and Andrews, D. (2003). GroupBar: The TaskBar Evolved. In Proceedings of OZCHI 2003, pp  Streitz, N.A., Geiler, J. & Holmer, T. (1998). Roomware for cooperative buildings: Integrated design of architectural spaces and information spaces. In Lecture Notes in Computer Science, Vol. 1370, pp. 4-*, Germany: Springer-Verlag.  Swaminathan, K. & Sato, K. (1997). Interaction design for large displays. In ACM’s Interactions (4), pp  Tan, D., Czerwinski, M., and Robertson, G. (2003). Women Go With the (Optical) Flow. In Proceedings of CHI 2003 – the ACM Conference on Human Factors in Computing Systems, pp  Tan, D.S. and Czerwinski, M. (2003). Information Voyeurism: Social impact of physically large displays on information privacy. Proceedings of CHI 2003 – the ACM Conference on Human Factors in Computing Systems. pp  Tan, D.S., Gergle, D., Scupelli, P. & Pausch, R. (2003). With similar viewing angles, larger displays improve spatial performance. In Proceedings of CHI 2003 – the ACM Conference on Human Factors in Computing Systems, pp  Tan, D.S., Gergle, D., Scupelli, P. & Pausch, R. (2004). Physically large displays improve path integration in 3D virtual navigation tasks. In Proceedings of CHI 2004 – the ACM Conference on Human Factors in Computing Systems, pp  Tan, D.S., Meyers, B. & Czerwinski, M. (2004). WinCuts: Manipulating arbitrary window regions for more effective use of screen space. In Extended Abstracts of Proceedings of CHI 2004, pp  Vogel, D. & Balakrishnan, R. (2005). Distant freehand pointing and clicking on very large high resolution displays. Proceedings of UIST the ACM Symposium on User Interface Software and Technology. pp Distant freehand pointing and clicking on very large high resolution displays.  Vogel, D. & Balakrishnan, R. (2004). Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users. Proceedings of UIST 2004 – the ACM Symposium on User Interface Software and Technology. pp Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users.