1. Large Display Research Overview
Mary Czerwinski, George Robertson, Brian Meyers, Greg Smith, Daniel Robbins & Desney Tan

2. Introduction
The increasing graphical processing power of the PC has fueled a powerful demand for larger displays
Despite the increasing affordability and availability of larger displays, most users’ display space represents less than 10% of their physical workspace area
Microsoft Research

3. Introduction continued
Current interfaces are designed around the assumption of a relatively small display providing access to a larger virtual world
How might users cope with and benefit from display devices that provide 25% to 35% of their physical desk area or perhaps one day cover entire office walls?
We evaluated usability issues for large displays and developed a series of research prototypes that address various issues we discovered
Also, large displays should and can present beautiful visualizations
Microsoft Research

4. Harris Poll responses (7/02, N=1197)
This was market research carried out with the shell MSX team, targeted at the LongHorn personas. Point of this slide is that 22% of your targeted LH personas are already running multimon!!! We think laptop users are driving this base rate now that laptops ship multimon ready.
Microsoft Research

5. Why a Larger Display Surface?
Productivity benefits 10-30% (despite sw usability issues)
Users prefer more display surface
Prices dropping fast
Footprints getting smaller
Microsoft Research

6. Productivity Study w/dSharp Display
Triple projection
Matrox parhelia card
3028 x764
resolution
42 in. across
Slightly curved
120 degree FOV
Microsoft Research

7. Task Times – Significant Benefits
Microsoft Research

8. User Satisfaction - Significant
Microsoft Research

9. Windows Layout - Significant
Microsoft Research

10. Cognitive Benefits of Large Displays
Czerwinski et al. document results showing that larger displays lead to improved recognition memory and peripheral awareness
Tan et al. demonstrate the advantages of large displays on 3D navigation in virtual worlds.
Wider fields of view lead to increased ability to process optical flow cues during navigation, cues that females are more reliant upon than males
Tan et al. also found that large displays provide for a more immersive experience when performing spatial tasks, building better cognitive maps of the virtual world
Microsoft Research

11. But…Usability Issues
Why click to bring a clearly visible window into focus? caused many errors
Where is my cursor?
Where is my start button?
Where is my taskbar?
Where are my dialogs?
The software doesn’t know where the bezel is…
Microsoft Research

12. Vibelog: How Users Interact with Displays
1st activity repository for studying windows usage in aggregate
can’t fix what you can’t measure
can profile users
can be extended
Single user: capture task contexts to surface pertinent ui or provide reminders
Microsoft Research

13. Multitasking Visualization
Colored block for each time point and app
Amount of shading indicates percentage of visibility of the window
Tasks
Subtasks
Microsoft Research

14. Task Switching Visualization
Switching tasks (red to blue)
How are windows arranged and used?
compare to...
Microsoft Research

15. Windows and Task Management Issues Emerge
More open windows
Users arrange windows spatially
Taskbar does not scale:
aggregation model not task-based
users can’t operate on groups of related windows
Microsoft Research

16. Changes in Window Access Patterns
Microsoft Research

17. Basic Usability Issues
Seven broad categories:
1. Input: Losing track of the cursor
2. Input: Distal access to information
3. Window management problems
4. Task management problems
5. Configuration problems
6. Failure to leverage the periphery
7. Failure to use displays artistically
In this overview, research prototypes will be described that address many of these problems across the research community
Microsoft Research

18. Input: High Density Cursor (Baudisch et al.)
Ask if they’ve seen—if no, then show Video
Microsoft Research

19. Input: Drag-and-Pop (Baudisch et al.)
Problem
Large displays create long distance mouse movement
Touch & pen input has problems moving between screen units
Solution
Drag-and-pop brings proxies of targets to the user from across display surfaces
The user can complete drag interactions locally—no need to deal with distances or to cross display borders
Ask if they’ve seen, if not, then show video
Microsoft Research

20. Input: Continued 1 Vacuum (Bezerianos & Balakrishnan)
Vision-tracked multi-finger gestural input (Malik, Ranjan & Balakrishnan)
Handheld projector (Forlines et al.)
Microsoft Research

21. Input: Continued 2
Vogel & Balakrishnan--Distant freehand pointing and clicking
Hand controls pointer position and makes click selection with finger or thumb
Microsoft Research

22. Input: Continued 3 Kahn et al.—”Frisbee”, a remote control UI
Grossman et al.—3D modeling techniques
Microsoft Research

23. FlowMenus and Zoomscapes
Gruimbretiere et al.
Zoomscapes allowed currently unused windows to hang around, but at 25% of their normal size
Flowmenus were pen-based, fluid interaction techniques for invoking commands at the user’s point of interest
Microsoft Research

24. Input: TableTop Interaction
MERL’s DiamondTouch system and two-handed touch gestures
Hinrichs et al. “Interface Currents” for collaboration
Microsoft Research

25. Window & Task Management Support
Table Cloth
Task flasher
LiveBoard
Task Zones
GroupBar
Scalable Fabric
Kimura
WinCuts
Microsoft Research

26. Window Management Challenges
What to do when you have your information spread out in physically large display surfaces?
Real display at Georgia Tech (Hutchings et al.)
Microsoft Research

27. Window Management: Task Flasher
A more visual alt + tab
Uses 3d scaling and zooming animation to show selected window
Windows stay on the monitor on which they are positioned
Microsoft Research

28. Window Management: Table Cloth
Problem:
User wants to access content physically far away
Solution:
Pan the desktop to user
Compress content to the right of focus
Grab content you need and snap back
Microsoft Research

29. Task Management: LiveBoard (Elrod et al.)
Pen-based group interaction around a large surface
Supported drawing, pop-up menus, selection and annotations
Boardwalk software provides “planks” or tasks, from which the user would choose
A plank automatically opened up a set of applications (e.g., meeting, scoreboard, slideshow, games, etc.)
Microsoft Research

30. TaskZones: Virtual Multimon Desktops (Hutchings et al.)
Problem: user has multiple desktops
wall of monitors
how to switch focus?
might be using a phone or remote control
Solution: TaskZones
Microsoft Research

31. Task Management: GroupBar
Taskbar for lightweight grouping of windows into tasks
Can have multiple bars for large displays
Download at (search for GroupBar)
~40,000 downloads
Desktop snapshots
Ask if they’ve seen—if not, show video
Microsoft Research

32. GroupBar Usage Study
Microsoft Research

33. Task Management: Scalable Fabric
Scaled down versions of grouped windows in periphery
Supports task switching and task reacquisition
(search for fabric); over 20,000 downloads
Scalable Fabric is an existing system developed at MSR that spatially organizing shrunken versions of windows into tasks in the periphery and enabling window management on a task level
Our study compares Scalable Fabric and 2 variations of it that use different abstraction techniques:
Change Borders (a version of change detection) & Clipping Lists (a version of semantic content extraction)
Tasks span multiple applications: this is why we used SF rather than testing isolated support for multitasking like in the earlier example. We want to test peripheral information design for any task using any application.
Scalable Fabric.
Users interact with windows in a central focus area of the screen in a normal manner, but when a user moves a window into the periphery, it shrinks.
The window “minimize” action is redefined to return the window to the user-chosen location in the periphery, rather than hiding the window.
Placing a set of windows near each other in the periphery dynamically creates a task composed of them. Clicking on a minimized task’s name restores all its windows into the central focus area.
Clicking again on the focal task’s name minimizes all its windows to the periphery.
Windows from all tasks are visible at all times (either in the focus area or as shrunken, peripheral windows), enabling easier task switching
Microsoft Research

34. Scalable Fabric Usage Study
Survey Question
(1=Disagree, 5=Agree)
TaskBar
Scalable Fabric
Task switching was easy to perform using the…
2.95
4.26
It was hard to go back and forth between my various windows and applications using…..
3.32
1.84
I was satisfied with the functionality of the ….
2.68
3.78
The TaskBar/Scalable Fabric is an attractive innovation for Windows.
3.16
4.47
Figure 9: Average task times +/- one standard error of the mean for TaskBar and Scalable Fabric.
Table 1: Average satisfaction ratings for the TaskBar and Scalable Fabric.
All ratings were significantly in favour of Scalable Fabric at the p<.05 level.
Microsoft Research

35. Task Management: Kimura (MacIntyre et al.)
Supported multitasking and background awareness using interactive peripheral displays
“Montages” or activities based on desktop interaction
Microsoft Research

36. WinCuts: Initial Motivation
Problems:
Sharing live windows/information is hard
Screen space is scarce and laying out information optimally is hard
Microsoft Research

37. WinCuts Video
Microsoft Research

38. Specify Region of Interest
Animated
advertisement
Seldom used
interface buttons
Scrolling ticker
Region of interest
Microsoft Research

39. Organize Content Rescale WinCut so graph scales are comparable
Relevant
content
Microsoft Research

40. Reconfigure Interfaces
Relevant
interface
elements
content
Microsoft Research

41. Sharing WinCuts across Machines
Click on “Share”
Specify destination (also running WinCuts)
WinCut appears on destination machine
Remote WinCuts work just like local WinCuts
Except input redirection disabled
Microsoft Research

42. Share Content when Collaborating
Microsoft Research

43. Now we’re Thinking… With remote input redirection working:
Create ad hoc remote controls and interfaces
Work across displays and devices
Laptop or
TabletPC
PDA
Cell Phone
Desktop
Microsoft Research

44. Huang & Mynatt’s Design Space for Peripheral Awareness Display Research
Microsoft Research

45. Interaction based on Distance
Vogel & Balakrishnan’s notion of an interactive, ambient, public display
Different functionality based on distance
Ambient, Implicit, Subtle and Personal spaces
Microsoft Research

46. Large Displays as Peripheral Awareness Surfaces
Brignull and Rogers’ Opinionizer
McCarthy’s et al.’s Unicast, Outcast & GroupCast
Izadi et al’s Dynamo
….etc.
Microsoft Research

47. And, Large Displays as Art and Info
Blinkenlights 2.0 in Berlin
Interactive waterfall display in children’s hospital
Weather patterns window in an art gallery at night
Etc….
Microsoft Research

48. Conclusions There is a clear trend toward larger displays
Large displays increase user productivity, aid user recognition memory, and in some cases can eliminate gender bias
User studies have identified numerous usability problems
Research prototypes were presented that outline techniques for solving many of these problems
The work of integrating these prototype solutions into one system remains to be done
Correcting these problems significantly improves the user experience on large displays
Large displays also useful tools for peripheral awareness and can be aesthetically pleasing
Microsoft Research

49. Acknowledgements Gary Starkweather Patrick Baudisch Ed Cutrell
Eric Horvitz
Jonathan Grudin
All of our colleagues doing large display research and kindly granted me permission to show their work
Microsoft Research

50. Thanks for your Attention!
Questions?
More information at:
Microsoft Research

51. References
Ballagas, R., Rohs, M. & Sheridan, J. (2005). Sweep and Point & Shoot: Phonecam-based interactions for large, public displays. In Proceedings of CHI 2005 – the ACM Conference on Human Factors in Computing Systems, pp. 1-4.
Baudisch, P., Cutrell, E., Robbins, D., Czerwinski, M., Tandler, P., Bederson, B., and Zierlinger, A. Drag-and-Pop and Drag-and-Pick: techniques for accessing remote screen content on touch- and pen-operated systems. In Proceedings of Interact 2003, pp
Baudisch, P. Cutrell, E, and Robertson, G. High-density cursor: a visualization technique that helps users keep track of fast-moving mouse cursors. In Proceedings of Interact 2003, pp
Bezerianos, A.& Balakrishan, R. (2005). Canvas Portals: View and space management on large displays. IEEE Computer Graphics and Applications, 25(4). pp
Bezerianos, A. & Balakrishnan, R. (2005). The Vacuum: Facilitating the manipulation of distant objects. Proceedings of CHI 2005 – the ACM Conference on Human Factors in Computing Systems. pp
Biehl, J.T. & Bailey, B.P. (2004). ARIS: An interface for application relocation in an interactive space. In Proceedings of Graphics Interface 2004, pp
Brignull, H. & Rogers, Y. (2003). Enticing people to interact with large public displays in public places. In Proceedings of Interact 2003, pp
Buxton, W., Fitzmaurice, G., Balakrishnan, R., Kurtenbach, G. (2000). Large displays in automotive design. IEEE Computer Graphics & Applications, July 2000.
Cao, X. & Blakrishnan, R. (2003). VisionWand: Interaction techniques for large displays using a passive wand tracked in 3D. Proceedings of UIST 2003 – the ACM Symposium on User Interface Software and Technology. pp
Czerwinski, M., Smith, G., Regan, T., Meyers, B., Robertson, G. and Starkweather, G. (2003). Toward characterizing the productivity benefits of very large displays. In Proceedings of Interact 2003, pp
Elrod, S., Bruce, R., Gold, R., Goldberg, D., Halasz, F., Janssen, W., Lee, D., McCall, K., Pedersen, E., Pier, K., Tang, J. & Welch, B. (1992). LiveBoard: A large interactive display supporting group meetings, presentations and remote collaboration. In Proceedings of CHI 1992 – the ACM Conference on Human Factors in Computing Systems, pp
Microsoft Research

52. References Continued 2 Microsoft Research
Grossman, T., Balakrishnan, R., Kurtenbach, G., Fitzmaurice, G., Khan, A. & Buxton, W.. (2001). Interaction techniques for 3D modeling on large displays. Proceedings of the ACM Symposium on Interactive 3D Graphics (I3DG2001), pp New York: ACM.
Grudin, J. (2002). Partitioning digital worlds: Focal and peripheral awareness in multiple monitor use. In Proceedings of CHI 2002 – the ACM Conference on Human Factors in Computing Systems, pp
Guimbretiere, F., Stone, M., and Winograd, T. (2001). Fluid interaction with high-resolution wall-size displays. In Proceedings of UIST 2001, pp
Huang, E.M., Russell, D.M. & Sue, A.E. (2004). IM Here: Public instant messaging on large, shared displays for workgroup interactions. In Proceedings of CHI 2004 – the ACM Conference on Human Factors in Computing Systems, pp
Hutchings, D., Czerwinski, M., Smith, G., Meyers, B., Robertson, G Display space usage and window management operation comparisons between single monitor and multiple monitor users. In Proceedings of AVI 2004, pp
Khan, A., Fitzmaurice, G., Almeida, D., Burtnyk, N. & Kurtenbach, G. (2003). A remote control interface for large displays. ACM UIST 2003 Symposium on User Interface Software & Technology, pp
Khan, A. Matejka, J. Fitzmaurice, G. & Kurtenbach, G. (2005). Spotlight: Directing users' visual attention on large displays. Proceedings of CHI 2005 – the ACM Conference on Human Factors in Computing Systems pp. 791 – 798.
MacIntyre, B., Mynatt, E., Voida, S., Hansen, K., Tullio, J., Corso, G. (2001). Support for multitasking and background awareness using interactive peripheral displays. In ACM UIST 2001 Symposium on User Interface Software & Technology, pp
Malik, S., Ranjan, A. & Balakrishnan, R. (2005). Interacting with large displays from a distance with vision-tracked multi-finger gestural input. Proceedings of UIST the ACM Symposium on User Interface Software and Technology. pp
McCarthy, J.F., Costa, T.J. & Liongosari, E.S. (2001). UniCast, OutCast & GroupCast: Three steps toward ubiquitous, peripheral displays. In Lecture Notes in Computer Science, Vol. 2201, pp. 332-*, Germany: Springer-Verlag.
Mynatt, E., Igarashi, T., Edwards, W., and LaMarca, A. (1999). Flatland: new dimensions in office whiteboards. In Proceedings of CHI 1999 – the ACM Conference on Human Factors in Computing Systems, pp
Paradiso, J.A., Leo, C.K., Checka, N. & Hsiao, K. (2002). Passive acoustic sensing for tracking knocks atop large interactive displays. In Proceedings of the IEEE Sensors 2002 Conference, pp. 1-6.
Pedersen, E., McCall, K., Moran, T., Halasz, F.G. (1993). Tivoli: An electronic whiteboard for informal workgroup meetings. In Proceedings of CHI 1993 – the ACM Conference on Human Factors in Computing Systems, pp
Robertson, G.G., Czerwinski, M., Baudisch, P., Meyers, B., Robbins, D., Smith, G., and Tan, D. (2005).  Large display user experience.  In IEEE CG&A special issue on large displays, 25(4), pp
Robertson, G., Horvitz, E., Czerwinski, M., Baudisch, P., Hutchings, D., Meyers, B., Robbins, D., and Smith, G. (2004), Scalable Fabric: Flexible Task Management. In Proceedings of AVI 2004 pp
Microsoft Research

53. References Continued 3
Russell, D. M., Drews, C. & Sue, A. (2002). Social aspects of using large interactive displays for collaboration. In Proceedings of UbiComp 2002: Ubiquitous Computing: 4th International Conference, pp
Smith, G., Baudisch, P., Robertson, G., Czerwinski, M., Meyers, B., Robbins, D., and Andrews, D. (2003). GroupBar: The TaskBar Evolved. In Proceedings of OZCHI 2003, pp
Streitz, N.A., Geiler, J. & Holmer, T. (1998). Roomware for cooperative buildings: Integrated design of architectural spaces and information spaces. In Lecture Notes in Computer Science, Vol. 1370, pp. 4-*, Germany: Springer-Verlag.
Swaminathan, K. & Sato, K. (1997). Interaction design for large displays. In ACM’s Interactions (4), pp
Tan, D., Czerwinski, M., and Robertson, G. (2003). Women Go With the (Optical) Flow. In Proceedings of CHI 2003 – the ACM Conference on Human Factors in Computing Systems, pp
Tan, D.S. and Czerwinski, M. (2003). Information Voyeurism: Social impact of physically large displays on information privacy. Proceedings of CHI 2003 – the ACM Conference on Human Factors in Computing Systems. pp
Tan, D.S., Gergle, D., Scupelli, P. & Pausch, R. (2003). With similar viewing angles, larger displays improve spatial performance. In Proceedings of CHI 2003 – the ACM Conference on Human Factors in Computing Systems, pp
Tan, D.S., Gergle, D., Scupelli, P. & Pausch, R. (2004). Physically large displays improve path integration in 3D virtual navigation tasks. In Proceedings of CHI 2004 – the ACM Conference on Human Factors in Computing Systems, pp
Tan, D.S., Meyers, B. & Czerwinski, M. (2004). WinCuts: Manipulating arbitrary window regions for more effective use of screen space. In Extended Abstracts of Proceedings of CHI 2004, pp
Vogel, D. & Balakrishnan, R. (2005). Distant freehand pointing and clicking on very large high resolution displays. Proceedings of UIST the ACM Symposium on User Interface Software and Technology. pp
Vogel, D. & Balakrishnan, R. (2004). Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users. Proceedings of UIST 2004 – the ACM Symposium on User Interface Software and Technology. pp
Microsoft Research