1. Jörg Müller, Michael Nischt Hans-joerg.mueller@telekom.de Deutsche Telekom Laboratories, TU Berlin Visualization and Interaction Techniques for Small Displays Mobile and Physical Interaction, WS 2010/2011

2. Mobile and Physical Interaction2WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Preview Showing content on small displays Locating off-screen objects Behind-the-device interaction Improving touch screen accuracy

3. Showing Content on Small Displays

4. Mobile and Physical Interaction4WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Web Pages Don’t fit on Small Screens

5. Mobile and Physical Interaction5WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Web Pages Don’t fit on Small Screens Solution approaches –Device-specific authoring –Automatic re-authoring –Client-side navigation Tap to zoom into region –Uses HTML DOM model iPhone zooms into columns –Double tap

6. Mobile and Physical Interaction6WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Overview + Detail Show details around cursor region But... –Detail region takes up valuable screen space –Overview region not readable –How to know what to highlight?

7. Mobile and Physical Interaction7WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Browsing Web Content Reorganize content in narrow vertical strip –Avoids horizontal scrolling 1-D browsing Narrow layout –Width = display width –Compact layout –Original layout destroyed –Little overview

8. Mobile and Physical Interaction8WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Context in Focus Display Nokia MiniMap –Overview is semi-transparently shown in detail view while scrolling –Content scaled but original layout preserved Even overview does not show whole page Detail view Indication of detail area Overview

9. Mobile and Physical Interaction9WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Nokia Symbian Browser Overview embedded in detail view Semi transparent detail view Site-specific adaptations

10. Mobile and Physical Interaction10WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Android Browser

11. Mobile and Physical Interaction11WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Collapse-to-Zoom Zoom into arbitrary rectangular areas User’s knowledge about relevance of areas relevant  zoom in irrelevant  collapse Baudisch, Xie, Wang, Ma: Collapse- to-zoom: Viewing web pages on small screen devices by interactively removing irrelevant content. UIST '04.

12. Mobile and Physical Interaction12WS 2010/2011Jörg Müller, Michael Nischt, T-Labs collapse-cell expand-cell collapse-column expand&zoom expand collapse an main com ds m Collapse-to-Zoom Gestures Similar to marking menus Browser gestures

13. Mobile and Physical Interaction13WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Thumbnail view: unreadably small Collapse-to-Zoom Walkthrough

14. Mobile and Physical Interaction14WS 2010/2011Jörg Müller, Michael Nischt, T-Labs User collapses “archive” column Collapse-to-Zoom Walkthrough

15. Mobile and Physical Interaction15WS 2010/2011Jörg Müller, Michael Nischt, T-Labs User collapses “menu” column Collapse-to-Zoom Walkthrough

16. Mobile and Physical Interaction16WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Content area expands more and is now readable Collapse-to-Zoom Walkthrough

17. Mobile and Physical Interaction17WS 2010/2011Jörg Müller, Michael Nischt, T-Labs User can follow links directly from within overview Collapse-to-Zoom Walkthrough

18. Mobile and Physical Interaction18WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Expand-and-zoom gesture… Collapse-to-Zoom Walkthrough

19. Mobile and Physical Interaction19WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Collapse-to-Zoom Walkthrough Leads to fully readable detail view

20. Mobile and Physical Interaction20WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Collapse-to-Zoom Placeholders for collapsed areas Bookmarking of collapsed state Video

21. Mobile and Physical Interaction21WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Summary Thumbnails unreadable readable Lam, Baudisch: Summary Thumbnails: Readable Overviews For Small Screen Web Browsers. CHI 2005. Video

22. Mobile and Physical Interaction22WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Summary Thumbnails – Process HTML For each object on the web page –Count # of lines –Increase font size –Reduce text to preserve # of lines Text reduction strategies –Remove words from the end –Remove most commonly occurring word (frequency dictionary) reduce text to fit scale font up original page

23. Mobile and Physical Interaction23WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Summary Thumbnails – Benefits No horizontal scrolling Preservation of original page layout Readable text But: Missing text Impact? User study hypotheses –Preserving layout is important to locate information –Readable text reduces the need to zoom –Effects of text reduction are acceptable

24. Mobile and Physical Interaction24WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Summary Thumbnails – User Study Design –Within subject Interfaces –Thumbnail –Summary thumbnail –Single-column –Desktop Dependent variables –Task time and accuracy –Amount of zooming –Amount of scrolling

25. Mobile and Physical Interaction25WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Ready To Start Task scenario Task summary

26. Mobile and Physical Interaction26WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Ready To Start Thumbnail interface

27. Mobile and Physical Interaction27WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Ready To Start Summary thumbnail interface

28. Mobile and Physical Interaction28WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Ready To Start Single column interface

29. Mobile and Physical Interaction29WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Ready To Start Desktop interface

30. Mobile and Physical Interaction30WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Summary Thumbnails – Study Results Task times –Summary thumbnails significantly different from single column Number of zooming events –Significantly different 0 5 10 15 20 25 ThumbnailSummary Th.Single ColumnDesktop Mean task times (s)

31. Mobile and Physical Interaction31WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Summary Thumbnails – Study Results Error rates User preference Thumbnailsummary Thumbnail Single Column # of participants 0 2 4 6 8 10 ThumbnailSummary Single ThumbnailColumn

32. Mobile and Physical Interaction32WS 2010/2011Jörg Müller, Michael Nischt, T-Labs DateLens (Bederson et al., 2004) Calendar with fisheye view and semantic zoom Bederson, Clamage, Czerwinski, Robertson: DateLens: A Fisheye Calendar Interface for PDAs. ACM TOCHI, 2004.

33. Locating Off-Screen Objects

34. Mobile and Physical Interaction34WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Halo (Baudisch & Rosenholtz, 2003) Source: Patrick Baudisch Baudisch, Rosenholtz: Halo: A Technique for Visualizing Off-Screen Locations. CHI 2003.

35. Mobile and Physical Interaction35WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Streetlamp Metaphor Aura visible from distance Aura is round Overlapping auras aggregate Fading of aura indicates distance Source: Patrick Baudisch

36. Mobile and Physical Interaction36WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Source: Patrick Baudisch Arrow-Based Techniques Games, maps Cinematography –Partially out of the frame  halos Requirements for “Partially out of the frame” –Mentally fill in missing parts –Convey position in space

37. Mobile and Physical Interaction37WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Shipley and Kellman 1992 Source: Patrick Baudisch Gestalt Laws: Perceptual Completion

38. Mobile and Physical Interaction38WS 2010/2011Jörg Müller, Michael Nischt, T-Labs arc/arrow fading off same selectable size halo ring distance from display border legend Source: Patrick Baudisch User Study: Halos vs. Arrows

39. Mobile and Physical Interaction39WS 2010/2011Jörg Müller, Michael Nischt, T-Labs click at expected location of off-screen targets Source: Patrick Baudisch 1. Locate Task

40. Mobile and Physical Interaction40WS 2010/2011Jörg Müller, Michael Nischt, T-Labs click arrow/arc or off-screen location closest to car Source: Patrick Baudisch 2. Closest Task

41. Mobile and Physical Interaction41WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Source: Patrick Baudisch 3. Traverse Task click on all targets in order, so as to form the shortest delivery path, beginning at the car

42. Mobile and Physical Interaction42WS 2010/2011Jörg Müller, Michael Nischt, T-Labs click on hospital farthest away from traffic jams Source: Patrick Baudisch 4. Avoidance Task

43. Mobile and Physical Interaction43WS 2010/2011Jörg Müller, Michael Nischt, T-Labs TaskArrow interfaceHalo interface Locate20.1 (7.3)16.8 (6.7) Closest9.9 (10.1)6.6 (5.3) Traverse20.6 (14.1)16.8 (8.7) Avoid9.2 (4.7)7.7 (5.8) Source: Patrick Baudisch Task Completion Time

44. Mobile and Physical Interaction44WS 2010/2011Jörg Müller, Michael Nischt, T-Labs TaskArrow interfaceHalo interface Locate23.5 pixels (21.6)28.4 pixels (33.8) Closest22% (42%)21% (41%) Traverse97.4 pixels (94.7)81.0 pixels (96.7) Avoid15% (35%)14% (34%) Source: Patrick Baudisch Error Rate Participants underestimated distances by 26% Participants saw ovals To compensate: width += 35%

45. Mobile and Physical Interaction45WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Source: Patrick Baudisch Subjective Preference

46. Mobile and Physical Interaction46WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Limitation of Halo: Clutter Clutter from overlapping or large number of halos Wedge: Isosceles triangles –Legs point towards target –Rotation, aperture No overlap –Layout algorithm adapts rotation and aperture Gustafson, Baudisch, Gutwin, Irani: Wedge: Clutter-Free Visualization of Off-Screen Locations. CHI 2008.

47. Mobile and Physical Interaction47WS 2010/2011Jörg Müller, Michael Nischt, T-Labs The Wedge Degrees of freedom –Rotation –Intrusion –Aperture

48. Mobile and Physical Interaction48WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Avoiding Overlap

49. Mobile and Physical Interaction49WS 2010/2011Jörg Müller, Michael Nischt, T-Labs User Study: Halos vs. Wedges Locate Avoid

50. Mobile and Physical Interaction50WS 2010/2011Jörg Müller, Michael Nischt, T-Labs User Study: Halos vs. Wedges Closest Subjective preferences

51. Behind-the-Device Interaction

52. Mobile and Physical Interaction52WS 2010/2011Jörg Müller, Michael Nischt, T-Labs LucidTouch Behind-the-device multitouch input Pseudo transparency –Enabling back of the device pointing –3 states + visual feedback = land-on selection Form-factor –Enabling multi-touch with all ten fingers Wigdor, Forlines, Baudisch, Barnwell, Shen: LucidTouch: A See-Through Mobile Device. UIST'07. Video

53. Mobile and Physical Interaction53WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Why Behind-the-Device Interaction? Avoid occlusion “Fat finger” problem

54. Mobile and Physical Interaction54WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Pseudo Transparency Show finger “shadows” as cues physical see-throughcamera see-through

55. Mobile and Physical Interaction55WS 2010/2011Jörg Müller, Michael Nischt, T-Labs LucidTouch Camera See-Through Finger shapes and positions tracked by camera –Hovering state (Multi-)touch detected by pad –Tracking state

56. Mobile and Physical Interaction56WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Out of Range Tracking Dragging State 0 State 1 State 2 Lift Mouse Replace Mouse Depress Button Release Button Buxton’s Three-State Model of Input Buxton: A Three-State Model of Graphical Input. In Proc. of INTERACT ’90.

57. Mobile and Physical Interaction57WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Passive Tracking Dragging State 2 State 0 Contact Release Contact Two-State Model for Touch Input

58. Mobile and Physical Interaction58WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Out of Range Tracking Dragging State 0 State 1 State 2 Contact Release Contact Finger Behind Back Remove Finger LucidTouch: Three Input States

59. Mobile and Physical Interaction59WS 2010/2011Jörg Müller, Michael Nischt, T-Labs LucidTouch Applications

60. Mobile and Physical Interaction60WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Back-of-Device Interaction Works for Very Small Screens Jewelry, watches, etc. Pseudo transparency –Capacitive touch pad –Clickable touch pad Baudisch, Chu: Back-of-Device Interaction Allows Creating Very Small Touch Devices. CHI'09.

61. Mobile and Physical Interaction61WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Pressure-Sensitive Two-Sided Multitouch Interaction Metaphor –Holding an object between thumb and index finger Common in everyday interactions –Grabbing, sliding, twisting, turning Local input with high expressivity –Precise pressure –Thumb - index finger positions

62. Mobile and Physical Interaction62WS 2010/2011Jörg Müller, Michael Nischt, T-Labs iPhone Sandwich: Three Input States Out of range Tracking Dragging State 0 State 1 State 2 Release contact Contact Increase pressure Decrease pressure State 2 State 2 State 2+ Increase pressure Decrease pressure Buxton: A Three-State Model of Graphical Input. In Proc. of INTERACT ’90.

63. Mobile and Physical Interaction63WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Pressure-Sensitive Map Zooming Essl, Rohs, Kratz: Squeezing the Sandwich: A Mobile Pressure- Sensitive Two-Sided Multi-Touch Prototype. Demo at UIST’09.

64. Mobile and Physical Interaction64WS 2010/2011Jörg Müller, Michael Nischt, T-Labs

65. Mobile and Physical Interaction65WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Virtual Trackballs for 3D Object Rotation x,y,z-axis rotation z-axis rotation x,y,z-axis rotation θ θ θ θ z-axis rotation

66. Mobile and Physical Interaction66WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Extension of Virtual Trackball to Back of Device x y z o Full sphere operated from both sides instead of hemisphere operated from front

67. Mobile and Physical Interaction67WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Gaussian Curve Avoids Discontinuity Gaussian curve avoids discontinuity Extension to back side

68. Mobile and Physical Interaction68WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Simultaneous Thumb and Index Finger Simultaneous interaction with thumb and index finger –Shift center of virtual sphere Works for larger display sizes –Specific control widgets

69. Mobile and Physical Interaction69WS 2010/2011Jörg Müller, Michael Nischt, T-Labs

70. Mobile and Physical Interaction70WS 2010/2011Jörg Müller, Michael Nischt, T-Labs User Study Time:Nasa TLX: frontreartiltfrontreartilt Task: Count number of labels Independent variables –Method: front, rear, tilt –Number of labels (3±1, 9±1) –Scene complexity (2x2, 4x4)

71. Mobile and Physical Interaction71WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Side-of-Device Interaction: SideSight Useful if device is placed on table Distance sensors along device edges –Multipoint interactions IR proximity sensors –Edge: 10x1 pixel “depth” image Left and right “depth” images Butler, Izadi, Hodges: SideSight: Multi-“touch” Interaction Around Small Devices. UIST’08.

72. Improving Touch Screen Accuracy

73. Mobile and Physical Interaction73WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Source: Patrick Baudisch Small Displays  Small Targets

74. Mobile and Physical Interaction74WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Stylus vs. Direct Finger Input Stylus or pen –Grabbing stylus takes to long for short interactions Bare finger input –Unclear contact point, imprecise –Finger occludes target

75. Mobile and Physical Interaction75WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Disadvantages of this “software stylus” 1.No visual feedback until contact, need to estimate offset 2.Makes some display areas unreachable 3.Unexpected offset affects walk-up-and-use scenarios Source: Patrick Baudisch Offset Cursor (Potter et al., 1988) Potter, Weldon, Shneiderman: Improving the accuracy of touch screens: an experimental evaluation of three strategies. CHI 1988.

76. Mobile and Physical Interaction76WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Source: Patrick Baudisch Shifting the whole screen is distracting Disorients users, negatively impacts performance Shifting the Whole Screen

77. Mobile and Physical Interaction77WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Source: Patrick Baudisch Shift Callout (Vogel & Baudisch, 2007) Only shift callout Enough context around target 26mm circular shape  occluded area under finger

78. Mobile and Physical Interaction78WS 2010/2011Jörg Müller, Michael Nischt, T-Labs no offset, click on the target itself Source: Patrick Baudisch Shift Needed Only for Small Targets

79. Mobile and Physical Interaction79WS 2010/2011Jörg Müller, Michael Nischt, T-Labs callout can go anywhere, no edge problems Source: Patrick Baudisch Corners and Edges

80. Mobile and Physical Interaction80WS 2010/2011Jörg Müller, Michael Nischt, T-Labs by default:dwell time (300 ms) extension:larger target  longer dwell time extension:shift learns dwell times vs. Source: Patrick Baudisch When to Show the Callout S T = target size S F = finger occlusion size

81. Mobile and Physical Interaction81WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Source: Patrick Baudisch Experiment: Shift vs. Offset vs. Touch Independent variables –3 techniques: shift, offset, touch –2 finger styles: nail, tip –6 target sizes

82. Mobile and Physical Interaction82WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Results: Error Rate

83. Mobile and Physical Interaction83WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Source: Patrick Baudisch Results: Time

84. Mobile and Physical Interaction84WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Source: Patrick Baudisch Results: Time

85. Mobile and Physical Interaction85WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Source: Patrick Baudisch Shift + Zoom Increased callout diameter to compensate for less context

86. Mobile and Physical Interaction86WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Escape: A Target Selection Technique Using Visually-cued Gestures Problem: Selecting a target on a touch-screen that is surrounded by other selectable objects Solution: The icons in “Escape” indicate directions. A finger tap followed by a movement in this direction enables disambiguation. Yatani, Partridge, Newman: Escape: A Target Selection Technique Using Visually-cued Gestures, CHI 2008

87. Mobile and Physical Interaction87WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Escape: A Target Selection Technique Using Visually-cued Gestures The icons in “Escape” indicate directions. A finger tap followed by a movement in this direction enables disambiguation. Can handle 2.3 icons per square centimeter Find an assignment that separates gestures –Similar to graph coloring  NP-complete –“Escape” uses heuristic algorithm http://www.youtube.com/watch?v=x3NeZswKkKw

88. Mobile and Physical Interaction88WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Experiment: Escape compared to Shift Independent variables –Technique: Escape or Shift –Target size: 6, 12, 18, 24 pixels –Exposure (fraction of target visible): 0.25, 0.5, 0.75, 1 Results –Escape significantly faster than Shift –No significant difference in error rate

89. Mobile and Physical Interaction89WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Experiment: Influence of Color and Beak Occlusion Independent variables –TargetSize: 6, 9, 12 pixels –Exposure: 0.25, 0.5, 0.75, 1 –Direction: 8 directions –Color: gray or colored by dir. –Beak occlusion: yes, no Results –Significant effect for beak occlusion –1-colored icon selection as fast as 8-colored icon selection

90. Mobile and Physical Interaction90WS 2010/2011Jörg Müller, Michael Nischt, T-Labs Escape: Advantages and Disadvantages Advantages –If target big enough, no need to gesture –No need to be 100% accurate –No need to wait for something to appear Disadvantages –Each icon takes up more space than original target –At screen edge limited gestures directions

91. Mobile and Physical Interaction91WS 2010/2011Jörg Müller, Michael Nischt, T-Labs