1 Ken Hinckley Gonzalo Ramos (Intern – U. Toronto) Francois Guimbretiere (Visiting Researcher -UMD) Patrick Baudisch Marc Smith Microsoft Research May 26 th, 2004 Stitching: Pen Gestures that Span Multiple Displays

2 Still an immature technology – Flaky, hard for users to configure, slow, … Lots of research has focused on small-screen presentation of web pages But what are the completely new & unique affordances of wireless connectivity? Odd phrase: not what it is, but what it lacks Wireless Networks

3 Wireless Network 2004 = Horseless Carriage 1904? Are Wireless Networks really just wired networks without the wires? Or are they something completely different?

4 Wireless Fills in Gaps between Devices: But Which Ones?

5 Name that Device Users need techniques to intuitively form bridges between devices How do users name the devices to connect? What is type / purpose of the connection? Parameters? (Who copies what, to where?) Stitching offers unique approach that combines all of these steps using the pen Spontaneous Device Sharing Problem:

6 Move the pen Cross over bezel Finish stroke on nearby tablet System infers connection Standard Pen: No new hardware required How to Stitch path taken by the pen transferred pictures path taken by the pen transferred pictures path taken by the pen transferred pictures

7 VIDEO

8 1.Connection: How is a connection established? 2.Command: What type of connection is required? 3.Operands: What information is shared? 4.Geometry: What is the spatial relationship btw devices? 5.Coexistence: How do stitching gestures coexist with other uses for the pen? 6.Proxemics: How do users share physical space? Implemented a photo sharing app Central Design Questions

9 1. Establishing a Connection There is nothing special about the pen! Recognizes timing & dynamics of pen trajectory Pen exits from edge of one screen & enters at edge of another within a timeout Tablet 1 Tablet 2 ΔtΔt

10 2. Type of Connection User selects a multi-device command e.g., Pie menu at end of stitching gesture

11 3. What Information to Share? Many commands may need operands Lasso to select, continue to other display to stitch. Sender/receiver (direction to send) naturally specified

12 4. What is Spatial Relationship: Automatic Screen Calibration Fits a line to absolute coords of user’s pen stroke

13 Stitching can occur in Tracking or Dragging Tracking is separate layer on top of GUI Dragging is only option for PDA’s Must undo drags that turn out to be stitches Works best full-screen; otherwise menus in way 5. Coexistence of Stitching with Traditional Pen Use

14 6. How Do Users Share Physical Space? Sociology  Proxemics: How people use space Invisible bubble that surrounds an individual

15 6. How Do Users Share Physical Space? Sociology  Proxemics: How people use space Invisible bubble that surrounds an individual Homework assignment: Sit right next to someone at airport (when it is not necessary to do so) Time how long it is before the other person leaves

16 6. How Do Users Share Physical Space? Sociology  Proxemics: How people use space Invisible bubble that surrounds an individual

17

18

19 Don’t require contact Touching is taboo in many cultures “Do they have to be right next to one another?” Participants separated devices by 10-40cm People clearly view their device as an extension of their physical body But users also see clear need for adjacent devices: “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Proxemic Lessons for Stitching (1)

20 Give users flexibility to be involved (or not) E. T. Hall’s Distances in Man Within Arm’s Length: Intimate, Personal Proxemic Lessons for Stitching (2)

21  Intimate 

22  Personal 

23 Give users flexibility to be involved (or not) E. T. Hall’s Distances in Man Within Arm’s Length: Intimate, Personal Beyond Arm’s Length: Social, Public Proxemic Lessons for Stitching (2)

24  Social 

25 Public

26 Give users flexibility to be involved (or not) E. T. Hall’s Distances in Man Within Arm’s Length: Intimate, Personal Beyond Arm’s Length: Social, Public Stitching supports: Intimate Spaces: Combining screens. For close collaboration with friend or trusted colleague Personal Spaces: Tablets can be separated by up to 30” yet still possible to stitch to give files to colleague, etc. Social Spaces: Once connected, “transporter” can be used to give files to a user beyond arm’s reach Proxemic Lessons for Stitching (2)

27 Proxemic Lessons for Stitching (3) 6 conversations Support varying relative body orientation Stitching supports: Shoulder-to-shoulder Face-to-face At 90 angles VIDEO Relative frequency:

28 Usability Testing 12 users in pairs, many had not used pens Very positive comments & reactions overall All users got making pen gesture across displays Simple way to share information Ability to easily combine devices

29 Usability Testing 12 users in pairs, many had not used pens Very positive comments & reactions overall All users got making pen gesture across displays Simple way to share information Ability to easily combine devices Problems Stitch starting at edge of screen User’s don’t want to be forced into contact Security / Privacy a common concern Various small issues, e.g. prefix vs. postfix menus

30 Combining heterogeneous devices Tablet + PDA? “The Candy Dish” Future Directions Tablet 1Tablet 2 PDA

31 Combining heterogeneous devices Tablet + PDA? “The Candy Dish” Future Directions Tablet 1Tablet 2 PDA

32 Combining heterogeneous devices Tablet + PDA? “The Candy Dish” Future Directions Tablet 1Tablet 2 PDA

33 Combining heterogeneous devices Tablet + PDA? “The Candy Dish” Future Directions Tablet 1Tablet 2 PDA

34 Combining heterogeneous devices Tablet + PDA? “The Candy Dish” Future Directions Tablet 1Tablet 2 PDA

35 Combining heterogeneous devices Tablet + PDA? “The Candy Dish” Cooperative Stitching Future Directions User 1 User 2 User 3 User 4

36 Combining heterogeneous devices Tablet + PDA? “The Candy Dish” Cooperative Stitching other users complete the gesture Future Directions User 1 User 2 User 3 User 4

37 Questions?

38 Can’t synchronize with everyone: n(n-1)/2 limited resources / power false positives through sheer chance Ideally, should sync only w/ co-located devices Possible scoping mechanisms proximity server: discover nearby devices through signal strengths (John Krumm) peer-to-peer: anyone in radio range sync’s Who to Synchronize With?

39 Wireless signal strength provides proximity Stitching offers explicit step of intentionality that user has control over selection done in real world via manipulation Stitching provides additional information edges to join, type of connection, direction, etc. Proximity provides list of devices to synchronize with Why Not Just Use Proximity Sensing?

40 Why not Infrared Port (IrDA)? Only one geometry: Relative orientation fixed Proximity does not imply desire to connect devices “Do you want to share files with Joe Blow in seat 8A?” Stitching, Bumping give user explicit control Lacks additional parameters (how to combine, operands, geometery of displays, sender vs. receiver, etc.) Mainly for sharing files, not about combining devices Still difficult to use and set up

41

42 Tandler, P., T. Prante, C. Müller-Tomfelde, N.A. Streitz and R. Steinmetz. Connectables: dynamic coupling of displays for the flexible creation of shared workspaces. in UIST Rekimoto, J. Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments. in Proc. ACM UIST'97 Symp. on User Interface Software & Technology Banff, Alberta, Canada. Rekimoto, J., Y. Ayatsuka and M. Kohno. SyncTap: An Interaction Technique for Mobile Networking. in Mobile HCI : Springer. Rekimoto, J. and M. Saitoh. Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments. in CHI' Related Work

43 Tandler - ConnecTables Displays must be within ~1 inch Single orientation Single operation

44 Rekimoto – Pick & Drop Copyright © Sony Computer Science Laboratories, Inc. Pen with unique ID “stores” information Devices don’t have to be placed together Application: sharing files (copy/paste)

45 Rekimoto - SyncTap Images Copyright © Sony Computer Science Laboratories, Inc. Brings up window for other device Which one sends? Spatial relationship?

46 Rekimoto- Augmented Surfaces Copyright © Sony Computer Science Laboratories, Inc.

47 Aug. Surfaces – “Hyperdragging” Copyright © Sony Computer Science Laboratories, Inc.

48

49 How Do Synchronous Gestures Enable Wireless Matchmaking?

50 Participating Devices Connect to Server Stitching Server Server may be hosted in environment, or on a tablet.

51 Send Sensor Data to Server Stitching Server

52 Time Sync Stitching Server

53 Bump Recognized! Stitching Server

54 Server Gives Partners Each Other’s IP Address Stitching Server Alternatively can have “private” connection via Stitching Server

55 Connect to Peer & Link Displays Stitching Server

56 Users want to Establish a connection, but then Relax increasing social tension by quickly exiting Observed lean back, short stitches; transporter popular One argument against remote menus Establish and Relax