1. 1 Ken Hinckley kenh@microsoft.com Microsoft Research Nov. 20 th, 2003 Distributed Sensing Techniques for Ubiquitous Mobile Devices

2. 2 Who am I anyway? Research Scientist, PhD in Comp Sci Background in Human-Computer Interaction Interests Input Devices Interaction Techniques Sensors Mobile Devices Ubiquitous Computing

3. 3 Some Recent Projects “Meat & Potatoes” Mouse Scrolling wheel – accelerated scrolling

4. 4 Accelerated Scrolling Wheel How do users roll the wheel? (1) trying to get somewhere fast, or (2) reading Roll faster  move further No loss of control The user does not have to learn anything new Exp. study: Does no harm, users can scroll faster

5. 5 Some Recent Projects “Meat & Potatoes” Mouse Scrolling wheel – accelerated scrolling Office Keyboard

6. 6 Navigation controls on left Scrolling - wide wheel Web - Forward / Back] Windows - AppToggle Cut, Copy, Paste keys on left RH: Compound selection / IP LH: articulation of command MS Office Keyboard Hugh McLoone, Ken Hinckley, Ken Fry, Steve Bathiche

7. 7 Uncanny Coincidence #1 October 2003 Office Keyboard Winner of HFES 2003 User Centered Consumer Product Design Award!  Office Keyboard discontinued by Microsoft

8. 8 Some Recent Projects “Meat & Potatoes” Mouse Scrolling wheel – accelerated scrolling Office Keyboard – bimanual navigation controls More far-ranging research Sensing Techniques for Mobile Interaction

9. 9 Sensors: Tilt – 2 axis accelerometer Touch Proximity inexpensive, low power What new features might inexpensive, ubiquitous sensors enable ? VIDEO ACM UIST 2000 Best Paper Award! Sensing Techniques for Mobile Interaction

10. 10 Uncanny Coincidence #2 11/20/2003 – Ken advocates coming ubiquity of tilt & other sensors sensors 11/18/2003 – Conveniently to make Ken’s point, Toshiba announces Portege M200 Tablet PC Tilt fwd/back to scroll Tilt L/R to flip pages Orient to change Portrait / Landscape

11. 11 Some Recent Projects “Meat & Potatoes” Mouse Scrolling wheel – accelerated scrolling Office Keyboard – bimanual navigation controls More far-ranging research Sensing Techniques for Mobile Interaction Distributed Sensing Techniques for Ubiquitous Mobile Devices

12. 12 Still an immature technology – Slow, flaky Do people really want to read web pages on their cell phones? Sensor nets - applications for mobile devices? Odd phrase: not what it is, but what it lacks In 100 years Wireless Network = ??? Wireless Networks

13. 13 Horseless Carriage?

14. 14 2003: Isolated Digital Islands

15. 15 Wireless Fills in Gaps between Devices: But Which Ones?

16. 16 Users need techniques to intuitively form bridges between devices How do users name the devices to connect? What is type / purpose of the connection? Parameters? (Who copies what, to where?) Name That Device

17. 17 User activity that is sensed by two or more devices in a distributed system Shared with nearby participating devices via wireless networking System synchronizes time & puts the pieces back together Our Strategy: Synchronous Gestures

18. 18 Shaking Hands

19. 19 Bowing

20. 20 Making a Toast

21. 21 VIDEOVIDEO– Perceptive TabletPC’s

22. 22 Any “bumps” while devices not being held are ignored Example “Bump” Data

23. 23 Bumping does more than name the devices: Edges to join: Left, Right, Top, Bottom 2-axis tilt sensor gives this information Direction (hierarchy in connection) One tablet is base, one tablet is connecting Without this, unclear which tablet is “boss” Touch sensor: which device is held? Bumping has Implicit Parameters

24. 24 How Synchronous Gestures Enable Wireless Matchmaking

25. 25 Participating Devices Connect to Server Sync Gesture Server Server may be hosted in environment, or on a tablet.

26. 26 Send Sensor Data to Server Sync Gesture Server

27. 27 Time Sync Sync Gesture Server

28. 28 Bump Recognized! Sync Gesture Server

29. 29 Server Gives Partners Each Other’s IP Address Sync Gesture Server Alternatively can have “private” connection via Sync Gesture Server

30. 30 Connect to Peer & Link Displays Sync Gesture Server

31. 31 Can’t synchronize with everyone: n(n-1)/2 limited resources / power false positives through sheer chance Ideally, should sync only w/ co-located devices Possible scoping mechanisms peer-to-peer: anyone in radio range sync’s proximity server: discover nearby devices through 802.11 signal strengths (John Krumm) Open Problem: Who to Synchronize With?

32. 32 Wireless signal strength provides proximity Bumping offers explicit step of intentionality that user has control over selection done in real world via manipulation Bumping provides additional information edges to join, type of connection, direction, etc. Proximity provides list of devices to synchronize with Why Not Just Use Proximity Sensing?

33. 33 Sharing information – mutual or one way Multiple tablets Face-to-face collaboration VIDEO #2 Other Ways to Use Bumping

34. 34 Bumping works well, intuitive, fun But requires hardware that is not (yet ) in common use A few operations possible, but seems hard to add further variations or semantics Started thinking about generalizing the issues we encountered The result was… What Next?

35. 35 Ken Hinckley Gonzalo Ramos (Intern) Francois Guimbretiere (Visiting Researcher) Patrick Baudisch Marc Smith Work currently submitted to CHI 2004 Stitching: Pen Gestures that Span Multiple Displays

36. 36 Move the pen Cross over bezel Finish stroke on nearby tablet System infers connection Yields versatile framework for connecting pen-operated devices Stitching

37. 37 VIDEO

38. 38 1.How is a connection established? 2.What type of connection is required? 3.What information is shared? 4.What is the spatial relationship btw devices? 5.How do stitching gestures coexist with other uses for the pen? 6.How do users share physical space? Implemented a photo sharing app Central Design Questions

39. 39 1. Establishing a Connection There is nothing special about the pen! Recognizes timing & dynamics of pen trajectory Pen exits from edge of one screen & enters at edge of another within a timeout Tablet 1 Tablet 2 ΔtΔt

40. 40 2. Type of Connection User selects a multi-device command e.g., Pie menu at end of stitching gesture

41. 41 3. What Information to Share? Many commands may need operands Lasso to select, continue to other display to stitch.

42. 42 4. What is Spatial Relationship: Automatic Screen Calibration Fits a line to absolute coords of user’s pen stroke

43. 43 Stitching can occur in Tracking or Dragging Tracking is separate layer on top of GUI Dragging is only option for PDA’s Must undo drags that turn out to be stitches 5. Coexistence of Stitching with Traditional Pen Use

44. 44 6. How Do Users Share Physical Space? Sociology  Proxemics: How people use space Invisible bubble that surrounds an individual

45. 45 6. How Do Users Share Physical Space? Sociology  Proxemics: How people use space Invisible bubble that surrounds an individual Homework assignment: Sit right next to someone at airport (when it is not necessary to do so) Time how long it is before the other person leaves

46. 46 6. How Do Users Share Physical Space? Sociology  Proxemics: How people use space Invisible bubble that surrounds an individual

47. 47

48. 48 Don’t require contact “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Proxemic Lessons for Stitching

49. 49 Don’t require contact “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Establish and Relax Users want to Establish a connection, but then Relax increasing social tension by quickly exiting Observed lean back, short stitches; transporter popular One argument against remote menus Flexibility: Allow users to be involved or not Proxemic Lessons for Stitching

50. 50 Intimate 0-18” Personal 18- 30” Social: 4’ – 7’ Public 12’ or more Distances vary by culture Touching is taboo in non-contact cultures E. T. Hall’s “Distances in Man” Within arm’s reach } Beyond arm’s reach }

51. 51

52. 52 Stitching supports Intimate, Personal & Social Distances Gives users flexibility to be involved, or not Intimate Spaces: Combining screens. For close collaboration with friend or trusted colleague Personal Spaces: Tablets can be separated by up to 30” yet still possible to stitch to give files to colleague, etc. Social Spaces: Once connected, “transporter” can be used to give files to a user beyond arm’s reach Features of Stitching Correspond to Hall’s Distances

53. 53 Proxemics: Orientation of Spaces 6 conversations 3 1 0 Communication patterns change wrt orientation Stitching should support: Shoulder-to-shoulder Face-to-face At 90 angles Relative frequency:

54. 54 Tablet 1Tablet 2 PDAPDA PDAPDA Porches Tablet 1Tablet 2 PDA The Candy Dish Alternative Proxemics Using PDA’s PDA Stitching not yet impl. but should be feasible

55. 55 Combining heterogeneous devices Tablet + PDA? PDA + keyboard? Jeff Pierce @ Ga. Tech: “Opportunistic Annexing” + Cell phone? + Watch? + Furby ?!? Multiple Devices & Orientations Future Directions User 1 User 2 User 3 User 4

56. 56 Questions?