1 Ken Hinckley Patrick Baudisch Gonzalo Ramos Francois Guimbretiere Microsoft Research Scriboli: High Performance Pen Interfaces

2 Copy + Paste: 1. Select Lasso Mode The Traveling (pen) Salesman Problem

3 2. Circle objects

4 The Traveling (pen) Salesman Problem 3. Invoke Edit Menu

5 The Traveling (pen) Salesman Problem 4. Choose Copy Command

6 The Traveling (pen) Salesman Problem 5. Invoke Edit Menu again

7 The Traveling (pen) Salesman Problem 6. Choose Paste

8 The Traveling (pen) Salesman Problem 7. Acquire the copy

9 The Traveling (pen) Salesman Problem 8. Drag copy to desired position.

10 The Traveling (pen) Salesman Problem The well traveled pen.

11 Waste time moving the pen Focus attention on small targets Same steps every time But what did I forget? The Traveling (pen) Salesman Problem

12 The Traveling (pen) Salesman Problem 9) Forgot to reselect PEN mode. Oops.

13 Speed without keyboard hotkeys Cognitive footprint diminishes with use Recall / declarative  procedural skill Minimal demands on visual attention Repeatable motions for the user Predictable & dependable system behavior Expressiveness Support many command structures & apps Economy of design only a few things to learn Scriboli Design Goals

14 SKETCH – Brown University Adding 1 more gesture breaks everything hard to learn, gestures specific to 3D editing

15 Recognize Pen Gestures? Classic problem: “ink” or “gesture” ? Hard problem: any ‘gesture’ could be ink Inferred decision comes too late real-time interactive feedback difficult Make it a physical skill All we need to solve this is ONE BIT of info physical button on bezel habit-forming, deterministic, low attentional demand nonpreferred hand, no trip to “lasso mode”

16 Ink/Gesture study w/ U. Washington 5 Techniques: Barrel Button Tap+hold NPH button Pen Pressure Pen Eraser

17 Phrase Structure for Scriboli Separator (start) Press “Gesture” Separator Scope/Command Separator Command/Parameters Separator (Done) ScopeCommandParameters Springboard mode Can’t get stuck, unavoidable feedback

18 Separator (start) Press “Gesture” Separator Scope/Command Pigtail Separator Command/Parameters Separator (Done) ScopeCommandParameters Fast, based on muscle “memory” Reliable and easily chunked by users Phrase Structure for Scriboli

19 Phrase Structure for Scriboli Separator (start) Press “Gesture” Separator Scope/Command Pigtail Separator Command/Parameters Crossing boundary Separator (Done) ScopeCommandParameters Merge command selection and direct manipulation FlowMenu [Guimbretiere 00], Control menu [Pook 00]

20 Phrase Structure for Scriboli Separator (start) Press “Gesture” Separator Scope/Command Pigtail Separator Command/Parameters Crossing boundary Separator (Done) ScopeCommandParameters One stroke for scope, command, and parameters Non-modal system (quasi-modal)

21 A delimiter is “something different” lexical structure of stroke e.g. self-crossing gesture stroke DEMO / VIDEO : delimiter techniquesVIDEO Delimiters for scope selection + marking

22 Summary: Grammar for pen input Scriboli proposes fundamental building blocks Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases Written text separate words sentence start sentence end delimits clause “The cat sat on the mat”Punct. space capitals period comma object verb indirect obj. Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip. phase

23 Stitching: Pen Gestures that Span Multiple Displays Microsoft Research Ken Hinckley Gonzalo Ramos Francois Guimbretiere Patrick Baudisch Marc Smith

24 Wireless Network 2004 = Horseless Carriage 1904? Are Wireless Networks really just wired networks without the wires? Or are they something completely different?

25 Wireless Device Soup: Which links does the user want?

26 Users need techniques to intuitively form bridges between devices How do users name the devices to connect? What is type / purpose of the connection? Parameters? (Who copies what, to where?) Name That Device

27 Pen stroke that spans displays Move the pen Cross over bezel Finish stroke on nearby tablet System infers connection Stitching

28 AVI’04 ACM Advanced Visual Interfaces 2004 VIDEO

29 Establishing a Connection Recognizes timing & dynamics of pen trajectory There is nothing special about the pen! Wireless signal strength determines nearby devices Tablet 1 Tablet 2 ΔtΔt

30 How Do Users Share Physical Space? Sociology  Proxemics: How people use space Invisible bubble that surrounds an individual

31 6. How Do Users Share Physical Space? Sociology  Proxemics: How people use space Invisible bubble that surrounds an individual Homework assignment: Sit right next to someone at airport (when it is not necessary to do so) Time how long it is before the other person leaves

32 5. How Do Users Share Physical Space? Sociology  Proxemics: How people use space Invisible bubble that surrounds an individual

33

34 Don’t require contact : touching is taboo “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Proxemic Lessons for Stitching

35 Don’t require contact “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Give users flexibility to be involved, or not Intimate Spaces: Combining screens. For close collaboration with friend or trusted colleague Personal Spaces: Tablets can be separated by up to 30” yet still possible to stitch to give files to colleague, etc. Social Spaces: Once connected, “transporter” can be used to give files to a user beyond arm’s reach Proxemic Lessons for Stitching

36 Multiple Devices: Cooperative Stitching Ongoing work User 1 User 2 User 3 User 4

37 Multiple Devices: Cooperative Stitching Ongoing work User 1 User 2 User 3 User 4

38 Questions?

39 Results – Completion Time Left bar – main block; right bar – repeated block Dotted bars are Pigtail2 design iteration Button Handle Pigtail Pigtail2 Timeout

40 Results – Error Rate Error rate of selecting wrong marking direction Button Handle Pigtail Pigtail2 Timeout

41 Handle can get in the way Adds some visual clutter Must check to be sure landed on handle Pigtail handles more than one scope elegantly Self-referential gesture Handle vs. Pigtail