1 Ken Hinckley Patrick Baudisch Gonzalo Ramos Francois Guimbretiere Microsoft Research Delimiters for Selection-Action Pen Gesture Phrases in Scriboli

2 Copy + Paste: 1. Select Lasso Mode The Traveling (pen) Salesman Problem

3 2. Circle objects

4 The Traveling (pen) Salesman Problem 3. Invoke Edit Menu

5 The Traveling (pen) Salesman Problem 4. Choose Copy Command

6 The Traveling (pen) Salesman Problem 5. Invoke Edit Menu again

7 The Traveling (pen) Salesman Problem 6. Choose Paste

8 The Traveling (pen) Salesman Problem 7. Acquire the copy

9 The Traveling (pen) Salesman Problem 8. Drag copy to desired position.

10 The Traveling (pen) Salesman Problem The well traveled pen.

11 Waste time moving the pen Focus attention on small targets Same steps every time But what did I forget? The Traveling (pen) Salesman Problem

12 The Traveling (pen) Salesman Problem 9) Forgot to reselect PEN mode. Oops.

13 Pen Computing Now?

14 Recognize Pen Gestures? Classic problem: “ink” or “gesture” ? Hard problem: any ‘gesture’ could be ink All we need to solve this is ONE BIT of info Scriboli: Make it a physical skill instead of guessing game physical button on bezel nonpreferred hand, no trip to “lasso mode” Next paper: Li, Hinckley, Guan & Landay works really well

15 Gestures for Nouns & Verbs Lasso selection Marking menus Crossing selection?Abandon lasso selection? How to combine scope + command efficiently?

16 A delimiter is “something different” lexical structure of stroke e.g. self-crossing gesture stroke VIDEO : delimiter techniquesVIDEO Delimiters for scope selection + marking

17 Experiment Design {Pigtail, Handle, Timeout, Button} X Practice: 32 trials Main Block: All 8 Marking Directions (N, NE, …) X 2 Selection Types (single, or row/column) X 5 trials = 80 main block trials Repeated Invocation (RI) Block Lasso one item, select E from menu 24x

18 Results – Completion Time Left bar – main block; right bar – repeated block Dotted bars are Pigtail2 design iteration Button Handle Pigtail Pigtail2 Timeout

19 Results – Learning Effects For the 24 trials of Repeated Invocation Block

20 Results – Error Rate Error rate of selecting wrong marking direction Button Handle Pigtail Pigtail2 Timeout

21 Improving Pigtail Problems 1. Pigtail Size No upper bound

22 Pigtail Design Iteration Problems 1. Pigtail Size No upper bound 2. Pretzels

23 Pigtail Design Iteration Problems 1. Size Limit A<5600 no good 2. Pretzels 1 st intersection? No.

24 Pigtail Design Iteration Problems 1. Size Limit A<5600 no good 2. Pretzels 1 st intersection? No. 2 nd intersection? Yep.

25 Pigtail2 Preliminary Results No errors at all in Repeated Invocation block! Button Handle Pigtail Pigtail2 Timeout

26 Scriboli DEMO DEMO Direct manipulation phase Scope, command, & manip in one stroke Self-revelation techniques Disjoint Selections Lassos, Crossing, Tapping Circles of exclusion

27 Conclusion Performance data on delimiter techniques Data suggest Handle has slight advantage But handle was large at 48x48 pixels pen up/ pen down + spatial mux Can’t be used e.g. in paper interface Pigtail offers design alternative Performs almost the same, lasso tough case? Seems to offer some unique design advantages

28 Future Work ‘grammar’ of building blocks for pen UI What are alternative designs for each part? More complex phrases possible? Written text separates words sentence start sentence end delimits clause “The cat sat on the mat”Punct. space capitals period comma object verb indirect obj. Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip. phase

29 Questions?

30 Handle vs. Pigtail Handle can get in the way Adds some visual clutter Pigtail handles more than one scope elegantly

31 Handwriting has a Deep Structure Learning cursive fast, brief, repetitive, pre-programmed variations on oscillatory movements The scribbling of Scriboli

32 Qualitative Results Favorite: 7 Handle, 4 Pigtail, 4 Timeout Handle two strokes {disruptive, natural} Clutter + box makes part of screen inaccessible Timeout good for beginners… but I hate waiting for it. I felt like I was almost too accurate with it Pigtail Felt just like handwriting… once I learned it, I could open up & fly! Pigtail was trouble. I got a little mad at it. Takes a little bit of practice.

33

34 Marking Menus BlindPrompted Scale-independent in blind mode (333ms) Visually-guided in prompted mode

35 Preserves scale-independence marking starts 333 ms visual menu pops up blind phase check for crossing cross outer boundary dragging Scale-ind. only matters for blind marking Dragging naturally lasts longer

36 GEdit – Kurtenbach & Buxton Delete group – end stroke inside lasso Move – end stroke outside. Objects move on PenUp. Copy – add final “C” to move One of the first works considering pen phrases Every stroke is gesture Only 3 cmds Group vs. single obj. inconsistent Delete one obj.

37 FlowMenu – Francois Guimbretiere Crossing in  out chooses Item… submenu Crossing out  in chooses Move command Dragging continues Integrates command selection with continuous parameter manipulation But no “blind marking,” size of gesture matters

38 Design Philosophy of Scriboli RAPID Fast motions, repeatable, no waiting / dwelling, minimal demands on visual attention UNAMBIGUOUS No guessing games No reco unless I tell the system otherwise (There is almost no reco in Scriboli) EXPRESSIVE Supports many commands, easy to grow design Mechanisms for a wide span of pen apps, not restricted to specific niche application

39 Handle 2 strokes Box makes part of screen inaccessible Timeout 500 ms dwell (stroke-and-hold) Button Hit when want to bring up menu too early / too late … Other Delimiter Techniques

40 Questions?