The End of Pointing and Clicking: Improving Computer Access with Goal Crossing 1 June 30, 2009

People 2

Design Criteria Ease of motor-impaired performance How easy is it motor-wise to acquire the target? Efficiency? No  t No multiple steps Speed for able-bodied? Speed for motor-impaired? Safety (Unwanted targets should not be accidentally triggered) How easy/difficult is it to make a false selection? Tolerance to movement error? Is the design reversible? Is it cancelable? Can the user abort? Scalability Does it handle high-density layouts? How does it scale to hundreds of menu items, links, buttons, and icons? Is it compatible with conventional point-and-click widgets? Can both coexist? Etc. Are the benefits of crossing preserved (e.g., “follow-through,” “aggregation”)? Is it easy to learn? Does it require preexisting knowledge? How hard will it be to implement? Is heavy animation required for the technique to work? Can it lend itself to a nice visual design? 3

1. Basic Crossing Ideas 1.1 Cross While Off-hand Button Held Down 1.2 Cross and Click Anywhere 1.3 Drag and Cross 1.4 Cross and Cross-back 1.5 Delta t 4

1.1 Cross While Off-hand Button Held Down OK 5

1.2 Cross and Click Anywhere OK Click! 6

1.3 Drag and Cross OK Drag&Cross 7

OK 1.4 Cross and Cross-back OK The target gives visual feedback Cross goal to activate Cross back to confirm selection Cross back to confirm selection Cross goal to activate ActivatedSelectedDouble clicked (or) Drag (or) Right clicked … 8

1.4 Cross and Cross-back (cont.) OK Then back over, turn 90-degrees and then cross the two bars Activate an icon by crossing over icon and initial bar Byron McMullen. Jacob Nelson. Tara Tarjoto. Tim Wibowo 9

1.5 Delta t OK Cross and stop for N seconds to confirm the selection 10

1.5 Delta t (cont.) Cross and Stop 11

2. Advanced Crossing Ideas 2.1 Secondary Goal Crossing 2.2 Gesture 2.3 Corners and Edges 2.4 Moving Targets 2.5 Features of Crossing Event 12

OK BUTTON first goal: button second goal (g2): crossing the g1 initiates g2 to appear g2 Length-to-the-g2 variationangle variation G2 design variation: how to make it look like a goal? 2.1 Secondary Goal Crossing 13

OK Crossing the first target activates the arc 2.1 Secondary Goal  Arc-shape Location of the 2 nd goal changes depending on the approaching angle (a) 14

g2 fades out if they are not being selected during a certain amount of time Right-handed, left-handed 2.1 Secondary Goal  L-Shape 15

MULTI-LAYERED MENU: Red bar appears only when the menu has no underlying menu items 2.1 Secondary Goal  I-Shape 16

GO 2.1 Secondary Goal  ‘dontclickit’ idea OK GO OK ANIMATION 17

2.1 Secondary Goal  Cluster T1 T2 T3 T4 T7 T8 T9 T6 Interactive prototype Targets are clustered (3*3 or 2*2) and the secondary goals appear at the fixed location

2.2 Gesture  Zig-zag / Scribble OK Nope, it’s not selected Yes 19

2.2 Gesture  Encircling Targets ANIMATION 20

2.2 Gesture  Pigtail ANIMATION 21

2.2 Gesture  Hover Widget cross cancel Turning 90° activates the most recently crossed goal. When the mouse crosses the end of the tunnel, the crossed goal is confirmed 22

2.2 Gesture  Shivering Around a Target Sparse layout, one small target 23

2.2 Gesture  Rectangle Lasso Philip Phung, Colin Booth, Mark Vitazko, Chris Claiborne Public Downloads File Folder To open 24

2.2 Gesture  Color In Philip Phung, Colin Booth, Mark Vitazko, Chris Claiborne Public Downloads File Folder 25

2.2 Gesture  Letter gesture Chris Sinco, Rishi Talwar, Justin Wilbourne, Ken Wong 26

OK 2.2 Gesture  Alexei Bespalov, Kevin Merritt, James Rodgers 27

2.3 Corners and Edges  Constraining mouse movement with edges 28

OK 2.3 Corners and Edges  Only Listen to Y-Movements Now only Listen to X-Movements Basic Idea: Person can make the computer ignore x or y movements, creating an edge to travel along. The edges (or guidelines, or whatever we call them) could be activated with a keyboard/mouse button? Any other options? Maybe both can be activated to keep the from moving during a click? Could be used for crossing too. 29

Click! Select Skip Select All Corners are easy to hit. Make a context menu out of a square, where corners are options. 2.3 Corners and Edges  Box 30

2.3 Corners and Edges  OK Cancel OpenC.M 31

2.3 Corners and Edges  Diamond Context Menu (C.M.) Open Cancel Crossing one object Previous Open Cancel Crossing more than one object Open all Previous Open Cancel Context Menu (C.M.) Open Cancel (Cancel) 32

2.3 Corners and Edges  Hover Widget + Diamond Crossing one objectCrossing more than one object Context Menu Open current Open multiple X X Open multiple Open current Hitting the corner triggers the queued up targets (aggregation) Select or delete the targets that are in the queue Context Menu C.M Open C.M Open 33

2.3 Corners and Edges  90-degree OK 34

2.3 Corners and Edges  TARGET 35

2.3 Corners and Edges  The mouse can travel through either of the entrances/exits to begin activating the button, but the cursor must travel all the way through to the other side in order to fully trigger the button’s action. OK Once the cursor enters the button, barriers at the top and bottom of the button prevent the passage out of the button. These barriers exist only once the mouse moves in a vertical direction. OK Chris Sinco, Rishi Talwar, Justin Wilbourne, Ken Wong 36

2.3 Corners and Edges  Bounding Buttons Nathan – Peter - Kaitlyn Certain elements have un-mousable “screen edges” affixed to them to avoid pointer overshoot. Ballistic movement can now hit and stick to targets. 37

2.4 Moving target  Coming Toward Closest target is triggered, and then the target is coming toward the cursor 38

2.4 Moving target  Stepping Backward Closest target is triggered, and then the target is stepping backward from the cursor 39

2.4 Moving target  Rearranging Targets 40

2.5 Features of Crossing Event  Velocity Not selectedSelected Needs to cross above certain speed in order to select the target Too slow 41

2.5 Features of Crossing Event  Acceleration Not selectedSelected Stop and then accelerate the crossing speed (A>0) a= 0 42

2.5 Features of Crossing Event  Angle SelectedNot selected Orthogonal crossing: selected 43

3. Other Crossing Ideas 3.1 Kinematic Template 3.2 Accessible Bubble Cursor 3.3 Switch 3.4 The Clock 3.5 Guideline 3.6 Trapezoid 3.7 Extended Crossing Line 3.8 Plough / Force Field Cursor 3.9 Edge.Point 3.10 Area Crosser 3.11 Stabbing 3.12 The Casual Fan 3.13 Triple Cross 3.14 Cross and Cross Back with Hover Widget 3.15 Corner Pocket 44

3.1 Kinematic Template Passive template: the cursor goes through the button with the help of the (passive) template OK 45

OK Cancel Edit Options Menu Open OK Cancel Edit Options Menu Open Cancel Options Edit Tab through targets! OK Menu OK Open Edit Cancel Menu 3.2 Accessible Bubble Cursor 46

3.3 Switch Gravity effect + visual feedback OK activated Switching on 47

3.4 The Clock In this concept, a lever appears inside a circular overlay on the target when the cursor gets close to a goal. To trigger the selection of a target, the user “pushes” the lever (like a clock hand) in a circle until a full rotation is made. The cursor “sticks” to lever, facilitating rotation like the hand on a clock. Once the rotation movement has begun, the user can escape the “stickiness” by sliding the curser off the outer edge of the lever towards the outside of the circle. Clarke Freeman, Alex Jansen, Kristofer Martin, Josh Rakita 48

3.5 Guideline If the cursor follows the guideline for a certain amount of time and then cross, the target is selected Selected Not Selected Stop for a while -> guideline appears Selected 49

3.6 Trapezoid OK 50

3.7 Extended Crossing Line  Extended Crossing Goal Voronoi tessellation 51

3.7 Extended Crossing Line  Extended Crossing Goal Voronoi tessellationClustering targets around the cursor 52

3.8 Plough / Force Field Cursor Clarke Freeman, Alex Jansen, Kristofer Martin, Josh Rakita Cross with acceleration Inspired by specifically considering the difficult circumstance of a densely arranged set of targets, the force field cursor effectively pushes targets around a circle, or force field, which remains centered on the cursor during tracking movement. Multiple targets in contact with the force field are distributed around the entire circumference. Selection is made by accelerating towards one of the acquired potential targets, achieving an “escape velocity” that allows the cursor leave the center of the field and cross over the desired target. 53

3.9 Edge.Point Nathan – Peter - Kaitlyn As you move your mouse around the screen all Icons in the range shown by the area cursor will visually “tunnel” to your cursor. To toggle between freezing and releasing the area cursor press space bar. Freezing the cursor will also freeze all tunnels that are displayed. While tunnels are frozen they will “catch” your mouse cursor and bound it within the tunnel. This allows for easy and quick navigation to all available targets. As a reminder the tunnels and their bounding properties can be escaped from at any time by hitting space bar. 54

3.10 Area Crosser Nathan – Peter - Kaitlyn An area cursor that traces a path from the mouse cursor to each target. Instead of traveling down the entire path, a goal is presented at the very beginning to cross and thus activate the button. Pros: Pure crossing goal, reduces cursor travel distance. Cons: Hard to implement actions other than "cross." CLICK! 55

3.11 Stabbing Philip Phung, Colin Booth, Mark Vitazko, Chris Claiborne A user moves to a group of items; as the cursor approaches them, the items move around the cursor in a fan motion allowing for the user to select one by moving their mouse backwards over the target. The user is then presented with a menu of possible selection, which they can thereby make their selection by passing back over the icon. Select Delete 56

3.12 The Casual Fan Chris Sinco, Rishi Talwar, Justin Wilbourne, Ken Wong 45° arch to commit to selection 90° arch for escape selection Cancel Activate the target 57

3.13 Triple Cross Alexei Bespalov, Kevin Verritt, James Rodgers The technique works by having the user cross over a goal three times, before he crosses over any other goals three times within additional crosses of the original goal. This technique is similar to the cross and cross-back technique, however it adds both safety and flexibility to the technique by adding a third cross of the goal. 58

Public Downloads File Folder 3.14 Cross & Cross-back with Hover Widget Alexei Bespalov, Kevin Verritt, James Rodgers The technique works by having the user cross over a goal three times, before he crosses over any other goals three times within additional crosses of the original goal. This technique is similar to the cross and cross-back technique, however it adds both safety and flexibility to the technique by adding a third cross of the goal. Copy Properties Rename Open Delete Cut Send 59

3.15 Corner Pocket Greg Bickford, Don Bushell, Gavin Elster, Andy Boer 60

4. Designing Actions and Context Menu 4.1 Accessing context menu with Hover Widget (list) 4.2 Accessing context menu with Hover Widget (circular) 4.3 Accessing context menu with Hover Widget + Diamond (list) 4.4 Accessing context menu with Hover Widget + Diamond (circular) 4.5 Accessing context menu with Secondary goal crossing 61

4.1 Accessing Context Menu with Hover Widget (list) Right turn: Access context menu (= right click) Move Copy Delete Open with… Cut Select Group Properties Move Copy Delete Open with… Cut Select Group Properties List context menu can handle sub-context menu well OpenOffice.org Microsoft Powerpoint Keynote ‘Select’ corresponds to the single left click in Windows Confirm Move Copy Delete Open with… Cut Select Group Properties Relative turn Absolute 90-degree turn 62

4.2 Accessing Context Menu with Hover Widget (circular) Cancel Move Copy Select Open with… Cancel Copy Select Open with… Cancel Cut Move Group Properties Move Copy Select Open with… Cancel Move Copy Select Open with… Cancel Right turn: Access context menu 63

4.3 Accessing Context Menu with Hover Widget + Diamond (list) Up Down Open Cut Copy Move Delete Select Rename Properties Open Move Copy Select all Open with… Cut Delete Group Properties Back ‘Select all’ becomes one of the context menu items Turn right Left turn, and at the right corner of the diamond 64

4.4 Accessing Context Menu with Hover Widget + Diamond (circular) C.M Previous C.M Delete Copy Select Open with… Cancel Group C.M Open Previous Cancel Open 65

4.5 Accessing Context Menu with Secondary Goal Crossing Move Copy Select Open with… Cancel Open Move Copy Select Open with… Cancel Open Sequence cross -> secondary goal appears -> turn 90 degree Cancel 1)timeout 2)menu item 3)cross back 4)loophole merge left-click, right-click, single-click and double-click Move Copy Select Open with… Open 66

The End 67