1. 12/5/2018
The subArctic Input System and Extensions for Handling Inputs with Ambiguity

2. 12/5/2018
subArctic
A Java-based GUI toolkit that I (along with Ian Smith) built and distributed in
Goal: highly extensible allowing support for lots of cool new interaction techniques
Emphasis on making new and strange widgets / components / interactors easy to create
“High ceiling”

3. Parties involved with a toolkit
12/5/2018
Parties involved with a toolkit
Toolkit designer (me)
Interactor designer
Interface programmer
User

4. Parties involved with a toolkit
12/5/2018
Parties involved with a toolkit
Toolkit designer (me)
Interactor designer
Interface programmer
User
Most toolkits target support here

5. Parties involved with a toolkit
12/5/2018
Parties involved with a toolkit
By moving work up (into reusable library)
Toolkit designer (me)
Interactor designer
Interface programmer
User

6. Parties involved with a toolkit
12/5/2018
Parties involved with a toolkit
Toolkit designer (me)
Interactor designer
Interface programmer
User
But typically don’t help much here
(assume a fixed library)

7. subArctic Toolkit designer (me) Interactor designer
12/5/2018
subArctic
Toolkit designer (me)
Interactor designer
Interface programmer
User
SA tries to move work for many kinds of interactors into toolkit infrastructure

8. Input system is a big part of that
12/5/2018
subArctic
Toolkit designer (me)
Interactor designer
Interface programmer
User
Input system is a big part of that
SA tries to move work for many kinds of interactors into toolkit infrastructure

9. Schema for pretty much all GUIs
12/5/2018
Schema for pretty much all GUIs
init();
for (;;) {
evt = wait_for_next_event();
dispatch(evt);
if ( damage_exists() ) redraw(); }

10. Schema of a GUI init(); for (;;) { }
12/5/2018
Event Record – recording of the relevant facts about some occurrence of interest (i.e., user has manipulated an input device)
Schema of a GUI
init();
for (;;) {
evt = wait_for_next_event();
dispatch(evt);
if ( damage_exists() ) redraw(); }

11. (e.g., widget/component/interactor)
12/5/2018
Schema of a GUI
Send (“dispatch”) the event to the object(s) that want it and/or know how to respond to it
(e.g., widget/component/interactor)
init();
for (;;) {
evt = wait_for_next_event();
dispatch(evt);
if ( damage_exists() ) redraw(); }

12. Event dispatch All the work happens here
12/5/2018
Event dispatch
All the work happens here
Typically delegated to interactors
E.g., buttons know how to respond to press and release like buttons should
Each object keeps track of its own state
... but which interactor gets it
Toolkit “event dispatch” process

13. Event dispatch policies
12/5/2018
Event dispatch policies
Two primary ways to decide which interactor gets an event
What are they?

14. Event dispatch policies
12/5/2018
Event dispatch policies
Two primary ways to decide which interactor gets an event
Positional dispatch
Based on where mouse is pointing
Examples…
Focus-based dispatch
Designated object always gets input

15. 12/5/2018
Pop quiz
Should input for dragging be dispatched via positional or focus?

16. Answer: No! (both) Pop quiz
12/5/2018
Pop quiz
Should input for dragging be dispatched via positional or focus?
Answer: No! (both)

17. subArctic input policies
12/5/2018
subArctic input policies
subArctic encapsulates these “ways of dispatching inputs” in “dispatch policy objects”
Manages bookkeeping (e.g., picking)
Extensible set
Turns out there are other useful policies (e.g., for modal dialogs)

18. When interactors get events…
12/5/2018
When interactors get events…
… they typically respond to them with the equivalent of a simple finite state machine
Press
Move
Release

19. subArctic has lib of common FSMs
12/5/2018
subArctic has lib of common FSMs
Move a lot of input handling work typically done by interactor programmer up into the toolkit
One (highly parameterized) FSM for all
Brad’s “interactor” model (awful terminology :-)
Many customized FSM (extensible set)
subArctic input model

20. FSMs moved to toolkit object
12/5/2018
FSMs moved to toolkit object
“Dispatch agent”
Translates low level input into higher level terms

21. Dispatch agent example: move_drag
12/5/2018
Dispatch agent example: move_drag
Press
Move
Release
Translated to calls in input protocol:
drag_start();
drag_feedback();
drag_end();
With useful parameters (e.g. new pos)

22. Dispatch agent example: move_drag
12/5/2018
Dispatch agent example: move_drag
Press
Move
Release
Translated to calls in input protocol:
drag_start();
drag_feedback();
drag_end();
With useful parameters (e.g. new pos)
Defined by Java interface

23. Set of dispatch agents is extensible
12/5/2018
Set of dispatch agents is extensible
E.g., can subclass for specialized kinds of drag such as “drag_within_box” or “snap_drag”
Can create custom for one interface
Once created can reuse

24. How it all goes together
12/5/2018
How it all goes together
Focus Policy
Positional Policy
Etc…
Events
Press
Click
Rollover
Etc...
Text
Move drag
Grow drag

25. How does interactor indicate it wants / can handle some type of input?
12/5/2018
How does interactor indicate it wants / can handle some type of input?
“… implements input_protocol”
Where “input_protocol” is interface with calls like drag_start(), etc.
For positional that’s it!
For focus-based must also ask for focus

26. Example: Hypertext for all
12/5/2018
Example: Hypertext for all
User (Ken Anderson) wanted to add hyperlinks to all objects
Hold down the control key and click
His external hyperlink database would take over and map interactor id to hyperlink target
But… how do you change every interactor to do this?

27. Example: Hypertext for all
12/5/2018
Example: Hypertext for all
In Swing, Motif, etc. this is essentially impossible
In SA, just insert a new subclass of the “click” dispatch agent that checks for the control key down
About 15 lines of code
Works for interactors written later!

28. Questions about the SA input system?
12/5/2018
Questions about the SA input system?

29. 12/5/2018
Providing Toolkit Level Support for Handling Ambiguity in Recognition-Based Input
See: and
with Jen Mankoff (her PhD thesis) & Gregory Abowd

30. 12/5/2018
Motivation
Recognition-based input offers the promise of naturalistic input modalities, BUT…

31. 12/5/2018
Motivation
Recognition-based input offers the promise of naturalistic input modalities, BUT…
Recognizers are imperfect
affects users
breaks current system models
New interfaces & mechanisms

32. Example Interaction SILK (James Landay’s dissertation work @ CMU)
12/5/2018
Example Interaction
SILK (James Landay’s dissertation CMU)
Hand-sketched interactors

33. Example Interaction SILK
12/5/2018
Example Interaction
SILK
Interface developer can replace interactors with best recognition result
A button

34. 12/5/2018
Example Interaction
Correction Dialog (mediator)
test

35. Example Interaction Problems with these dialogs
12/5/2018
Example Interaction
Problems with these dialogs
Not reusable or customizable
Hard to grow your own
Basically we don’t have toolkit support for recognition based UI

36. Motivation (cont.) At much the same stage we were at for GUIs in 1983
12/5/2018
Motivation (cont.)
At much the same stage we were at for GUIs in 1983
No common model for input
No re-use
Infrastructure
“widget library”

37. An alternative: Burlap
12/5/2018
An alternative: Burlap
VIDEO

38. Goals of This Work Robust, reusable infrastructure Reusable library
12/5/2018
Goals of This Work
Robust, reusable infrastructure
Reusable library
Integrate with convent. toolkit
Don’t throw out the baby with the bathwater

39. Roadmap for rest of this section
12/5/2018
Roadmap for rest of this section
Requirements for handling uncertain input
Extending toolkits to handle it
Interaction techniques for ambiguity
Implementation

40. Invoking Application Actions
12/5/2018
Invoking Application Actions
Action often done by callbacks
Direct procedure call to application
Hierarchical events are alternate approach
Delivered to app as well as toolkit

41. 12/5/2018
Hierarchical Events
Low-level events contribute to production of higher-level events
[Green TOG ‘86; Myers & Kosbie CHI ‘96]
circle
stroke
down
drag up
• • •
Corresponding Events
User Input

42. Implicit Assumption of Certainty
12/5/2018
Implicit Assumption of Certainty
Implicit in all this is the assumption that the events really happened as reported
Problems arise when this isn’t true
E.g., brittle dialogs

43. Needed to Handle Uncertainty:
12/5/2018
Needed to Handle Uncertainty:
Allow for (and explicitly model) multiple alternatives
alternative higher level events
in recognition context: interpretations
Detect conflicting interpretations
Mediation of conflicts

44. Needed to Handle Uncertainty:
12/5/2018
Needed to Handle Uncertainty:
Lexical feedback about uncertain events
split “feedback” from “action”
Library of mediators

45. 12/5/2018
How do we do this...

46. Extended Event Model Uncertainty results in multiple interpretations
12/5/2018
Extended Event Model
Uncertainty results in multiple interpretations
 interpretation graph
circle
stroke
down
drag up
• • •
Certain Input
Uncertain Input
circle
box
stroke
down
drag up
• • •

47. Toolkit Extensions Toolkit’s job is still to deliver events to objects
12/5/2018
Toolkit Extensions
Toolkit’s job is still to deliver events to objects
Now delivered to recognizers, interactors, and application objects
Button
Checkbox
Menu
Recog

48. Toolkit Extensions Toolkit’s job is still to deliver events to objects
12/5/2018
Toolkit Extensions
Toolkit’s job is still to deliver events to objects
Objects initially only produce (reversible) feedback, no actions
Button
Checkbox
Menu
Recog

49. Uncertain Event Hierarchy
12/5/2018
Another Change:
Events dispatched to all who might use it
Uncertain Event Hierarchy
circle
box
stroke
down
drag up
• • •
Interface Appearance

50. Details: Arranging for Mediation
12/5/2018
Details: Arranging for Mediation
Identify any conflicts
Look for a mediator
Pluggable list of them in toolkit
Mediator chosen by meta-mediator
Mediator can:
“Pass”, “Pause”, “Accept”

51. Doing Mediation Example: User selects interpretation circle circle box
12/5/2018
Doing Mediation
Example: User selects interpretation
circle
circle
box

52. Doing Mediation (cont.)
12/5/2018
Doing Mediation (cont.)
Mediator prunes interpretation graph to tree
App informed of accept & reject
circle
box
stroke
down
drag up
• • •

53. Mediation Strategies Many mediation strategies
12/5/2018
Mediation Strategies
Many mediation strategies
e.g., Automatic vs. user involvement
Toolkit is fully “pluggable” (!)
Library of mediators provided, but
Can extend/build new ones as needed
Research goal:Finding new ones

54. Providing a Library of Mediators
12/5/2018
Providing a Library of Mediators

55. Providing a Library of mediators
12/5/2018
Providing a Library of mediators
Survey of existing techniques [Abowd & Mankoff GVU Tech Report 99]
Three major classes
Explored additional techniques

56. Providing a Library of mediators
12/5/2018
Providing a Library of mediators
Survey of existing techniques
Three major classes
Repetition

57. Providing a Library of mediators
12/5/2018
Providing a Library of mediators
Survey of existing techniques
Three major classes
Repetition
Choice: Ripe for toolkit support
Presentation form
Instantiation time
Contextual information
Interaction
Feedback

58. Providing a Library of mediators
12/5/2018
Providing a Library of mediators
Survey of existing techniques
Three major classes
Repetition
Choice
Automatic
Thresholds
Confusion matrix
Plug in machine learning?

59. Providing a Library of mediators
12/5/2018
Providing a Library of mediators
Survey of existing techniques
Three major classes
Explored additional techniques

60. Example: Target Ambiguity
12/5/2018
Example: Target Ambiguity
Problem: There may be multiple targets of a user action
Example: clicking
Kabbash (95)
Worden (97)

61. Example: Target Ambiguity
12/5/2018
Example: Target Ambiguity
Problem: There may be multiple targets of a user action
Example: Clicking
Solution: Give the user a choice of all of the targets

62. 12/5/2018
Implementation
Added support for mediation; ambiguity to subArctic toolkit
Reusable
Fully “pluggable”
Full existing library still works as is (!)
Small library of mediators
Also worked on non-GUI toolkit [Dey, Mankoff ToCHI 05] (

63. Conclusions Reusable infrastructure to support ambiguous input
12/5/2018
Conclusions
Reusable infrastructure to support ambiguous input
Reduces difficulty of creating UIs
Easier to explore new design space
Done by modifying a toolkit, not a separate mechanism
Integrated with conventional input
Other support from toolkit still useful

64. 12/5/2018