12/5/2018 The subArctic Input System and Extensions for Handling Inputs with Ambiguity

12/5/2018 subArctic A Java-based GUI toolkit that I (along with Ian Smith) built and distributed in 1996-97 Goal: highly extensible allowing support for lots of cool new interaction techniques Emphasis on making new and strange widgets / components / interactors easy to create “High ceiling”

Parties involved with a toolkit 12/5/2018 Parties involved with a toolkit Toolkit designer (me) Interactor designer Interface programmer User

Parties involved with a toolkit 12/5/2018 Parties involved with a toolkit Toolkit designer (me) Interactor designer Interface programmer User Most toolkits target support here

Parties involved with a toolkit 12/5/2018 Parties involved with a toolkit By moving work up (into reusable library) Toolkit designer (me) Interactor designer Interface programmer User

Parties involved with a toolkit 12/5/2018 Parties involved with a toolkit Toolkit designer (me) Interactor designer Interface programmer User But typically don’t help much here (assume a fixed library)

subArctic Toolkit designer (me) Interactor designer 12/5/2018 subArctic Toolkit designer (me) Interactor designer Interface programmer User SA tries to move work for many kinds of interactors into toolkit infrastructure

Input system is a big part of that 12/5/2018 subArctic Toolkit designer (me) Interactor designer Interface programmer User Input system is a big part of that SA tries to move work for many kinds of interactors into toolkit infrastructure

Schema for pretty much all GUIs 12/5/2018 Schema for pretty much all GUIs init(); for (;;) { evt = wait_for_next_event(); dispatch(evt); if ( damage_exists() ) redraw(); }

Schema of a GUI init(); for (;;) { } 12/5/2018 Event Record – recording of the relevant facts about some occurrence of interest (i.e., user has manipulated an input device) Schema of a GUI init(); for (;;) { evt = wait_for_next_event(); dispatch(evt); if ( damage_exists() ) redraw(); }

(e.g., widget/component/interactor) 12/5/2018 Schema of a GUI Send (“dispatch”) the event to the object(s) that want it and/or know how to respond to it (e.g., widget/component/interactor) init(); for (;;) { evt = wait_for_next_event(); dispatch(evt); if ( damage_exists() ) redraw(); }

Event dispatch All the work happens here 12/5/2018 Event dispatch All the work happens here Typically delegated to interactors E.g., buttons know how to respond to press and release like buttons should Each object keeps track of its own state ... but which interactor gets it Toolkit “event dispatch” process

Event dispatch policies 12/5/2018 Event dispatch policies Two primary ways to decide which interactor gets an event What are they?

Event dispatch policies 12/5/2018 Event dispatch policies Two primary ways to decide which interactor gets an event Positional dispatch Based on where mouse is pointing Examples… Focus-based dispatch Designated object always gets input

12/5/2018 Pop quiz Should input for dragging be dispatched via positional or focus?

Answer: No! (both) Pop quiz 12/5/2018 Pop quiz Should input for dragging be dispatched via positional or focus? Answer: No! (both)

subArctic input policies 12/5/2018 subArctic input policies subArctic encapsulates these “ways of dispatching inputs” in “dispatch policy objects” Manages bookkeeping (e.g., picking) Extensible set Turns out there are other useful policies (e.g., for modal dialogs)

When interactors get events… 12/5/2018 When interactors get events… … they typically respond to them with the equivalent of a simple finite state machine Press Move Release

subArctic has lib of common FSMs 12/5/2018 subArctic has lib of common FSMs Move a lot of input handling work typically done by interactor programmer up into the toolkit One (highly parameterized) FSM for all Brad’s “interactor” model (awful terminology :-) Many customized FSM (extensible set) subArctic input model

FSMs moved to toolkit object 12/5/2018 FSMs moved to toolkit object “Dispatch agent” Translates low level input into higher level terms

Dispatch agent example: move_drag 12/5/2018 Dispatch agent example: move_drag Press Move Release Translated to calls in input protocol: drag_start(); drag_feedback(); drag_end(); With useful parameters (e.g. new pos)

Dispatch agent example: move_drag 12/5/2018 Dispatch agent example: move_drag Press Move Release Translated to calls in input protocol: drag_start(); drag_feedback(); drag_end(); With useful parameters (e.g. new pos) Defined by Java interface

Set of dispatch agents is extensible 12/5/2018 Set of dispatch agents is extensible E.g., can subclass for specialized kinds of drag such as “drag_within_box” or “snap_drag” Can create custom for one interface Once created can reuse

How it all goes together 12/5/2018 How it all goes together Focus Policy Positional Policy Etc… Events Press Click Rollover Etc... Text Move drag Grow drag

How does interactor indicate it wants / can handle some type of input? 12/5/2018 How does interactor indicate it wants / can handle some type of input? “… implements input_protocol” Where “input_protocol” is interface with calls like drag_start(), etc. For positional that’s it! For focus-based must also ask for focus

Example: Hypertext for all 12/5/2018 Example: Hypertext for all User (Ken Anderson) wanted to add hyperlinks to all objects Hold down the control key and click His external hyperlink database would take over and map interactor id to hyperlink target But… how do you change every interactor to do this?

Example: Hypertext for all 12/5/2018 Example: Hypertext for all In Swing, Motif, etc. this is essentially impossible In SA, just insert a new subclass of the “click” dispatch agent that checks for the control key down About 15 lines of code Works for interactors written later!

Questions about the SA input system? 12/5/2018 Questions about the SA input system?

12/5/2018 Providing Toolkit Level Support for Handling Ambiguity in Recognition-Based Input See: http://doi.acm.org/10.1145/354401.354407 and http://doi.acm.org/10.1145/332040.332459 with Jen Mankoff (her PhD thesis) & Gregory Abowd

12/5/2018 Motivation Recognition-based input offers the promise of naturalistic input modalities, BUT…

12/5/2018 Motivation Recognition-based input offers the promise of naturalistic input modalities, BUT… Recognizers are imperfect affects users breaks current system models New interfaces & mechanisms

Example Interaction SILK (James Landay’s dissertation work @ CMU) 12/5/2018 Example Interaction SILK (James Landay’s dissertation work @ CMU) Hand-sketched interactors

Example Interaction SILK 12/5/2018 Example Interaction SILK Interface developer can replace interactors with best recognition result A button

12/5/2018 Example Interaction Correction Dialog (mediator) test

Example Interaction Problems with these dialogs 12/5/2018 Example Interaction Problems with these dialogs Not reusable or customizable Hard to grow your own Basically we don’t have toolkit support for recognition based UI

Motivation (cont.) At much the same stage we were at for GUIs in 1983 12/5/2018 Motivation (cont.) At much the same stage we were at for GUIs in 1983 No common model for input No re-use Infrastructure “widget library”

An alternative: Burlap 12/5/2018 An alternative: Burlap VIDEO http://www.youtube.com/watch?v=TrWvUcGOq6c

Goals of This Work Robust, reusable infrastructure Reusable library 12/5/2018 Goals of This Work Robust, reusable infrastructure Reusable library Integrate with convent. toolkit Don’t throw out the baby with the bathwater

Roadmap for rest of this section 12/5/2018 Roadmap for rest of this section Requirements for handling uncertain input Extending toolkits to handle it Interaction techniques for ambiguity Implementation

Invoking Application Actions 12/5/2018 Invoking Application Actions Action often done by callbacks Direct procedure call to application Hierarchical events are alternate approach Delivered to app as well as toolkit

12/5/2018 Hierarchical Events Low-level events contribute to production of higher-level events [Green TOG ‘86; Myers & Kosbie CHI ‘96] circle stroke down drag up • • • Corresponding Events User Input

Implicit Assumption of Certainty 12/5/2018 Implicit Assumption of Certainty Implicit in all this is the assumption that the events really happened as reported Problems arise when this isn’t true E.g., brittle dialogs

Needed to Handle Uncertainty: 12/5/2018 Needed to Handle Uncertainty: Allow for (and explicitly model) multiple alternatives alternative higher level events in recognition context: interpretations Detect conflicting interpretations Mediation of conflicts

Needed to Handle Uncertainty: 12/5/2018 Needed to Handle Uncertainty: Lexical feedback about uncertain events split “feedback” from “action” Library of mediators

12/5/2018 How do we do this...

Extended Event Model Uncertainty results in multiple interpretations 12/5/2018 Extended Event Model Uncertainty results in multiple interpretations  interpretation graph circle stroke down drag up • • • Certain Input Uncertain Input circle box stroke down drag up • • •

Toolkit Extensions Toolkit’s job is still to deliver events to objects 12/5/2018 Toolkit Extensions Toolkit’s job is still to deliver events to objects Now delivered to recognizers, interactors, and application objects Button Checkbox Menu Recog

Toolkit Extensions Toolkit’s job is still to deliver events to objects 12/5/2018 Toolkit Extensions Toolkit’s job is still to deliver events to objects Objects initially only produce (reversible) feedback, no actions Button Checkbox Menu Recog

Uncertain Event Hierarchy 12/5/2018 Another Change: Events dispatched to all who might use it Uncertain Event Hierarchy circle box stroke down drag up • • • Interface Appearance

Details: Arranging for Mediation 12/5/2018 Details: Arranging for Mediation Identify any conflicts Look for a mediator Pluggable list of them in toolkit Mediator chosen by meta-mediator Mediator can: “Pass”, “Pause”, “Accept”

Doing Mediation Example: User selects interpretation circle circle box 12/5/2018 Doing Mediation Example: User selects interpretation circle circle box

Doing Mediation (cont.) 12/5/2018 Doing Mediation (cont.) Mediator prunes interpretation graph to tree App informed of accept & reject circle box stroke down drag up • • •

Mediation Strategies Many mediation strategies 12/5/2018 Mediation Strategies Many mediation strategies e.g., Automatic vs. user involvement Toolkit is fully “pluggable” (!) Library of mediators provided, but Can extend/build new ones as needed Research goal:Finding new ones

Providing a Library of Mediators 12/5/2018 Providing a Library of Mediators

Providing a Library of mediators 12/5/2018 Providing a Library of mediators Survey of existing techniques [Abowd & Mankoff GVU Tech Report 99] Three major classes Explored additional techniques

Providing a Library of mediators 12/5/2018 Providing a Library of mediators Survey of existing techniques Three major classes Repetition http://www.youtube.com/watch?v=WiDEUAontbM

Providing a Library of mediators 12/5/2018 Providing a Library of mediators Survey of existing techniques Three major classes Repetition Choice: Ripe for toolkit support Presentation form Instantiation time Contextual information Interaction Feedback

Providing a Library of mediators 12/5/2018 Providing a Library of mediators Survey of existing techniques Three major classes Repetition Choice Automatic Thresholds Confusion matrix Plug in machine learning?

Providing a Library of mediators 12/5/2018 Providing a Library of mediators Survey of existing techniques Three major classes Explored additional techniques

Example: Target Ambiguity 12/5/2018 Example: Target Ambiguity Problem: There may be multiple targets of a user action Example: clicking Kabbash (95) Worden (97)

Example: Target Ambiguity 12/5/2018 Example: Target Ambiguity Problem: There may be multiple targets of a user action Example: Clicking Solution: Give the user a choice of all of the targets http://www.youtube.com/watch?v=om4F0W0i9dE

12/5/2018 Implementation Added support for mediation; ambiguity to subArctic toolkit Reusable Fully “pluggable” Full existing library still works as is (!) Small library of mediators Also worked on non-GUI toolkit [Dey, Mankoff ToCHI 05] (http://doi.acm.org/10.1145/1057237.1057241)

Conclusions Reusable infrastructure to support ambiguous input 12/5/2018 Conclusions Reusable infrastructure to support ambiguous input Reduces difficulty of creating UIs Easier to explore new design space Done by modifying a toolkit, not a separate mechanism Integrated with conventional input Other support from toolkit still useful

12/5/2018