1. 1 The subArctic Input System and Extensions for Handling Inputs with Ambiguity

2. 2 subArctic n A Java-based GUI toolkit that I (along with Ian Smith) built and distributed in 1996-97 n Goal: highly extensible allowing support for lots of cool new interaction techniques – Emphasis on making new and strange widgets / components / interactors easy to create – “High ceiling”

3. 3 Parties involved with a toolkit n Toolkit designer (me) n Interactor designer n Interface programmer n User

4. 4 Parties involved with a toolkit n Toolkit designer (me) n Interactor designer n Interface programmer n User Most toolkits target support here

5. 5 Parties involved with a toolkit n Toolkit designer (me) n Interactor designer n Interface programmer n User By moving work up (into reusable library)

6. 6 Parties involved with a toolkit n Toolkit designer (me) n Interactor designer n Interface programmer n User But typically don’t help much here (assume a fixed library)

7. 7 subArctic n Toolkit designer (me) n Interactor designer n Interface programmer n User SA tries to move work for many kinds of interactors into toolkit infrastructure

8. 8 subArctic n Toolkit designer (me) n Interactor designer n Interface programmer n User SA tries to move work for many kinds of interactors into toolkit infrastructure Input system is a big part of that

9. 9 Schema for pretty much all GUIs init(); for (;;) { evt = wait_for_next_event(); dispatch(evt); if ( damage_exists() ) redraw(); }

10. 10 Schema of a GUI init(); for (;;) { evt = wait_for_next_event(); dispatch(evt); if ( damage_exists() ) redraw(); } Event Record – recording of the relevant facts about some occurrence of interest (i.e., user has manipulated an input device)

11. 11 Schema of a GUI init(); for (;;) { evt = wait_for_next_event(); dispatch(evt); if ( damage_exists() ) redraw(); } Send (“dispatch”) the event to the object(s) that want it and/or know how to respond to it (e.g., widget/component/interactor)

12. 12 Event dispatch n All the work happens here n Typically delegated to interactors – E.g., buttons know how to respond to press and release like buttons should – Each object keeps track of its own state n... but which interactor gets it  Toolkit “event dispatch” process

13. 13 Event dispatch policies n Two primary ways to decide which interactor gets an event n What are they?

14. 14 Event dispatch policies n Two primary ways to decide which interactor gets an event – Positional dispatch F Based on where mouse is pointing F Examples… – Focus-based dispatch F Designated object always gets input F Examples…

15. 15 Pop quiz n Should input for dragging be dispatched via positional or focus?

16. 16 Pop quiz n Should input for dragging be dispatched via positional or focus? Answer: No! (both)

17. 17 subArctic input policies n subArctic encapsulates these “ways of dispatching inputs” in “dispatch policy objects” – Manages bookkeeping (e.g., picking) – Extensible set F Turns out there are other useful policies (e.g., for modal dialogs)

18. 18 When interactors get events… n … they typically respond to them with the equivalent of a simple finite state machine Press Move Release

19. 19 subArctic has lib of common FSMs n Move a lot of input handling work typically done by interactor programmer up into the toolkit n One (highly parameterized) FSM for all – Brad’s “interactor” model (awful terminology :-) n Many customized FSM (extensible set) – subArctic input model

20. 20 FSMs moved to toolkit object n “Dispatch agent” n Translates low level input into higher level terms

21. 21 Dispatch agent example: move_drag n Translated to calls in input protocol: – drag_start(); – drag_feedback(); – drag_end(); n With useful parameters (e.g. new pos) Press Move Release

22. 22 Dispatch agent example: move_drag n Translated to calls in input protocol: – drag_start(); – drag_feedback(); – drag_end(); n With useful parameters (e.g. new pos) Press Move Release Defined by Java interface

23. 23 Set of dispatch agents is extensible n E.g., can subclass for specialized kinds of drag such as “drag_within_box” or “snap_drag” – Can create custom for one interface – Once created can reuse

24. 24 How it all goes together Focus Policy Positional Policy Etc… Events Press Click Rollover Etc... Text Move drag Grow drag Etc...

25. 25 How does interactor indicate it wants / can handle some type of input? n “… implements input_protocol” – Where “ input_protocol ” is interface with calls like drag_start(), etc. n For positional that’s it! n For focus-based must also ask for focus

26. 26 Example: Hypertext for all n User (Ken Anderson) wanted to add hyperlinks to all objects – Hold down the control key and click – His external hyperlink database would take over and map interactor id to hyperlink target – But… how do you change every interactor to do this?

27. 27 Example: Hypertext for all n In Swing, Motif, etc. this is essentially impossible n In SA, just insert a new subclass of the “click” dispatch agent that checks for the control key down – About 15 lines of code – Works for interactors written later!

28. 28 Questions about the SA input system?

29. 29 Providing Toolkit Level Support for Handling Ambiguity in Recognition- Based Input See: http://doi.acm.org/10.1145/354401.354407 and http://doi.acm.org/10.1145/332040.332459http://doi.acm.org/10.1145/354401.354407http://doi.acm.org/10.1145/332040.332459 Jennifer Mankoff, Gregory Abowd Georgia Institute of Technology Scott Hudson Carnegie Mellon University

30. 30 Motivation n Recognition-based input offers the promise of naturalistic input modalities, BUT…

31. 31 Motivation n Recognition-based input offers the promise of naturalistic input modalities, BUT… n Recognizers are imperfect – affects users – breaks current system models è New interfaces & mechanisms

32. 32 Example Interaction n SILK n Hand-sketched interactors

33. 33 Example Interaction n SILK n Interface developer can replace interactors with best recognition result A button

34. 34 Example Interaction n Correction Dialog (mediator) test

35. 35 Example Interaction n Correction interactor (mediator) test text teat ted N-Best List Keyboard Character Correction Revert to Strokes

36. 36 Example Interaction n Problems with dialog – Not reusable or customizable – Hard to grow your own è Basically we don’t have toolkit support for recognition based UI

37. 37 Motivation (cont.) n At much the same stage we were at for GUIs in 1983 – No common model for input – No re-use F Infrastructure F “widget library”

38. 38 An alternative: Burlap VIDEO

39. 39 Goals of This Work n Robust, reusable infrastructure n Reusable library n Integrate with convent. toolkit – Don’t throw out the baby with the bathwater

40. 40 Talk Roadmap n Requirements for handling uncertain input n Extending toolkits to handle it n Interaction techniques for ambiguity n Implementation

41. 41 Invoking Application Actions n Action often done by callbacks – Direct procedure call to application n Hierarchical events are alternate approach – Delivered to app as well as toolkit

42. 42 Hierarchical Events n Low-level events contribute to production of higher-level events [Green TOG ‘86; Myers & Kosbie CHI ‘96] User Input circle stroke downdragup Corresponding Events

43. 43 Implicit Assumption of Certainty n Implicit in all this is the assumption that the events really happened as reported n Problems arise when this isn’t true – E.g., brittle dialogs

44. 44 Needed to Handle Uncertainty: n Allow for (and explicitly model) multiple alternatives – alternative higher level events – in recognition context: interpretations n Detect conflicting interpretations n Mediation of conflicts

45. 45 Needed to Handle Uncertainty: n Lexical feedback about uncertain events – split “feedback” from “action” n Library of mediators

46. 46 How do we do this...

47. 47 Extended Event Model n Uncertainty results in multiple interpretations  interpretation graph Uncertain Input circlebox stroke downdragup circle stroke downdragup Certain Input

48. 48 Toolkit Extensions n Toolkit’s job is still to deliver events to objects – Now delivered to recognizers, interactors, and application objects Button Checkbox Menu Recog

49. 49 Toolkit Extensions n Toolkit’s job is still to deliver events to objects – Objects initially only produce (reversible) feedback, no actions Button Checkbox Menu Recog

50. 50 Another Change: Interface Appearance Uncertain Event Hierarchy circlebox stroke downdragup n Events dispatched to all who might use it

51. 51 Details: Arranging for Mediation n Identify any conflicts n Look for a mediators – Pluggable list of them in toolkit n Mediator chosen by meta- mediator n Mediator can: “Pass”, “Pause”, “Accept”

52. 52 Doing Mediation n Example: User selects interpretation circle box circle

53. 53 Doing Mediation (cont.) n Mediator prunes interpretation graph to tree – App informed of accept & reject circlebox stroke downdragup circle stroke downdragup

54. 54 Mediation Strategies n Many mediation strategies – e.g., Automatic vs. user involvement n Toolkit is fully “pluggable” (!) – Library of mediators provided, but – Can extend/build new ones as needed n Research goal:Finding new ones

55. 55 Providing a Library of Mediators

56. 56 Providing a Library of mediators n Survey of existing techniques [Abowd & Mankoff GVU Tech Report 99] n Three major classes n Explored additional techniques

57. 57 Providing a Library of mediators n Survey of existing techniques n Three major classes – Repetition

58. 58 Providing a Library of mediators n Survey of existing techniques n Three major classes – Repetition – Choice: Ripe for toolkit support F Presentation form F Instantiation time F Contextual information F Interaction F Feedback

59. 59 Providing a Library of mediators n Survey of existing techniques n Three major classes – Repetition – Choice – Automatic F Thresholds F Confusion matrix F Plug in machine learning?

60. 60 Providing a Library of mediators n Survey of existing techniques n Three major classes n Explored additional techniques

61. 61 Example: Target Ambiguity n Problem: There may be multiple targets of a user action n Example: clicking – Kabbash (95) – Worden (97)

62. 62 Example: Target Ambiguity n Problem: There may be multiple targets of a user action n Example: Clicking n Solution: Give the user a choice of all of the targets

63. 63 Example: Target Ambiguity n Problem: There may be multiple targets of a user action n Example: Clicking n Solution: Give the user a choice of all of the targets n Other applications: – Any interface involving mouse press/release – Requires separation of concerns – Works with all interactors

64. 64 Implementation n Added support for mediation; ambiguity to subArctic toolkit – Reusable – Fully “pluggable” – Full existing library still works as is (!) n Small library of mediators n Also working on non-GUI toolkit (http://dx.doi.org/10.1145/572003)http://dx.doi.org/10.1145/572003

65. 65 Experience n Major example: Burlap – Smaller version of SILK [Landay] – For sketching UI designs and turning them into functioning interfaces

66. 66 Conclusions n Reusable infrastructure to support ambiguous input – Reduces difficulty of creating UIs – Easier to explore new design space n Done by modifying a toolkit, not a separate mechanism – Integrated with conventional input – Other support from toolkit still useful

67. 67