1. Interaction Techniques for Ambiguity Resolution in Recognition-based Interfaces Jennifer Mankoff CoC & GVU Center Georgia Tech

2. Acknowledgements  Gregory Abowd & Scott Hudson  FCE Group & GVU  NSF

3. Outline  Motivation  Definitions & Illustration  Broad Solution: OOPS  Specific Solutions  Conclusion & Future Work

4. Ambiguity  am·big·u·ous 1 a : doubtful or uncertain especially from obscurity or indistinctness b : INEXPLICABLE 2 : capable of being understood in two or more possible senses or ways  An anathema to computers  Normal for humans

5. Where does ambiguity arise?  Web search  user doesn’t know correct answer  multiple correct answers  Implicit input  user not involved with application  Multiple users  System states may not agree  Recognition

6. Focus: Recognition  Recognition is becoming ubiquitous  Recognition is difficult to use  A range of interface problems result  OOPS toolkit helps solve them

7. Research Methodology  Motivated by real world, non-CS problems  Evaluators  Study individual problems/solutions  Design space of possible solutions  Builders  Facilitate solutions to sets of problems (Architectural / toolkit solutions)  Design space exploration

8. Outline  Motivation  Definitions & Illustration  Broad Solution: OOPS  Specific Solutions  Conclusions & Future Work

9. Definitions  Mediation  dialogue between user and computer  used for resolving ambiguity  Recognizer  interprets user input  creates ambiguity  Error  mistake from user’s perspective  represented with ambiguity

10. SILK (Landay, 1996)

11. Burlap

12. Outline  Motivation  Definitions & Illustration  Broad Solution: OOPS  Specific Solutions  Conclusions & Future Work

13. OOPS Toolkit (CHI’00)  Toolkit-level support for handling ambiguity in recognition  Library of mediators  Architectural support  Based on subArctic

14. Library of mediators  Design space based on survey  Generic and re-usable  Three major classes  Repetition  Choice  Automatic

15. Library of mediators  Design space based on survey  Generic and re-usable  Three major classes  Repetition  Choice  Automatic

16. Library of mediators  Design space based on survey  Generic and re-usable  Three major classes  Repetition  Choice  Automatic if (result is “W.”) reject it else do nothing

17. Architectural Support  INDEPENDENT of any specific toolkit  Separation of mediators, recognizers, and application  Communication by a common internal model (ambiguous hierarchical events)  Maintains ambiguity indefinitely

18. Three key pieces  Ambiguous hierarchical events  Changes to event dispatch  Mediation subsystem

19. downdragup s Ambiguous Hierarchical Events stroke downdragup sc

20. med1 med2 med3 med4 medn rec1 rec2 rec3 rec4 recn Event Dispatch 1.A sensed event arrives event Input handler 2. It is dispatched to all recognizers 3. It is mediated

21. Mediation Subsystem  Ambiguity is identified automatically  Presence of multiple interactor leaf nodes  Hierarchy is passed to mediators  Recognizers, recipients informed of accept/reject decisions  Accept/reject modifies hierarchy  Application selects mediators from library

22. Outline  Motivation  Definitions & Illustration  Broad Solution: OOPS  Specific Solutions  Conclusions & Future Work

23. Problem Areas  Errors & Ambiguity  rejection errors  target ambiguity  Mediation  adding alternatives  occlusion

24. Rejection Errors  Problem: The user’s input is completely missed

25. Rejection Errors  Problem: The user’s input is completely missed  Solution: Allow the user to tell the system

26. Rejection Errors  Problem: The user’s input is completely missed  Solution: Allow the user to tell the system  Other applications:  Substitution errors

27. Rejection Errors  Problem: The user’s input is completely missed  Solution: Allow the user to tell the system  Other applications:  Substitution errors  Any spatial recognition  Requires extended recognizer API

28. Target Ambiguity  Problem: There may be multiple targets of a user action  Example: clicking

29. Target Ambiguity  Problem: There may be multiple targets of a user action  Example: Clicking  Solution: Give the user a choice of all of the targets

30. Target Ambiguity  Problem: There may be multiple targets of a user action  Example: Clicking  Solution: Give the user a choice of all of the targets  Other applications:  Any interface involving mouse press/release  Requires separation of concerns  Works with all interactors

31. Occlusion  Problem: A mediator may obscure important information

32. Occlusion  Problem: A mediator may obscure important information  Solution: Move that information into a more visible location

33. Occlusion  Problem: A mediator may obscure important information  Solution: Move that information into a more visible location  Other applications:  Any crowded interface that uses an n-best list  Requires extensible mediators  Requires separation of concerns

34. Adding alternatives  Problem: The correct choice isn’t always present

35. Adding alternatives  Problem: The correct choice isn’t always present  Example: word- prediction

36. Adding alternatives  Problem: The correct choice isn’t always present  Example: word- prediction  Solution: Allow the user to add choices

37. Adding alternatives  Problem: The correct choice isn’t always present  Example: word- prediction  Solution: Allow the user to add choices  Other applications:  Closely related choices (e.g. URL prediction)  Requires extensible mediators  Benefits from recognizer API

38. Outline  Motivation  Definitions & Illustration  Broad Solution: OOPS  Specific Solutions  Conclusions & Future Work

39. Conclusions  Resolution of ambiguity in recognition through mediation  General toolkit architecture (CHI 00)  flexible, re-usable support for mediation  separates recognition, mediation, and applications  allows exploration of design space

40. Next Steps  Implicit input  Sensed information about environment  Ambiguous output  Ambient displays  Brain-computer interface  Ambiguous, limited, error-prone input

41. 500,000 people worldwide Locked-In Syndrome  Disease, stroke, accident survivors  Completely paraylzed  Unable to speak  Cognitively intact

42. Brain-computer interface  Problem: locked-in syndrome  No alternative modalities available  Need for efficient error handling  Challenge: interpret brain signal in as rich a form as possible

43. A new hope  Brain signals can be intercepted  Implanted electrodes (Schwartz, Chapin et al.)  External sensors (Junker, Wolpaw, Middendorf, Birbaumer et al., Spencer et al.)  Signals can be produced and controlled by imagined movements  Signals can be interpreted by a computer

44. A Neurotrophic Electrode  Cone electrode invented in 1987  Animal studies showed it to be stable  FDA permission given for human implantation in 1996

45. Project Goals  High level:  Recreate movement  Restore communication  Turn disability into ability  Low level:  Intelligent Interpretation

46. Recreating movement Haptic feedback Muscle stimulation  Eventually re-connect nerves

47. Restore Communication  A basic need  From virtual keyboards to word- prediction

48. Turning disability into ability: S upporting Creativity  Converting a the brain signal to music  Example Example  Two possible experiences  Trying to play a piano with ones feet  Having an artistic voice no one else can reproduce

49. Intelligent Interpretation  Signals are difficult to control  Daily variability in signal  Patient Endurance  Adaption necessary

50. Intelligent Interpretation  Raw signal-> mouse  Logical control  Neural gestures

51. Further Information http://www.cc.gatech.edu/fce/errata/ jmankoff@cc.gatech.edu

52. Alternative Forms of Input  Cirrin (UIST 98)  Locked-In Syndrome (Brain-UI; Current)  Cerebral Palsy (Cursor Activity Recognition; Current)

53. How general is “general”?  Two Toolkits  Two complex applications  Library of existing mediators  Explorations of 4 problem areas (UIST 00)

54. Further generalization  Testing

55. Further generalization  Testing  Implicit input (CT-OOPS; Current)

56. Further generalization  Testing  Implicit input  Arbitrary input devices

57. Further generalization  Testing  Implicit input  Arbitrary input devices  Ambiguity

58. Related Themes  Input  Output  Ambient Displays (Ten Inch Pixels; 1999)

59. Is subArctic doing the work here?  No, our minimal requirements are common in today’s toolkits:  An event-based toolkit  An input-handling module that delivers events to the appropriate places  A library of interactors/widgets  Access to source code (OOPS is not just a library!)

60. Recognizer  Definition:  something that interprets user input  generally has a domain (of input) and a range (of output)  Examples:  DragonDictate (speech to text)  GDT (strokes to gestures)  Problem areas:  Support for correction of errors

61. Error  Definition:  a mistaken interpretation (from the user’s perspective)  Examples:  substitution  rejection  insertion  Problem areas:  rejection

62. Mediation  Definition:  a dialogue between the user and application used to determine the correct interpretation  Problem areas  Occlusion  Wrong choices  Examples:

63. Ambiguity  Definition  A case where there is more than one potentially correct interpretation of the user’s input  Problem areas  target ambiguity  Examples  target ambiguity  segmentation ambiguity  recognition ambiguity

64. Future Work  Testing  Implicit input  Arbitrary input devices  Ambiguity

65. Conclusions  The problem areas are not intractable  Toolkit-level support allows us to explore them  OOPS allows us to build general, re- usable solutions

66. Other thesis results  Survey of mediation techniques found in existing interfaces to recognition systems  Two implementations of our architecture

67. Comparing SILK and Burlap

69. Architectural support architecture interactors interface application Input handler

70. inters OOPS Architecture architecture interactors interface application recognizers Input handler meds application interface stroke downdragup s c stroke downdragup s

71. Big Picture  1/5 People have disabilities  Everyone has temporary disabilities  Computers can help to overcome disabilities… … or make them worse

72. Modified Event Dispatch 1. A sensed event arrives 2a. It is dispatched to all unambiguous components 2b. It is dispatched to all recognizers 3. It is mediated 4. Any accepted leaf nodes are dispatched to all unambiguous components

73. Comparison with GUI event interactors interface application Input handler recognizers meds application interface Input handler inter recognizersmediators