1. Programming Support for Natural Interaction
Jennifer Mankoff
CoC & GVU Center
Georgia Tech

2. Acknowledgements Gregory Abowd & Scott Hudson FCE Group & GVU
NSF & IBM

3. Natural Interaction
Mouse + Keyboard work great for killer apps of the GUI world
Problems Arise…
Off the desktop (Ubiquitous computing)
People with disabilities
My interest: Providing general support for interaction design in this space of problems

4. Outline of Talk Output Input Augmenting physical objects and spaces
Recognition in the face of errors
Motivation and Design Space
Contribution: Toolkit-level support
Validation: 2 Toolkits, Problem areas
Low bandwidth, ambiguous input

5. Output: Augmenting physical objects and spaces
PALplates
User study
Exploration
Domisilica
Dual augmentation
General infrastructure

6. Contents:
an Instant Jello
Pudding:French
Vanilla
a 16oz Ferrara
Capellini-Angel
Hair
a Hamburger
Helper: Beef
Noodle
...

7. Outline of Talk Output Input Augmenting physical objects and spaces
General support for exploring new domains
Input

8. Input Recognition Low Bandwidth, Ambiguous Input
Motivation and Design Space
Contribution: Toolkit-level support
Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input

9. Recognition Recognition is becoming ubiquitous
Recognition is difficult to use
A range of interface problems result
OOPS toolkit helps solve them

10. Research Method Evaluators Builders
Study individual problems/solutions
Design space of possible solutions
Builders
Facilitate solutions to sets of problems (Architectural / toolkit solutions)
Design space exploration by a larger community

11. Input Recognition Low Bandwidth, Ambiguous Input
Motivation and Design Space
Contribution: Toolkit-level support
Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input

12. Definitions Recognizer Error Mediation interprets user input
creates ambiguity
Error
mistake from user’s perspective
represented with ambiguity
Mediation
dialogue between user and computer
used for resolving ambiguity

13. SILK (Landay, 1996)

14. Burlap

15. Design Space of Mediation Techniques
Multimodal
Suhm (98)
McGee et al (98)
Handwriting
Huerst, Yang & Waibel (98)
Speech
Ainsworth & Pratt (92)
Baber & Hone (93)
IUIs
Horvitz (99)
Many Others…

16. Design Space of Mediation Techniques
Two major classes
Repetition
Choice

17. Design Space of Mediation Techniques
Two major classes
Repetition
Choice

18. Input Motivation: Alternatives to Keyboard/Mouse Recognition
Motivation and Design Space
Contribution: Toolkit-level support
Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input

19. OOPS Toolkit (CHI’00)
Toolkit-level support for handling ambiguity in recognition
Library of mediators
Architectural support
Based on subArctic

20. Architectural Support
INDEPENDENT of any specific toolkit
Separation of mediators, recognizers, and application
Communication by a common internal model (ambiguous hierarchical events)
Maintains ambiguity indefinitely

21. Related Work GUI GUI+Recognition Context Multimodal
Garnet/interactors (Myers ‘90)
X toolkits (Rosenberg et al. ’90)
GUI+Recognition
SATIN (Hong et al ’00)
Extended Amulet (Landay & Myers ’93)
PPSM (Hudson&Newell ’92)
Artkit (Henry et al. ’90)
Context
Context Toolkit (Dey et al ’00)
PARCPad, etc (Schilit et al. ’94)
Multimodal
OAA (Cohen et al ’94/Oviatt ’99)
PAC-Amodeus (Nigay& Coutaz ’95)

22. Three key pieces Ambiguous hierarchical events Mediation subsystem
Changes to event dispatch

23. Ambiguous Hierarchical Events
Myers & Kosbie ‘96
stroke
down
drag up
• • • s c
down
drag up
• • •
stroke s

24. Mediation Subsystem Ambiguity is identified automatically
Presence of multiple interactor nodes
Hierarchy is passed to mediators
Recognizers, recipients informed of accept/reject decisions
Accept/reject modifies hierarchy
Application selects mediators from library

25. Event Dispatch Problem: all recipients must support ambiguity
A sensed event arrives
It is dispatched to all recognizers
It is mediated
rec1
rec2
Input
handler
event
rec3
rec4
recn
med2
med1
med4
med3
medn

26. Modified Event Dispatch
1. A sensed event arrives
2a. It is dispatched to ambiguity unaware components
2b. It is dispatched to all recognizers
It is mediated
Any accepted leaf nodes are dispatched to ambiguity unaware components

27. Comparison OOPS 1990’s GUI recogs mediators application application
recognizers
meds
inter
interactors
Input
handler
Input
handler
stroke s c

28. Input Recognition Low Bandwidth, Ambiguous Input
Motivation and Design Space
Contribution: Toolkit-level support
Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input

29. Validation Doing what’s been done Architecture: 2 Toolkits
Burlap (complex app)
~15 Mediators (based on design space)
Architecture: 2 Toolkits
“direct” hookup in GUI world
Context Toolkit extensions (Dey)
Making new things possible: 4 problem areas

30. Problem Areas (UIST ’00) Errors & Ambiguity Mediation rejection errors
target ambiguity
Mediation
adding alternatives
occlusion

31. Target Ambiguity
Problem: There may be multiple targets of a user action
Example: clicking
Kabbash (95)
Worden (97)

32. Target Ambiguity
Problem: There may be multiple targets of a user action
Example: Clicking
Solution: Give the user a choice of all of the targets

33. Target Ambiguity Analysis:
Any interface involving mouse press/release
Requires separation of concerns
Works with all interactors
Problem: There may be multiple targets of a user action
Example: Clicking
Solution: Give the user a choice of all of the targets

34. Adding alternatives
Problem: The correct choice isn’t always present

35. Adding alternatives Problem: The correct choice isn’t always present
Example: word-prediction

36. Adding alternatives Problem: The correct choice isn’t always present
Example: word-prediction
Solution: Allow the user to add choices

37. Adding alternatives Problem: The correct choice isn’t always present
Example: word-prediction
Solution: Allow the user to add choices
Analysis:
Closely related choices (e.g. URL prediction)
Requires extensible mediators
Benefits from recognizer API

38. Contributions (Recognition)
Resolution of ambiguity in recognition through mediation
Architectural support (2 toolkits)
Exploration of domain of mediators

39. Future Work (Recognition)
Testing
Implicit Input (Current)
Intelligent Mediation
“Lazy” alternative generation
Meta-intelligence
General support for varied inputs

40. Input Recognition Low Bandwidth, Ambiguous Input
Motivation and Design Space
Contribution: Toolkit-level support
Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input

41. Brain-computer interface (Current)
Locked-in Syndrome
Neural Mouse Control

42. Low-Bandwidth Input No alternative modalities available
Need for efficient error handling
Challenge: interpret brain signal in as rich a form as possible

43. Automatic Conversion for Low Bandwidth Input
Web Browser Example:
Automatic URL extraction
Preview Area
Generalization – Next step for natural interaction

44. Conclusions Interaction design for ubiquity, disability
Dual augmentation
Ambiguity in recognition
Low bandwidth or error-prone input
Exploration of new interaction techniques
Four mediation problems
Low Bandwidth input
General, re-usable solutions
Domisilica
OOPS toolkit
Extended Context Toolkit

45. http://www.cc.gatech.edu/fce/errata/ jmankoff@cc.gatech.edu
Further Information

46. What about Alternative Explosion?
Fairly independent, discrete interactions
Some stuff stays in recognizers
Recognition API helps
Only cache ambiguous events
Work to be done…

47. How general is “general”?
Two Toolkits
Two complex applications
Library of existing mediators
Explorations of 4 problem areas (UIST 00)

48. Is subArctic doing the work here?
No, our minimal requirements are common in today’s toolkits:
An event-based toolkit
An input-handling module that delivers events to the appropriate places
A library of interactors/widgets
Access to source code (OOPS is not just a library!)

49. Comparing SILK and Burlap

50. Alternative Forms of Input
Cirrin (UIST 98)

51. Alternative Forms of Input
Cirrin (UIST 98)
Cerebral Palsy (Cursor Activity Recognition; Current)

52. Alternative Forms of Input
Cirrin (UIST 98)
Cerebral Palsy (Cursor Activity Recognition; Current)
Recognition

53. Interaction design for ambiguity: Supporting Creativity
Converting a the brain signal to music
Example
Two possible experiences
Trying to play a piano with ones feet
Having an artistic voice no one else can reproduce

54. Library of mediators Design space based on survey
Generic and re-usable
Three major classes
Repetition
Choice
Automatic
if (result is “dos”)
reject it
else
do nothing

56. Comparison OOPS 1990’s GUI recogs mediators application application
recognizers
meds
inter
interactors
Input
handler
Input
handler
stroke s c