Programming Support for Natural Interaction Jennifer Mankoff CoC & GVU Center Georgia Tech

Acknowledgements Gregory Abowd & Scott Hudson FCE Group & GVU NSF & IBM

Natural Interaction Mouse + Keyboard work great for killer apps of the GUI world Problems Arise… Off the desktop (Ubiquitous computing) People with disabilities My interest: Providing general support for interaction design in this space of problems

Outline of Talk Output Input Augmenting physical objects and spaces Recognition in the face of errors Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas Low bandwidth, ambiguous input

Output: Augmenting physical objects and spaces PALplates User study Exploration Domisilica Dual augmentation General infrastructure

Contents: an Instant Jello Pudding:French Vanilla a 16oz Ferrara Capellini-Angel Hair a Hamburger Helper: Beef Noodle ...

Outline of Talk Output Input Augmenting physical objects and spaces General support for exploring new domains Input

Input Recognition Low Bandwidth, Ambiguous Input Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas Low Bandwidth, Ambiguous Input

Recognition Recognition is becoming ubiquitous Recognition is difficult to use A range of interface problems result OOPS toolkit helps solve them

Research Method Evaluators Builders Study individual problems/solutions Design space of possible solutions Builders Facilitate solutions to sets of problems (Architectural / toolkit solutions) Design space exploration by a larger community

Input Recognition Low Bandwidth, Ambiguous Input Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas Low Bandwidth, Ambiguous Input

Definitions Recognizer Error Mediation interprets user input creates ambiguity Error mistake from user’s perspective represented with ambiguity Mediation dialogue between user and computer used for resolving ambiguity

SILK (Landay, 1996)

Burlap

Design Space of Mediation Techniques Multimodal Suhm (98) McGee et al (98) Handwriting Huerst, Yang & Waibel (98) Speech Ainsworth & Pratt (92) Baber & Hone (93) IUIs Horvitz (99) Many Others…

Design Space of Mediation Techniques Two major classes Repetition Choice

Design Space of Mediation Techniques Two major classes Repetition Choice

Input Motivation: Alternatives to Keyboard/Mouse Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas Low Bandwidth, Ambiguous Input

OOPS Toolkit (CHI’00) Toolkit-level support for handling ambiguity in recognition Library of mediators Architectural support Based on subArctic

Architectural Support INDEPENDENT of any specific toolkit Separation of mediators, recognizers, and application Communication by a common internal model (ambiguous hierarchical events) Maintains ambiguity indefinitely

Related Work GUI GUI+Recognition Context Multimodal Garnet/interactors (Myers ‘90) X toolkits (Rosenberg et al. ’90) GUI+Recognition SATIN (Hong et al ’00) Extended Amulet (Landay & Myers ’93) PPSM (Hudson&Newell ’92) Artkit (Henry et al. ’90) Context Context Toolkit (Dey et al ’00) PARCPad, etc (Schilit et al. ’94) Multimodal OAA (Cohen et al ’94/Oviatt ’99) PAC-Amodeus (Nigay& Coutaz ’95)

Three key pieces Ambiguous hierarchical events Mediation subsystem Changes to event dispatch

Ambiguous Hierarchical Events Myers & Kosbie ‘96 stroke down drag up • • • s c down drag up • • • stroke s

Mediation Subsystem Ambiguity is identified automatically Presence of multiple interactor nodes Hierarchy is passed to mediators Recognizers, recipients informed of accept/reject decisions Accept/reject modifies hierarchy Application selects mediators from library

Event Dispatch Problem: all recipients must support ambiguity A sensed event arrives It is dispatched to all recognizers It is mediated rec1 rec2 Input handler event rec3 rec4 recn med2 med1 med4 med3 medn

Modified Event Dispatch 1. A sensed event arrives 2a. It is dispatched to ambiguity unaware components 2b. It is dispatched to all recognizers It is mediated Any accepted leaf nodes are dispatched to ambiguity unaware components

Comparison OOPS 1990’s GUI recogs mediators application application recognizers meds inter interactors Input handler Input handler stroke s c

Input Recognition Low Bandwidth, Ambiguous Input Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas Low Bandwidth, Ambiguous Input

Validation Doing what’s been done Architecture: 2 Toolkits Burlap (complex app) ~15 Mediators (based on design space) Architecture: 2 Toolkits “direct” hookup in GUI world Context Toolkit extensions (Dey) Making new things possible: 4 problem areas

Problem Areas (UIST ’00) Errors & Ambiguity Mediation rejection errors target ambiguity Mediation adding alternatives occlusion

Target Ambiguity Problem: There may be multiple targets of a user action Example: clicking Kabbash (95) Worden (97)

Target Ambiguity Problem: There may be multiple targets of a user action Example: Clicking Solution: Give the user a choice of all of the targets

Target Ambiguity Analysis: Any interface involving mouse press/release Requires separation of concerns Works with all interactors Problem: There may be multiple targets of a user action Example: Clicking Solution: Give the user a choice of all of the targets

Adding alternatives Problem: The correct choice isn’t always present

Adding alternatives Problem: The correct choice isn’t always present Example: word-prediction

Adding alternatives Problem: The correct choice isn’t always present Example: word-prediction Solution: Allow the user to add choices

Adding alternatives Problem: The correct choice isn’t always present Example: word-prediction Solution: Allow the user to add choices Analysis: Closely related choices (e.g. URL prediction) Requires extensible mediators Benefits from recognizer API

Contributions (Recognition) Resolution of ambiguity in recognition through mediation Architectural support (2 toolkits) Exploration of domain of mediators

Future Work (Recognition) Testing Implicit Input (Current) Intelligent Mediation “Lazy” alternative generation Meta-intelligence General support for varied inputs

Input Recognition Low Bandwidth, Ambiguous Input Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas Low Bandwidth, Ambiguous Input

Brain-computer interface (Current) Locked-in Syndrome Neural Mouse Control

Low-Bandwidth Input No alternative modalities available Need for efficient error handling Challenge: interpret brain signal in as rich a form as possible

Automatic Conversion for Low Bandwidth Input Web Browser Example: Automatic URL extraction Preview Area Generalization – Next step for natural interaction

Conclusions Interaction design for ubiquity, disability Dual augmentation Ambiguity in recognition Low bandwidth or error-prone input Exploration of new interaction techniques Four mediation problems Low Bandwidth input General, re-usable solutions Domisilica OOPS toolkit Extended Context Toolkit

http://www.cc.gatech.edu/fce/errata/ jmankoff@cc.gatech.edu Further Information http://www.cc.gatech.edu/fce/errata/ jmankoff@cc.gatech.edu

What about Alternative Explosion? Fairly independent, discrete interactions Some stuff stays in recognizers Recognition API helps Only cache ambiguous events Work to be done…

How general is “general”? Two Toolkits Two complex applications Library of existing mediators Explorations of 4 problem areas (UIST 00)

Is subArctic doing the work here? No, our minimal requirements are common in today’s toolkits: An event-based toolkit An input-handling module that delivers events to the appropriate places A library of interactors/widgets Access to source code (OOPS is not just a library!)

Comparing SILK and Burlap

Alternative Forms of Input Cirrin (UIST 98)

Alternative Forms of Input Cirrin (UIST 98) Cerebral Palsy (Cursor Activity Recognition; Current)

Alternative Forms of Input Cirrin (UIST 98) Cerebral Palsy (Cursor Activity Recognition; Current) Recognition

Interaction design for ambiguity: Supporting Creativity Converting a the brain signal to music Example Two possible experiences Trying to play a piano with ones feet Having an artistic voice no one else can reproduce

Library of mediators Design space based on survey Generic and re-usable Three major classes Repetition Choice Automatic if (result is “dos”) reject it else do nothing

Comparison OOPS 1990’s GUI recogs mediators application application recognizers meds inter interactors Input handler Input handler stroke s c