1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction

2 Acknowledgements  Gregory Abowd & Scott Hudson  FCE Group & GVU  NSF & IBM

3 Natural Interaction  Mouse + Keyboard work great for killer apps of the GUI world  Problems Arise…  Off the desktop (Ubiquitous computing)  People with disabilities  My interest: Providing general support for interaction design in this space of problems

4 Outline of Talk  Output  Augmenting physical objects and spaces  Input  Recognition in the face of errors  Motivation and Design Space  Contribution: Toolkit-level support  Validation: 2 Toolkits, Problem areas  Low bandwidth, ambiguous input

5 Output: Augmenting physical objects and spaces  PALplates  User study  Exploration  Domisilica  Dual augmentation  General infrastructure

6 PALplates (CHI’97)  Based on study of paper prototype  Point of need  Dynamic UI  Identity  Location (FX-Pal)

7 Domisilica  Dual Augmentation  Augment virtual world with information about real world state  Augment real world with virtual services, info (recipes, notes)  General Infrastructure  Example app: Cyberfridge (CVE’98)

8 Contents: an Instant Jello Pudding:French Vanilla an Instant Jello Pudding:French Vanilla a 16oz Ferrara Capellini-Angel Hair a Hamburger Helper: Beef Noodle...

9 Outline of Talk  Output  Augmenting physical objects and spaces  General support for exploring new domains  Input

10 Input  Recognition  Motivation and Design Space  Contribution: Toolkit-level support  Validation: 2 Toolkits, Problem areas  Low Bandwidth, Ambiguous Input

11 Recognition  Recognition is becoming ubiquitous  Recognition is difficult to use  A range of interface problems result  OOPS toolkit helps solve them

12 Research Method  Evaluators  Study individual problems/solutions  Design space of possible solutions  Builders  Facilitate solutions to sets of problems (Architectural / toolkit solutions)  Design space exploration by a larger community

13 Input  Recognition  Motivation and Design Space  Contribution: Toolkit-level support  Validation: 2 Toolkits, Problem areas  Low Bandwidth, Ambiguous Input

14 Definitions  Recognizer  interprets user input  creates ambiguity  Error  mistake from user’s perspective  represented with ambiguity  Mediation  dialogue between user and computer  used for resolving ambiguity

15 SILK (Landay, 1996)

16 Burlap

17 Design Space of Mediation Techniques  Multimodal  Suhm (98)  McGee et al (98)  Handwriting  Huerst, Yang & Waibel (98)  Speech  Ainsworth & Pratt (92)  Baber & Hone (93)  IUIs  Horvitz (99)  Many Others…

18 Design Space of Mediation Techniques  Two major classes  Repetition  Choice

19 Design Space of Mediation Techniques  Two major classes  Repetition  Choice

20 Input  Motivation: Alternatives to Keyboard/Mouse  Recognition  Motivation and Design Space  Contribution: Toolkit-level support  Validation: 2 Toolkits, Problem areas  Low Bandwidth, Ambiguous Input

21 OOPS Toolkit (CHI’00)  Toolkit-level support for handling ambiguity in recognition  Library of mediators  Architectural support  Based on subArctic

22 Architectural Support  INDEPENDENT of any specific toolkit  Separation of mediators, recognizers, and application  Communication by a common internal model (ambiguous hierarchical events)  Maintains ambiguity indefinitely

23 Related Work  GUI  Garnet/interactors (Myers ‘90)  X toolkits (Rosenberg et al. ’90)  GUI+Recognition  SATIN (Hong et al ’00)  Extended Amulet (Landay & Myers ’93)  PPSM (Hudson&Newell ’92)  Artkit (Henry et al. ’90)  Context  Context Toolkit (Dey et al ’00)  PARCPad, etc (Schilit et al. ’94)  Multimodal  OAA (Cohen et al ’94/Oviatt ’99)  PAC-Amodeus (Nigay& Coutaz ’95)

24 Three key pieces  Ambiguous hierarchical events  Mediation subsystem  Changes to event dispatch

25 downdragup s Ambiguous Hierarchical Events stroke downdragup sc Myers & Kosbie ‘96

26 Mediation Subsystem  Ambiguity is identified automatically  Presence of multiple interactor nodes  Hierarchy is passed to mediators  Recognizers, recipients informed of accept/reject decisions  Accept/reject modifies hierarchy  Application selects mediators from library

27 med1 med2 med3 med4 medn rec1 rec2 rec3 rec4 recn Event Dispatch 1.A sensed event arrives event Input handler 2. It is dispatched to all recognizers 3. It is mediated Problem: all recipients must support ambiguity

28 Modified Event Dispatch 1. A sensed event arrives 2a. It is dispatched to ambiguity unaware components 2b. It is dispatched to all recognizers 3. It is mediated 4. Any accepted leaf nodes are dispatched to ambiguity unaware components

29 Comparison stroke interactors application recognizers application Input handler inter recogsmediators Input handler meds sc sc OOPS 1990’s GUI

30 Input  Recognition  Motivation and Design Space  Contribution: Toolkit-level support  Validation: 2 Toolkits, Problem areas  Low Bandwidth, Ambiguous Input

31 Validation  Doing what’s been done  Burlap (complex app)  ~15 Mediators (based on design space)  Architecture: 2 Toolkits  “direct” hookup in GUI world  Context Toolkit extensions (Dey)  Making new things possible: 4 problem areas

32 Problem Areas (UIST ’00)  Errors & Ambiguity  rejection errors  target ambiguity  Mediation  adding alternatives  occlusion

33 Target Ambiguity  Problem: There may be multiple targets of a user action  Example: clicking  Kabbash (95)  Worden (97)

34 Target Ambiguity  Problem: There may be multiple targets of a user action  Example: Clicking  Solution: Give the user a choice of all of the targets

35 Target Ambiguity  Problem: There may be multiple targets of a user action  Example: Clicking  Solution: Give the user a choice of all of the targets  Analysis:  Any interface involving mouse press/release  Requires separation of concerns  Works with all interactors

36 Adding alternatives  Problem: The correct choice isn’t always present

37 Adding alternatives  Problem: The correct choice isn’t always present  Example: word- prediction

38 Adding alternatives  Problem: The correct choice isn’t always present  Example: word- prediction  Solution: Allow the user to add choices

39 Adding alternatives  Problem: The correct choice isn’t always present  Example: word- prediction  Solution: Allow the user to add choices  Analysis:  Closely related choices (e.g. URL prediction)  Requires extensible mediators  Benefits from recognizer API

40 Contributions (Recognition)  Resolution of ambiguity in recognition through mediation  Architectural support (2 toolkits)  Exploration of domain of mediators

41 Future Work (Recognition)  Testing  Implicit Input (Current)  Intelligent Mediation  “Lazy” alternative generation  Meta-intelligence  General support for varied inputs

42 Input  Recognition  Motivation and Design Space  Contribution: Toolkit-level support  Validation: 2 Toolkits, Problem areas  Low Bandwidth, Ambiguous Input

43 Brain-computer interface (Current)  Locked-in Syndrome  Neural Mouse Control

44 Low-Bandwidth Input  No alternative modalities available  Need for efficient error handling  Challenge: interpret brain signal in as rich a form as possible

45 Automatic Conversion for Low Bandwidth Input  Web Browser Example:  Automatic URL extraction  Preview Area  Generalization – Next step for natural interaction

46 Conclusions  Interaction design for ubiquity, disability  Dual augmentation  Ambiguity in recognition  Low bandwidth or error-prone input  Exploration of new interaction techniques  Four mediation problems  Low Bandwidth input  General, re-usable solutions  Domisilica  OOPS toolkit  Extended Context Toolkit

47 Further Information

48 What about Alternative Explosion?  Fairly independent, discrete interactions  Some stuff stays in recognizers  Recognition API helps  Only cache ambiguous events  Work to be done…

49 How general is “general”?  Two Toolkits  Two complex applications  Library of existing mediators  Explorations of 4 problem areas (UIST 00)

50 Is subArctic doing the work here?  No, our minimal requirements are common in today’s toolkits:  An event-based toolkit  An input-handling module that delivers events to the appropriate places  A library of interactors/widgets  Access to source code (OOPS is not just a library!)

51 Comparing SILK and Burlap

52 Alternative Forms of Input  Cirrin (UIST 98)

53 Alternative Forms of Input  Cirrin (UIST 98)  Cerebral Palsy (Cursor Activity Recognition; Current)

54 Alternative Forms of Input  Cirrin (UIST 98)  Cerebral Palsy (Cursor Activity Recognition; Current)  Recognition

55 Interaction design for ambiguity: S upporting Creativity  Converting a the brain signal to music  Example Example  Two possible experiences  Trying to play a piano with ones feet  Having an artistic voice no one else can reproduce

56 Library of mediators  Design space based on survey  Generic and re-usable  Three major classes  Repetition  Choice  Automatic if (result is “dos”) reject it else do nothing

57

58 Comparison stroke interactors application recognizers application Input handler inter recogsmediators Input handler meds sc sc OOPS 1990’s GUI