Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.

Similar presentations


Presentation on theme: "1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction."— Presentation transcript:

1 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction

2 2 Acknowledgements  Gregory Abowd & Scott Hudson  FCE Group & GVU  NSF & IBM

3 3 Natural Interaction  Mouse + Keyboard work great for killer apps of the GUI world  Problems Arise…  Off the desktop (Ubiquitous computing)  People with disabilities  My interest: Providing general support for interaction design in this space of problems

4 4 Outline of Talk  Output  Augmenting physical objects and spaces  Input  Recognition in the face of errors  Motivation and Design Space  Contribution: Toolkit-level support  Validation: 2 Toolkits, Problem areas  Low bandwidth, ambiguous input

5 5 Output: Augmenting physical objects and spaces  PALplates  User study  Exploration  Domisilica  Dual augmentation  General infrastructure

6 6 PALplates (CHI’97)  Based on study of paper prototype  Point of need  Dynamic UI  Identity  Location (FX-Pal)

7 7 Domisilica  Dual Augmentation  Augment virtual world with information about real world state  Augment real world with virtual services, info (recipes, notes)  General Infrastructure  Example app: Cyberfridge (CVE’98)

8 8 Contents: an Instant Jello Pudding:French Vanilla an Instant Jello Pudding:French Vanilla a 16oz Ferrara Capellini-Angel Hair a Hamburger Helper: Beef Noodle...

9 9 Outline of Talk  Output  Augmenting physical objects and spaces  General support for exploring new domains  Input

10 10 Input  Recognition  Motivation and Design Space  Contribution: Toolkit-level support  Validation: 2 Toolkits, Problem areas  Low Bandwidth, Ambiguous Input

11 11 Recognition  Recognition is becoming ubiquitous  Recognition is difficult to use  A range of interface problems result  OOPS toolkit helps solve them

12 12 Research Method  Evaluators  Study individual problems/solutions  Design space of possible solutions  Builders  Facilitate solutions to sets of problems (Architectural / toolkit solutions)  Design space exploration by a larger community

13 13 Input  Recognition  Motivation and Design Space  Contribution: Toolkit-level support  Validation: 2 Toolkits, Problem areas  Low Bandwidth, Ambiguous Input

14 14 Definitions  Recognizer  interprets user input  creates ambiguity  Error  mistake from user’s perspective  represented with ambiguity  Mediation  dialogue between user and computer  used for resolving ambiguity

15 15 SILK (Landay, 1996)

16 16 Burlap

17 17 Design Space of Mediation Techniques  Multimodal  Suhm (98)  McGee et al (98)  Handwriting  Huerst, Yang & Waibel (98)  Speech  Ainsworth & Pratt (92)  Baber & Hone (93)  IUIs  Horvitz (99)  Many Others…

18 18 Design Space of Mediation Techniques  Two major classes  Repetition  Choice

19 19 Design Space of Mediation Techniques  Two major classes  Repetition  Choice

20 20 Input  Motivation: Alternatives to Keyboard/Mouse  Recognition  Motivation and Design Space  Contribution: Toolkit-level support  Validation: 2 Toolkits, Problem areas  Low Bandwidth, Ambiguous Input

21 21 OOPS Toolkit (CHI’00)  Toolkit-level support for handling ambiguity in recognition  Library of mediators  Architectural support  Based on subArctic

22 22 Architectural Support  INDEPENDENT of any specific toolkit  Separation of mediators, recognizers, and application  Communication by a common internal model (ambiguous hierarchical events)  Maintains ambiguity indefinitely

23 23 Related Work  GUI  Garnet/interactors (Myers ‘90)  X toolkits (Rosenberg et al. ’90)  GUI+Recognition  SATIN (Hong et al ’00)  Extended Amulet (Landay & Myers ’93)  PPSM (Hudson&Newell ’92)  Artkit (Henry et al. ’90)  Context  Context Toolkit (Dey et al ’00)  PARCPad, etc (Schilit et al. ’94)  Multimodal  OAA (Cohen et al ’94/Oviatt ’99)  PAC-Amodeus (Nigay& Coutaz ’95)

24 24 Three key pieces  Ambiguous hierarchical events  Mediation subsystem  Changes to event dispatch

25 25 downdragup s Ambiguous Hierarchical Events stroke downdragup sc Myers & Kosbie ‘96

26 26 Mediation Subsystem  Ambiguity is identified automatically  Presence of multiple interactor nodes  Hierarchy is passed to mediators  Recognizers, recipients informed of accept/reject decisions  Accept/reject modifies hierarchy  Application selects mediators from library

27 27 med1 med2 med3 med4 medn rec1 rec2 rec3 rec4 recn Event Dispatch 1.A sensed event arrives event Input handler 2. It is dispatched to all recognizers 3. It is mediated Problem: all recipients must support ambiguity

28 28 Modified Event Dispatch 1. A sensed event arrives 2a. It is dispatched to ambiguity unaware components 2b. It is dispatched to all recognizers 3. It is mediated 4. Any accepted leaf nodes are dispatched to ambiguity unaware components

29 29 Comparison stroke interactors application recognizers application Input handler inter recogsmediators Input handler meds sc sc OOPS 1990’s GUI

30 30 Input  Recognition  Motivation and Design Space  Contribution: Toolkit-level support  Validation: 2 Toolkits, Problem areas  Low Bandwidth, Ambiguous Input

31 31 Validation  Doing what’s been done  Burlap (complex app)  ~15 Mediators (based on design space)  Architecture: 2 Toolkits  “direct” hookup in GUI world  Context Toolkit extensions (Dey)  Making new things possible: 4 problem areas

32 32 Problem Areas (UIST ’00)  Errors & Ambiguity  rejection errors  target ambiguity  Mediation  adding alternatives  occlusion

33 33 Target Ambiguity  Problem: There may be multiple targets of a user action  Example: clicking  Kabbash (95)  Worden (97)

34 34 Target Ambiguity  Problem: There may be multiple targets of a user action  Example: Clicking  Solution: Give the user a choice of all of the targets

35 35 Target Ambiguity  Problem: There may be multiple targets of a user action  Example: Clicking  Solution: Give the user a choice of all of the targets  Analysis:  Any interface involving mouse press/release  Requires separation of concerns  Works with all interactors

36 36 Adding alternatives  Problem: The correct choice isn’t always present

37 37 Adding alternatives  Problem: The correct choice isn’t always present  Example: word- prediction

38 38 Adding alternatives  Problem: The correct choice isn’t always present  Example: word- prediction  Solution: Allow the user to add choices

39 39 Adding alternatives  Problem: The correct choice isn’t always present  Example: word- prediction  Solution: Allow the user to add choices  Analysis:  Closely related choices (e.g. URL prediction)  Requires extensible mediators  Benefits from recognizer API

40 40 Contributions (Recognition)  Resolution of ambiguity in recognition through mediation  Architectural support (2 toolkits)  Exploration of domain of mediators

41 41 Future Work (Recognition)  Testing  Implicit Input (Current)  Intelligent Mediation  “Lazy” alternative generation  Meta-intelligence  General support for varied inputs

42 42 Input  Recognition  Motivation and Design Space  Contribution: Toolkit-level support  Validation: 2 Toolkits, Problem areas  Low Bandwidth, Ambiguous Input

43 43 Brain-computer interface (Current)  Locked-in Syndrome  Neural Mouse Control

44 44 Low-Bandwidth Input  No alternative modalities available  Need for efficient error handling  Challenge: interpret brain signal in as rich a form as possible

45 45 Automatic Conversion for Low Bandwidth Input  Web Browser Example:  Automatic URL extraction  Preview Area  Generalization – Next step for natural interaction

46 46 Conclusions  Interaction design for ubiquity, disability  Dual augmentation  Ambiguity in recognition  Low bandwidth or error-prone input  Exploration of new interaction techniques  Four mediation problems  Low Bandwidth input  General, re-usable solutions  Domisilica  OOPS toolkit  Extended Context Toolkit

47 47 Further Information http://www.cc.gatech.edu/fce/errata/ jmankoff@cc.gatech.edu

48 48 What about Alternative Explosion?  Fairly independent, discrete interactions  Some stuff stays in recognizers  Recognition API helps  Only cache ambiguous events  Work to be done…

49 49 How general is “general”?  Two Toolkits  Two complex applications  Library of existing mediators  Explorations of 4 problem areas (UIST 00)

50 50 Is subArctic doing the work here?  No, our minimal requirements are common in today’s toolkits:  An event-based toolkit  An input-handling module that delivers events to the appropriate places  A library of interactors/widgets  Access to source code (OOPS is not just a library!)

51 51 Comparing SILK and Burlap

52 52 Alternative Forms of Input  Cirrin (UIST 98)

53 53 Alternative Forms of Input  Cirrin (UIST 98)  Cerebral Palsy (Cursor Activity Recognition; Current)

54 54 Alternative Forms of Input  Cirrin (UIST 98)  Cerebral Palsy (Cursor Activity Recognition; Current)  Recognition

55 55 Interaction design for ambiguity: S upporting Creativity  Converting a the brain signal to music  Example Example  Two possible experiences  Trying to play a piano with ones feet  Having an artistic voice no one else can reproduce

56 56 Library of mediators  Design space based on survey  Generic and re-usable  Three major classes  Repetition  Choice  Automatic if (result is “dos”) reject it else do nothing

57 57

58 58 Comparison stroke interactors application recognizers application Input handler inter recogsmediators Input handler meds sc sc OOPS 1990’s GUI


Download ppt "1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction."

Similar presentations


Ads by Google