1. Enabling Natural Interaction Randall Davis Aaron Adler, Sonya Cates, Jacob Eisenstein, Tracy Hammond, Mike Oltmans, Metin Sezgin, Chen Li, David Pitman MIT CSAIL Randall Davis Aaron Adler, Sonya Cates, Jacob Eisenstein, Tracy Hammond, Mike Oltmans, Metin Sezgin, Chen Li, David Pitman MIT CSAIL

2. Consider This Device...

3. Our Model The designer sketches with pen and paper The observer interprets the sketch The observer and designer interact

4. Sketches are Ubiquitous

5. Sketches are Dead

6. Magic Paper

7. Demo

8. More Complex Example

11. A Prevasive Theme Natural interaction is knowledge-based.

12. Blackboard Hand-coded recognizers Early processing Architecture Overview

13. Define AndGate line L1 L2 L3 arc A semi-circle A1 orientation(A1, 180) vertical L3 parallel L1 L2 same-horiz-position L1 L2 connected A.p1 L3.p1 connected A.p2 L3.p2 meets L1.p2 L3 meets L2.p2 L3 Describing New Symbols

14. Define AndGate line L1 L2 L3 arc A semi-circle A1 orientation(A1, 180) vertical L3 parallel L1 L2 same-horiz-position L1 L2 connected A.p1 L3.p1 connected A.p2 L3.p2 meets L1.p2 L3 meets L2.p2 L3 Learning New Symbols From hand-drawn example to a shape description

15. Learning New Symbols Knowledge about human perception: Some properties are more important Experiments by Goldmeier (1936) identify perceptually relevant properties and relations; singularities Knowledge about human perception: Some properties are more important Experiments by Goldmeier (1936) identify perceptually relevant properties and relations; singularities

16. Blackboard Hand-coded recognizers Multi-modal recognizer Learn icon Symbolic Descr. Shape & Seq. Description 1 example Early processing Code Gen. New recognizers Architecture Overview

17. Pixels Objects Descriptions of Objects Application Sketch-Enabled Interaction Appl’n 2 Appl’n 3

18. Multi-Modal Interaction

19. “Three identical touching pendulums…” Graphical vs verbal communication: What’s best said, what’s best sketched?

21. Understanding Gestures Lexicon: What gestures do people make? Depends on task. Our task: explaining how something works. Lexicon: What gestures do people make? Depends on task. Our task: explaining how something works.

22. Understanding Gestures 96% of gestures refer to the diagram. Two-handed gestures are common. Deixis is more frequent.

23. Sketching and Talking: Laboratory Software

24. Natural, Unrestricted Input Domain: Sketching family trees Lexicon male female child-of married-to divorce Domain: Sketching family trees Lexicon male female child-of married-to divorce

25. Natural, Unrestricted Input Task is incremental Signal is noisy Styles vary Segmentation is difficult The signal is 2-d, non-chronological Task is incremental Signal is noisy Styles vary Segmentation is difficult The signal is 2-d, non-chronological 1 2 3 4 56 7 89 10 11 12

26. Low Level Context Use

27. Higher Level Context Grammar Lexicon male female child-of married-to divorce

28. Higher Level Context

29. Description Matching is Slow

30. Find Possible Objects Quickly

32. Domains Mechanical engineering Family trees Software design Circuit analysis Medical applications Mechanical engineering Family trees Software design Circuit analysis Medical applications

33. Sketching Software

36. //Source file: c:\\video\\Deck.java public class Deck extends Hand implements Dealable { public Card theCard; public Game theGame; /** * @roseuid 3C21348C0257 */ public Deck() { }

37. Sketching Software //Source file: c:\\video\\Deck.java public class Deck extends Hand implements Dealable { public Card theCard; public Game theGame; /** * @roseuid 3C21348C0257 */ public Deck() { }

38. Circuit Analysis

39. Specifying Shapes

40. Hardware

41. Summary Once the sketch is understood, many things are possible. Break down the barriers: eliminate interfaces Natural interaction is enabled by intelligence About sketching, gesturing, speech About the domain About the task Once the sketch is understood, many things are possible. Break down the barriers: eliminate interfaces Natural interaction is enabled by intelligence About sketching, gesturing, speech About the domain About the task