Enabling Natural Interaction Randall Davis Aaron Adler, Sonya Cates, Jacob Eisenstein, Tracy Hammond, Mike Oltmans, Metin Sezgin, Chen Li, David Pitman MIT CSAIL Randall Davis Aaron Adler, Sonya Cates, Jacob Eisenstein, Tracy Hammond, Mike Oltmans, Metin Sezgin, Chen Li, David Pitman MIT CSAIL

Consider This Device...

Our Model The designer sketches with pen and paper The observer interprets the sketch The observer and designer interact

Sketches are Ubiquitous

Sketches are Dead

Magic Paper

Demo

More Complex Example

A Prevasive Theme Natural interaction is knowledge-based.

Blackboard Hand-coded recognizers Early processing Architecture Overview

Define AndGate line L1 L2 L3 arc A semi-circle A1 orientation(A1, 180) vertical L3 parallel L1 L2 same-horiz-position L1 L2 connected A.p1 L3.p1 connected A.p2 L3.p2 meets L1.p2 L3 meets L2.p2 L3 Describing New Symbols

Define AndGate line L1 L2 L3 arc A semi-circle A1 orientation(A1, 180) vertical L3 parallel L1 L2 same-horiz-position L1 L2 connected A.p1 L3.p1 connected A.p2 L3.p2 meets L1.p2 L3 meets L2.p2 L3 Learning New Symbols From hand-drawn example to a shape description

Learning New Symbols Knowledge about human perception: Some properties are more important Experiments by Goldmeier (1936) identify perceptually relevant properties and relations; singularities Knowledge about human perception: Some properties are more important Experiments by Goldmeier (1936) identify perceptually relevant properties and relations; singularities

Blackboard Hand-coded recognizers Multi-modal recognizer Learn icon Symbolic Descr. Shape & Seq. Description 1 example Early processing Code Gen. New recognizers Architecture Overview

Pixels Objects Descriptions of Objects Application Sketch-Enabled Interaction Appl’n 2 Appl’n 3

Multi-Modal Interaction

“Three identical touching pendulums…” Graphical vs verbal communication: What’s best said, what’s best sketched?

Understanding Gestures Lexicon: What gestures do people make? Depends on task. Our task: explaining how something works. Lexicon: What gestures do people make? Depends on task. Our task: explaining how something works.

Understanding Gestures 96% of gestures refer to the diagram. Two-handed gestures are common. Deixis is more frequent.

Sketching and Talking: Laboratory Software

Natural, Unrestricted Input Domain: Sketching family trees Lexicon male female child-of married-to divorce Domain: Sketching family trees Lexicon male female child-of married-to divorce

Natural, Unrestricted Input Task is incremental Signal is noisy Styles vary Segmentation is difficult The signal is 2-d, non-chronological Task is incremental Signal is noisy Styles vary Segmentation is difficult The signal is 2-d, non-chronological

Low Level Context Use

Higher Level Context Grammar Lexicon male female child-of married-to divorce

Higher Level Context

Description Matching is Slow

Find Possible Objects Quickly

Domains Mechanical engineering Family trees Software design Circuit analysis Medical applications Mechanical engineering Family trees Software design Circuit analysis Medical applications

Sketching Software

//Source file: c:\\video\\Deck.java public class Deck extends Hand implements Dealable { public Card theCard; public Game theGame; /** 3C21348C0257 */ public Deck() { }

Sketching Software //Source file: c:\\video\\Deck.java public class Deck extends Hand implements Dealable { public Card theCard; public Game theGame; /** 3C21348C0257 */ public Deck() { }

Circuit Analysis

Specifying Shapes

Hardware

Summary Once the sketch is understood, many things are possible. Break down the barriers: eliminate interfaces Natural interaction is enabled by intelligence About sketching, gesturing, speech About the domain About the task Once the sketch is understood, many things are possible. Break down the barriers: eliminate interfaces Natural interaction is enabled by intelligence About sketching, gesturing, speech About the domain About the task