1. Enabling Natural Interaction Randall Davis, Howard Shrobe Aaron Adler, Christine Alvarado, Mark Foltz, Tracy Hammond, Mike Oltmans, Metin Sezgin,Olga Veselova MIT Artificial Intelligence Lab

2. Randall Davis Consider This Device...

3. Randall Davis Natural Interaction? What if we had to deal with one another as we deal with our software?

4. Randall Davis Let’s Talk

5. Randall Davis Our Vision

6. Randall Davis Our Model

7. Randall Davis Rationale Capture Rationale capture doesn’t happen. –Because it’s too much trouble Rationale capture tools don’t work –Because they’re too much trouble The key: it needs to be less trouble than it’s worth. The key to less trouble: break down the barriers

8. Randall Davis Sketch Understanding

9. Randall Davis Why Sketching? Informal Succinct Natural High in information content Preserve topological and spatial properties

10. Randall Davis Sketch Understanding Requires intelligence –at the signal level: early processing –at the interpretation level: handling ambiguity –at the engineering level: intended behavior

11. Randall Davis Other Research Additional domain: software diagrams Multi-modal interaction Describing and learning new icons Generating efficient recognizers

12. Randall Davis Sketching Software

13. Randall Davis Sketching Software

14. Randall Davis Sketching Software

15. Randall Davis Sketching Software

16. Randall Davis Sketching Software //Source file: c:\\video\\Deck.java public class Deck extends Hand implements Dealable { public Card theCard; public Game theGame; /** * @roseuid 3C21348C0257 */ public Deck() { } //Source file: c:\\video\\Dealable.java public interface Dealable { } //Source file: c:\\video\\Game.java public class Game { public Graphics theGraphics; public Deck theDeck; public Player thePlayer; /** * @roseuid 3C21348C02E3 */ public Game() { }

17. Randall Davis Multi-Modal Interaction “Three identical, equally spaced pendulums…” Graphical vs verbal communication: What’s best said, what’s best sketched?

18. Randall Davis New Icons A shape description language A drawing sequence description language Define AndGate line L1 L2 L3 arc A semi-circle A1 orientation(A1, 180) vertical L3 parallel L1 L2 same-horiz-position L1 L2 connected A.p1 L3.p1 connected A.p2 L3.p2 meets L1.p2 L3 meets L2.p2 L3

19. Randall Davis Sketch Understanding Architecture Mechanical Engineering devices simple compositions primitive objects geometryrectangles, curves

20. Randall Davis Sketch Understanding Architecture Mechanical Engineering devices simple compositions primitive objectssprings, joints geometryrectangles, curves

21. Randall Davis Sketch Understanding Architecture Mechanical Engineering devices simple compositionsratchet, latch primitive objectssprings, joints geometryrectangles, curves

22. Randall Davis Sketch Understanding Architecture Mechanical Engineering devicesmechanisms simple compositionsratchet, latch primitive objectssprings, joints geometryrectangles, curves

23. Randall Davis Sketch Understanding Architecture MechanicalSoftwareEngineering devicesmechanisms simple compositionsratchet, latch primitive objectssprings, jointsobjects, methods geometryrectangles, curves

24. Randall Davis Sketch Understanding Architecture MechanicalSoftwareEngineering devicesmechanisms simple compositionsratchet, latchhierarchies primitive objectssprings, jointsobjects, methods geometryrectangles, curves

25. Randall Davis Sketch Understanding Architecture MechanicalSoftwareEngineering devicesmechanismsmodules simple compositionsratchet, latchhierarchies primitive objectssprings, jointsobjects, methods geometryrectangles, curves

26. Randall Davis Design Rationale Capture Make it less trouble than it’s worth –Break down the barriers: eliminate interfaces –Natural interaction is enabled by intelligence About sketching, gesturing About the domain About the task