1. Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab

2. Michael Oltmans Roadmap  The problem Our approach Implementation –System architecture –How ASSISTANCE interprets descriptions –Demonstrating understanding Evaluation and contributions Related and future work

3. Michael Oltmans Sketches Models We have a sketch of a device A simulation model can be generated from the sketch Life is good… or is it?

4. Michael Oltmans

5. The Problem No representation of intended behavior People talk and sketch but the computer doesn’t understand

6. Michael Oltmans Task Understand descriptions of device behavior: –Given: A model of the device’s structure A natural explanation of the behavior –Generate a causal model of behavior

7. Michael Oltmans

8. Roadmap The problem  Our approach Implementation –System architecture –How ASSISTANCE interprets descriptions –Demonstrating understanding Evaluation and contributions Related and future work

9. Michael Oltmans Naturally Conveyed Explanations Natural input modalities –Sketched devices –Sketched gestures –Speech Natural content of descriptions –Causal –Behavioral

10. Michael Oltmans Example: Describing the Behavior of a Spring Tool:Description:

11. Michael Oltmans Example: Describing the Behavior of a Spring Tool:Description: Mechanical CAD Spring length = 2.3cm Rest length = 3.0cm

12. Michael Oltmans Example: Describing the Behavior of a Spring Tool:Description: Mechanical CAD Spring length = 2.3cm Rest length = 3.0cm Qualitative Reasoner (< (length spring) (rest-length spring))

13. Michael Oltmans Example: Describing the Behavior of a Spring Tool:Description: Mechanical CAD Spring length = 2.3cm Rest length = 3.0cm Qualitative Reasoner (< (length spring) (rest-length spring)) ASSISTANCE “The spring pushes the block”

14. Michael Oltmans Sources of power Conventions in explanations aide interpretation –Description order suggests causal order Constrained vocabulary Overlapping descriptions provide constraints on interpretations

15. Michael Oltmans Roadmap The problem Our approach  Implementation  System architecture  How ASSISTANCE interprets descriptions  Demonstrating understanding Evaluation and contributions Related and future work

16. Michael Oltmans A SSIST Recognize sketch A SSIST Recognize sketch SketchSpeech Causal Model and Simulation Causal Model and Simulation ViaVoice™ Recognize speech Parse ViaVoice™ Recognize speech Parse LTRE Truth Maintenance Rule System LTRE Truth Maintenance Rule System A SSISTANCE Interpret explanation

17. Michael Oltmans Outputs Consistent causal model –Tree –Nodes are events –Links indicate causal relationships Demonstration of understanding –Natural language descriptions of causality –Parameter constraints

18. Michael Oltmans The Representation of Utterances Input comes from ViaVoice™ : –Grammar constructed based on observed explanations –Tagged with parts of speech and semantic categories

19. Michael Oltmans Representing the parse tree SENTENCE SIMPLE_SENTENCE (… “body 1 pushes body 2” (S0) t1) SENTENCE SIMPLE_SENTENCE (… “body 1 pushes body 2” (S0) t1) DIRECT_OBJECT NOUN NOUN-PHRASE (… “body 2” (S0 t1 t3) t5) DIRECT_OBJECT NOUN NOUN-PHRASE (… “body 2” (S0 t1 t3) t5) PROPELS VERB (… “pushes” (S0 t1 t3) t4) PROPELS VERB (… “pushes” (S0 t1 t3) t4) SUBJECT NOUN NOUN-PHRASE (… “body 1” (S0 t1) t2) SUBJECT NOUN NOUN-PHRASE (… “body 1” (S0 t1) t2) VERB_PHRASE (… “pushes body 2” (S0 t1) t3) VERB_PHRASE (… “pushes body 2” (S0 t1) t3) “body 1 pushes body 2”

20. Michael Oltmans Steps In Interpreting Explanations: 1.Infer motions from annotations and build event representations 2.Find causal connections 3.Search for consistent causal structures 4.Pick best causal structure

21. Michael Oltmans Step 1: Inferring Motions from Annotations Inputs: –Arrows –Utterances “moves,” “pushes,” “the spring releases” Outputs: –(moves body-1 moves-body-1-394) –(describes arrow-2 moves-body-1-394)

22. Michael Oltmans Rule triggers: –Arrow –Arrow referent (i.e. a body) –The body is mobile Rule body records that: –The body moves –The arrow describes the path Inferring Motion From Arrows

23. Michael Oltmans (rlet ((?id (new-id “Moves” ?name))) (rassert! (:implies (:AND ?f1 ?f2 ?f3) (:AND (moves ?body ?id) (describes ?arrow ?id))) :ARROW-IS-MOTION))) Inferring Motion From Arrows (rule ((:TRUE (arrow ?arrow) :VAR ?f1) (:TRUE (arrow-referent ?arrow ?body) :VAR ?f2) (:TRUE (can-move ?body) :VAR ?f3) (:TRUE (name ?name ?body)))

24. Michael Oltmans Multi-Modal References Match a sentence whose subject is “this” and a pointing gesture Assert the referent as the subject of the sentence Limitations: –User must point at referent before the utterance –Allow one “this” per utterance

25. Michael Oltmans Redundant Events Redundant explanations lead to multiple move statements for some events Merge them into a unique event statement “Body 1” falls (moves body-1 id-1) (moves body-1 id-2) Event 1

26. Michael Oltmans Step 2: Find Causal Connections Plausible causes –Arrow indicating motion near another object –Exogenous forces Definite causes –“When … then …” utterances –“Body 1 pushes body 2”

27. Michael Oltmans Step 3: Search for Consistent Causal Structures Some events have several possible causes Find consistent causal chains Search –Forward looking depth-first-search –Avoids repeating bad choices by recording bad combinations of assumptions

28. Michael Oltmans Step 4: Find the Best Interpretation Filter out interpretations that have unnecessary exogenous causes Pick the interpretation that most closely matches the explanation order While there are multiple valid interpretations –Choose one event with multiple possible causes –Assume the causal relation whose cause has the earliest description time

29. Michael Oltmans Answer Queries and Adjust Parameters Queries: –Designer: What is body 2 involved in? –A SSISTANCE : The motion of body 3 causes the motion of body 2 which causes the motion of body 5 Parameter Adjustment –Set spring length

30. Michael Oltmans Roadmap The problem Our approach Implementation –System architecture –How ASSISTANCE interprets descriptions –Demonstrating understanding  Evaluation and contributions Related and future work

31. Michael Oltmans Limitations of the Implementation Scope of applicability restricted –State transitions are one step deep –Cannot handle conjunctions of causes Limited knowledge about common device patterns –Latches, linkages, etc… –Supports and prevents Natural language limitations –Use a full featured NL system like START –Formally determine the grammar

32. Michael Oltmans Evaluation of the Approach Advantages –Focus on behavior in accordance with survey results –Move away from rigidity of WIMP interfaces –Similar to person-to-person interaction Alternatives –More dialog and feedback –Natural vs. efficient –Open claim that the domain is adequately constrained

33. Michael Oltmans Contributions Understanding naturally conveyed descriptions of behavior Generating representations of device behavior –Match the designer’s explanation –Generate simple explanations of causality –Allow the calculation of simulation parameters

34. Michael Oltmans Related Work Understanding device sketches –Alvarado 2000 Multimodal interfaces –Oviatt and Cohen Causality –C. Rieger and M. Grinberg 1977

35. Michael Oltmans Future Work Direct manipulation Dialog Expand natural language capabilities Smart design tools