Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab.

Slides:



Advertisements
Similar presentations
Creating a Multimodal Design Environment Using Speech and Sketching Aaron Adler Student Oxygen Workshop September 12, 2003.
Advertisements

ARCHITECTURES FOR ARTIFICIAL INTELLIGENCE SYSTEMS
Chapter 11 user support. Issues –different types of support at different times –implementation and presentation both important –all need careful design.
Semantics Static semantics Dynamic semantics attribute grammars
Open Provenance Model Tutorial Session 2: OPM Overview and Semantics Luc Moreau University of Southampton.
Bayesian Network and Influence Diagram A Guide to Construction And Analysis.
SECOND MIDTERM REVIEW CS 580 Human Computer Interaction.
Using the Crosscutting Concepts As conceptual tools when meeting an unfamiliar problem or phenomenon.
Background information Formal verification methods based on theorem proving techniques and model­checking –to prove the absence of errors (in the formal.
Artificial Intelligence Knowledge-based Agents Russell and Norvig, Ch. 6, 7.
CPSC 322 Introduction to Artificial Intelligence September 20, 2004.
Architecture-driven Modeling and Analysis By David Garlan and Bradley Schmerl Presented by Charita Feldman.
File Systems and Databases
Designing Help… Mark Johnson Providing Support Issues –different types of support at different times –implementation and presentation both important.
Natural Language Processing AI - Weeks 19 & 20 Natural Language Processing Lee McCluskey, room 2/07
Knowledge Acquisition CIS 479/579 Bruce R. Maxim UM-Dearborn.
Marakas: Decision Support Systems, 2nd Edition © 2003, Prentice-Hall Chapter Chapter 1: Introduction to Decision Support Systems Decision Support.
Marakas: Decision Support Systems, 2nd Edition © 2003, Prentice-Hall Chapter Chapter 7: Expert Systems and Artificial Intelligence Decision Support.
Process Modeling SYSTEMS ANALYSIS AND DESIGN, 6 TH EDITION DENNIS, WIXOM, AND ROTH © 2015 JOHN WILEY & SONS. ALL RIGHTS RESERVED. 1 Roberta M. Roth.
Building the Design Studio of the Future Aaron Adler Jacob Eisenstein Michael Oltmans Lisa Guttentag Randall Davis October 23, 2004.
Intelligent Tutoring Systems Traditional CAI Fully specified presentation text Canned questions and associated answers Lack the ability to adapt to students.
CSC271 Database Systems Lecture # 21. Summary: Previous Lecture  Phases of database SDLC  Prototyping (optional)  Implementation  Data conversion.
Katanosh Morovat.   This concept is a formal approach for identifying the rules that encapsulate the structure, constraint, and control of the operation.
Chapter 14: Artificial Intelligence Invitation to Computer Science, C++ Version, Third Edition.
MGS4020_02.ppt/Jan 22, 2013/Page 1 Georgia State University - Confidential MGS 4020 Business Intelligence Ch 1 – Introduction to DSS Jan 22, 2013.
Context Tailoring the DBMS –To support particular applications Beyond alphanumerical data Beyond retrieve + process –To support particular hardware New.
Artificial intelligence project
Mobile Topic Maps for e-Learning John McDonald & Darina Dicheva Intelligent Information Systems Group Computer Science Department Winston-Salem State University,
A Cognitive Substrate for Natural Language Understanding Nick Cassimatis Arthi Murugesan Magdalena Bugajska.
Chapter 6 Use Cases. Use Cases: –Text stories Some “actor” using system to achieve a goal –Used to discover and record requirements –Serve as input to.
Keyword Searching and Browsing in Databases using BANKS Seoyoung Ahn Mar 3, 2005 The University of Texas at Arlington.
ISBN Chapter 3 Describing Semantics -Attribute Grammars -Dynamic Semantics.
Enabling Natural Interaction Randall Davis, Howard Shrobe Aaron Adler, Christine Alvarado, Mark Foltz, Tracy Hammond, Mike Oltmans, Metin Sezgin,Olga Veselova.
Slide 1 Propositional Definite Clause Logic: Syntax, Semantics and Bottom-up Proofs Jim Little UBC CS 322 – CSP October 20, 2014.
An Ontological Framework for Web Service Processes By Claus Pahl and Ronan Barrett.
Discovering Descriptive Knowledge Lecture 18. Descriptive Knowledge in Science In an earlier lecture, we introduced the representation and use of taxonomies.
Formal Specification of Intrusion Signatures and Detection Rules By Jean-Philippe Pouzol and Mireille Ducassé 15 th IEEE Computer Security Foundations.
University of Windsor School of Computer Science Topics in Artificial Intelligence Fall 2008 Sept 11, 2008.
Automated Critique of Sketched Mechanisms Jon Wetzel and Ken Forbus Qualitative Reasoning Group Northwestern University.
Artificial Intelligence Research Center Pereslavl-Zalessky, Russia Program Systems Institute, RAS.
Triplet Extraction from Sentences Technical University of Cluj-Napoca Conf. Dr. Ing. Tudor Mureşan “Jožef Stefan” Institute, Ljubljana, Slovenia Assist.
Chap#11 What is User Support?
Computer Science CPSC 322 Lecture 22 Logical Consequences, Proof Procedures (Ch 5.2.2)
Formal Specification: a Roadmap Axel van Lamsweerde published on ICSE (International Conference on Software Engineering) Jing Ai 10/28/2003.
Class Diagrams. Terms and Concepts A class diagram is a diagram that shows a set of classes, interfaces, and collaborations and their relationships.
Abdul Rahim Ahmad MITM 613 Intelligent System Chapter 10: Tools.
Artificial Intelligence
Issues in Temporal and Causal Inference Pei Wang Temple University, USA Patrick Hammer Graz University of Technology, Austria.
Some Thoughts to Consider 5 Take a look at some of the sophisticated toys being offered in stores, in catalogs, or in Sunday newspaper ads. Which ones.
#1 Make sense of problems and persevere in solving them How would you describe the problem in your own words? How would you describe what you are trying.
Maestro AI Vision and Design Overview Definitions Maestro: A naïve Sensorimotor Engine prototype. Sensorimotor Engine: Combining sensory and motor functions.
Network Management Lecture 13. MACHINE LEARNING TECHNIQUES 2 Dr. Atiq Ahmed Université de Balouchistan.
Mechanisms for Requirements Driven Component Selection and Design Automation 최경석.
Estimation of Distribution Algorithm and Genetic Programming Structure Complexity Lab,Seoul National University KIM KANGIL.
Human Computer Interaction Lecture 21 User Support
gesture features for coreference
Chapter 4: Business Process and Functional Modeling, continued
Business System Development
Human Computer Interaction Lecture 21,22 User Support
Architecture Components
Randall Davis Howard Shrobe
Artificial Intelligence: Logic agents
Multimodal Human-Computer Interaction New Interaction Techniques 22. 1
MGS 4020 Business Intelligence Ch 1 – Introduction to DSS Jun 7, 2018
Chapter 11 user support.
ONTOMERGE Ontology translations by merging ontologies Paper: Ontology Translation on the Semantic Web by Dejing Dou, Drew McDermott and Peishen Qi 2003.
Information Retrieval
Randall Davis Howard Shrobe
Representations & Reasoning Systems (RRS) (2.2)
Presentation transcript:

Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab

Michael Oltmans Roadmap  The problem Our approach Implementation –System architecture –How ASSISTANCE interprets descriptions –Demonstrating understanding Evaluation and contributions Related and future work

Michael Oltmans Sketches Models We have a sketch of a device A simulation model can be generated from the sketch Life is good… or is it?

Michael Oltmans

The Problem No representation of intended behavior People talk and sketch but the computer doesn’t understand

Michael Oltmans Task Understand descriptions of device behavior: –Given: A model of the device’s structure A natural explanation of the behavior –Generate a causal model of behavior

Michael Oltmans

Roadmap The problem  Our approach Implementation –System architecture –How ASSISTANCE interprets descriptions –Demonstrating understanding Evaluation and contributions Related and future work

Michael Oltmans Naturally Conveyed Explanations Natural input modalities –Sketched devices –Sketched gestures –Speech Natural content of descriptions –Causal –Behavioral

Michael Oltmans Example: Describing the Behavior of a Spring Tool:Description:

Michael Oltmans Example: Describing the Behavior of a Spring Tool:Description: Mechanical CAD Spring length = 2.3cm Rest length = 3.0cm

Michael Oltmans Example: Describing the Behavior of a Spring Tool:Description: Mechanical CAD Spring length = 2.3cm Rest length = 3.0cm Qualitative Reasoner (< (length spring) (rest-length spring))

Michael Oltmans Example: Describing the Behavior of a Spring Tool:Description: Mechanical CAD Spring length = 2.3cm Rest length = 3.0cm Qualitative Reasoner (< (length spring) (rest-length spring)) ASSISTANCE “The spring pushes the block”

Michael Oltmans Sources of power Conventions in explanations aide interpretation –Description order suggests causal order Constrained vocabulary Overlapping descriptions provide constraints on interpretations

Michael Oltmans Roadmap The problem Our approach  Implementation  System architecture  How ASSISTANCE interprets descriptions  Demonstrating understanding Evaluation and contributions Related and future work

Michael Oltmans A SSIST Recognize sketch A SSIST Recognize sketch SketchSpeech Causal Model and Simulation Causal Model and Simulation ViaVoice™ Recognize speech Parse ViaVoice™ Recognize speech Parse LTRE Truth Maintenance Rule System LTRE Truth Maintenance Rule System A SSISTANCE Interpret explanation

Michael Oltmans Outputs Consistent causal model –Tree –Nodes are events –Links indicate causal relationships Demonstration of understanding –Natural language descriptions of causality –Parameter constraints

Michael Oltmans The Representation of Utterances Input comes from ViaVoice™ : –Grammar constructed based on observed explanations –Tagged with parts of speech and semantic categories

Michael Oltmans Representing the parse tree SENTENCE SIMPLE_SENTENCE (… “body 1 pushes body 2” (S0) t1) SENTENCE SIMPLE_SENTENCE (… “body 1 pushes body 2” (S0) t1) DIRECT_OBJECT NOUN NOUN-PHRASE (… “body 2” (S0 t1 t3) t5) DIRECT_OBJECT NOUN NOUN-PHRASE (… “body 2” (S0 t1 t3) t5) PROPELS VERB (… “pushes” (S0 t1 t3) t4) PROPELS VERB (… “pushes” (S0 t1 t3) t4) SUBJECT NOUN NOUN-PHRASE (… “body 1” (S0 t1) t2) SUBJECT NOUN NOUN-PHRASE (… “body 1” (S0 t1) t2) VERB_PHRASE (… “pushes body 2” (S0 t1) t3) VERB_PHRASE (… “pushes body 2” (S0 t1) t3) “body 1 pushes body 2”

Michael Oltmans Steps In Interpreting Explanations: 1.Infer motions from annotations and build event representations 2.Find causal connections 3.Search for consistent causal structures 4.Pick best causal structure

Michael Oltmans Step 1: Inferring Motions from Annotations Inputs: –Arrows –Utterances “moves,” “pushes,” “the spring releases” Outputs: –(moves body-1 moves-body-1-394) –(describes arrow-2 moves-body-1-394)

Michael Oltmans Rule triggers: –Arrow –Arrow referent (i.e. a body) –The body is mobile Rule body records that: –The body moves –The arrow describes the path Inferring Motion From Arrows

Michael Oltmans (rlet ((?id (new-id “Moves” ?name))) (rassert! (:implies (:AND ?f1 ?f2 ?f3) (:AND (moves ?body ?id) (describes ?arrow ?id))) :ARROW-IS-MOTION))) Inferring Motion From Arrows (rule ((:TRUE (arrow ?arrow) :VAR ?f1) (:TRUE (arrow-referent ?arrow ?body) :VAR ?f2) (:TRUE (can-move ?body) :VAR ?f3) (:TRUE (name ?name ?body)))

Michael Oltmans Multi-Modal References Match a sentence whose subject is “this” and a pointing gesture Assert the referent as the subject of the sentence Limitations: –User must point at referent before the utterance –Allow one “this” per utterance

Michael Oltmans Redundant Events Redundant explanations lead to multiple move statements for some events Merge them into a unique event statement “Body 1” falls (moves body-1 id-1) (moves body-1 id-2) Event 1

Michael Oltmans Step 2: Find Causal Connections Plausible causes –Arrow indicating motion near another object –Exogenous forces Definite causes –“When … then …” utterances –“Body 1 pushes body 2”

Michael Oltmans Step 3: Search for Consistent Causal Structures Some events have several possible causes Find consistent causal chains Search –Forward looking depth-first-search –Avoids repeating bad choices by recording bad combinations of assumptions

Michael Oltmans Step 4: Find the Best Interpretation Filter out interpretations that have unnecessary exogenous causes Pick the interpretation that most closely matches the explanation order While there are multiple valid interpretations –Choose one event with multiple possible causes –Assume the causal relation whose cause has the earliest description time

Michael Oltmans Answer Queries and Adjust Parameters Queries: –Designer: What is body 2 involved in? –A SSISTANCE : The motion of body 3 causes the motion of body 2 which causes the motion of body 5 Parameter Adjustment –Set spring length

Michael Oltmans Roadmap The problem Our approach Implementation –System architecture –How ASSISTANCE interprets descriptions –Demonstrating understanding  Evaluation and contributions Related and future work

Michael Oltmans Limitations of the Implementation Scope of applicability restricted –State transitions are one step deep –Cannot handle conjunctions of causes Limited knowledge about common device patterns –Latches, linkages, etc… –Supports and prevents Natural language limitations –Use a full featured NL system like START –Formally determine the grammar

Michael Oltmans Evaluation of the Approach Advantages –Focus on behavior in accordance with survey results –Move away from rigidity of WIMP interfaces –Similar to person-to-person interaction Alternatives –More dialog and feedback –Natural vs. efficient –Open claim that the domain is adequately constrained

Michael Oltmans Contributions Understanding naturally conveyed descriptions of behavior Generating representations of device behavior –Match the designer’s explanation –Generate simple explanations of causality –Allow the calculation of simulation parameters

Michael Oltmans Related Work Understanding device sketches –Alvarado 2000 Multimodal interfaces –Oviatt and Cohen Causality –C. Rieger and M. Grinberg 1977

Michael Oltmans Future Work Direct manipulation Dialog Expand natural language capabilities Smart design tools