DenK and iCat Two Projects on Cooperative Electronic Assistants (CEA’s) Robbert-Jan Beun, Rogier van Eijk & Huub Prüst Department of Information and Computing Sciences Utrecht University

DenK (Dialoogvoering en Kennisopbouw) - Collaborative project between University Tilburg, Eindhoven University and IPO – Fundamental principles of communication Generic architecture - Natural language interface - Knowledge representation - Object oriented animation - Result: implementation, 8 dissertations, conference proceedings, papers, reports, …

Triangle-paradigm A human-computer interface should correspond to our natural interaction in two ways: - Direct: -physical world -perception and action - Indirect -with humans by language -speech acts User direct observation and manipulation Taskdomain CEA language/symbolic softwareinterface

DenK properties - Mental representation -Context -Reasoning - Communication -Typed English text - Interaction -Evaluation (software interface) - Task domain -Electron microscope (EM) User observation & action Taskdomain CEA communication interaction

C2-lens mini- condensor specimen screen Spotsize Magnification On/Stand by Beam Shift Y Beam Shift X I/D Intensity Focus μP/nP Filament Emission GUN Example DenK Interface A: Why is the contrast of the visible image regulated by this aperture? B: Because the diffraction image is in the OA-plane when the microscope is in HM-mode.

Mental Representation Contexts in Type Theory Contains - ontological information - private beliefs about application domain (EM) - shared beliefs about application domain (EM) - dialogue history (discourse) Beliefs, Shared Beliefs & Dialogue Dialogue ≤ Shared beliefs ≤ Beliefs Extra context: Pending stack

Type Theory - Ontological information -Contexts and sequentiality [recipe : *s] [sobanito : recipe] [vegetarian : recipe  *p] [macrobiotic : recipe  *p] - Reasoning -Legal extension (well-formedness) [macrobiotic(sobanito) : *p] -Propositions-as-types (beliefs) [p17 : macrobiotic(sobanito)]

From Natural Language to Type Theory to Behavior structural analysis pragmatic processing semantic interpretation Natural language ULF Type Theory Behavior grammar lexicon conceptual lexicon context information error/help

DenK-architecture Evaluation NL-parser NL- production Contextual interpreter Response generator Pending stack Shared beliefs Private beliefs User Task domain CEA CC-interface HC-interface

Context Types in DenK - Physical Context - Private Belief - Shared Belief - Dialogue - Pending Stack - Sentence User Task domain CEA

Context 1 The dialogue U: Turn on the microscope! C: OK U: Which button is on? C: The blue one. U: Turn it off! - ‘ it ’ refers to the last introduced button

U: Is the button next to the start-button on? C: Yes U: Turn it off! - Two objects: button_1 and button_2 - ‘ it ’ refers not to the last introduced button! - ‘ it ’ refers to the last resolved button. Context 2 The dialogue

Context 3 The dialogue U: Is the red button on? C: Yes U1: Turn it/that/this off! U2: Turn that/this one off! U3: Turn the/that/this button off! - The object can be in different contexts - NL provides information about context search

Context 4 Beliefs of system Ontological U1: Can you restart the internet? S1: It is impossible to restart the internet. U2: Is this the engine of that car? S2: Yes. U2: Is this the key that starts it? S2: Yes. U2: Does it run smoothly?

Context 5 Goals Remove the black block!

Context 6 Physical domain Remove this one!

Conclusions Use linguistic component (not only icons and menus) Use linguistic component (not only icons and menus) This component should have knowledge about: This component should have knowledge about: –the task domain (ontology, private beliefs) –the user (shared beliefs, dialogue) –general communication principles Use context for interpretation and production of messages Use context for interpretation and production of messages The knowledge to build this component partly exists (but is distributed): The knowledge to build this component partly exists (but is distributed): –parsers (from language to mental states) –mental states in Type Theory (to model belief states) –Gricean rules for communication (in progress) In iCat many results from DenK can be reused In iCat many results from DenK can be reused