1. A Cognitive Substrate for Natural Language Understanding Nick Cassimatis Arthi Murugesan Magdalena Bugajska

2. Language & Cognition N. L. Cassimatis, J. Trafton, M. Bugajska, A. Schultz (2004). Integrating Cognition, Perception and Action through Mental Simulation in Robots. Journal of Robotics and Autonomous Systems. Volume 49, Issues 1-2, 30 November 2004, Pages 13-23.

3. What is difficult? Integration of various sources of information and constraints Language Social cues (pointing) Visual information Concepts like object Spatial, physics

4. Why integration is difficult? Fodor’s Modularity of Mind Properties of Modular systems: Domain specificity : certain kinds of inputs Informational encapsulation Shallow outputs Mind’s Central Processing Vision Motor ReasoningLanguage Pictures Sounds Physical objects ? ???

5. Problems lead to a different goal and Tailored Evaluations Current standards in AI have become: Not sentence understanding or question answering but -Part of speech tagging (98%) -PCFG (Probabilistic Context Free Grammar) -Evaluation Metrics -Precision & Recall – 90% -Exact match – 20- 40% HPSG semantics oriented – 70%

6. Our approach – Substrate

7. Non Modular Focus of Attention (buffer) Identity Difference Temporal Constraint Identity Hypothesis World Event Space Temporal Perception Category Conflict Resolution Substrate: Representation Procedural Multiple processes Language is a part of and interacts freely with the greater cognitive system N.L. Cassimatis (2006). A Cognitive Substrate for Human-Level Intelligence. AI Magazine. Volume 27 Number 2.

8. Substrate Mappings: The particular substrate : Physical reasoning : –N. L. Cassimatis (2002). Polyscheme: A Cognitive Architecture for Integrating Multiple Representation and Inference Schemes. Doctoral Dissertation, Media Laboratory, Massachusetts Institute of Technology, Cambridge, MA Word Learning : –M. Bugajska, N.L. Cassimatis (2006). Beyond Association: Social Cognition in Word Learning. In Proceedings of the International Conference on Development and Learning. Social Cognition: –P. Bello, N.L. Cassimatis (2006). Developmental Accounts of Theory-of-Mind Acquisition: Achieving Clarity via Computational Cognitive Modeling. In Proceedings of 28th Annual Conference of the Cognitive Science Society. –P. Bello & N.L. Cassimatis (2006). Understanding other Minds: A Cognitive Modeling Approach. In Proceedings of the International Conference on Cognitive Modeling.

9. Language to Substrate Mapping

10. HPSG Mapping: A. Murugesan, N.L. Cassimatis (2006). A Model of Syntactic Parsing Based on Domain-General Cognitive Mechanisms. In Proceedings of 28th Annual Conference of the Cognitive Science Society.

11. Example of Syntax Semantics interaction

12. Semantics & Syntax Interaction E. g. : Given a sentence with an ambiguous word – choose the correct interpretation of the word; “The bug needs a battery” bug animal insect System error listening device annoy (verb) Eavesdrop (verb)

13. Implementation (default) Rules: 1.By default the most probable [bug - animal insect] is chosen -e.g. of such a sentence : “The bug crawled” Phonology ?phrase ‘bug’ ~~> Lexicon ?phrase animalBug Abnormality predicates are used to prioritize interpretations Phonology ?phrase ‘bug’ + Blocked ?phrase animalBug ~~> Lexicon ?phrase systemBug Phonology ?phrase ‘bug’ + Blocked ?phrase systemBug ~~> Lexicon ?phrase listeningDeviceBug Phonology ?phrase ‘bug’ + Blocked ?phrase listeningDeviceBug ~~> Lexicon ?phrase annoyBug Phonology ?phrase ‘bug’ + Blocked ?phrase annoyBug ~~> Lexicon ?phrase eavesdropBug Blocked ?phrase ?prevLexicon = = > NOT Lexicon ?phrase ?prevLexicon likely!

14. Implementation (Semantics) Two implicit requirements here : 1. generate semantics of a sentence 2. availability of background information Walk through the example : “The bug needs a battery”

15. Background knowledge Entity AbstractPhysical OrganicInorganic ISA(?obj, Inorganic) = = > ISA(?obj, Physical) ISA(?obj, Organic) = = > ISA(?obj, Physical) ISA(?obj, Inorgainc ) = = > NOT ISA (?obj, Organic) ISA(?obj, Orgainc ) = = > NOT ISA (?obj, Inorganic) Category hierarchy Need ( ?object, ?neededObj) + ISA(?neededObj,battery) = = > ISA(?object, Inorganic)

16. Conflict in Semantics Lexicon(?phrase,animalBug) + Referent(?phrase, ?phraseRef) = = > ISA(?phraseRef,Organic) According to default rule Lexicon(?phrase,animalBug) is likely true (l,?) Therefore by the above rule ISA(?phraseRef,Organic) is also likely true However once the sentence is formed and Needs(?phraseRef, ?batteryObj) is asserted ; according to background knowledge ?phraseRef must be Inorganic and NOT Organic! i.e. ISA(?phraseRef,Organic) is Certainly false (?,C) animalBug animalBug-1 (l,C) conflict

17. Contribution A framework for integration Implausibility of non modular approach is reduced Learnability of language Seamless integration