1. Mental Development and Representation Building through Motivated Learning Janusz A. Starzyk, Ohio University, USA, Pawel Raif, Silesian University of Technology, Poland, Ah-Hwee Tan, Nanyang Technological University, Singapore 2010 International Joint Conference on Neural Networks, Barcelona

2. Embodied Intelligence (EI) Embodiment of Mind Computational Approaches to Machine Learning How to Motivate a Machine Motivated Learning (ML) Building representation through motivated learning – ML agent in „Normal” vs. „Graded” Environment – ML agent vs. RL agent in „Graded” Environment Future work Outline

3. Traditional AI Embodied Intelligence Abstract intelligence – attempt to simulate “highest” human faculties: language, discursive reason, mathematics, abstract problem solving Environment model – Condition for problem solving in abstract way – “brain in a vat” Embodiment – knowledge is implicit in the fact that we have a body embodiment supports brain development Intelligence develops through interaction with environment – Situated in environment – Environment is its best model

4. Embodied Intelligence Mechanism: biological, mechanical or virtual agent with embodied sensors and actuators EI acts on environment and perceives its actions Environment hostility: is persistent and stimulates EI to act Hostility: direct aggression, pain, scarce resources, etc EI learns so it must have associative self-organizing memory Knowledge is acquired by EI  Definition Embodied Intelligence (EI) is a mechanism that learns how to minimize hostility of its environment

5. Intelligence An intelligent agent learns how to survive in a hostile environment.

6. Embodiment of a Mind  Embodiment: is a part of environment under control of the mind  It contains intelligence core and sensory motor interfaces to interact with environment  It is necessary for development of intelligence  It is not necessarily constant

7.  Changes in embodiment modify brain’s self-determination  Brain learns its own body’s dynamics  Self-awareness is a result of identification with own embodiment  Embodiment can be extended by using tools and machines  Successful operation is a function of correct perception of environment and own embodiment Embodiment of Mind

8. Computational Approaches to Machine Learning  Machine Learning  Supervised  Unsupervised  Reinforced  problems with Complex environments  lack of motivation  Motivated Learning  Definition  Need for benchmarks

9. How to Motivate a Machine ? A fundamental question is how to motivate an agent to do anything, and in particular, to enhance its own complexity? What drives an agent to explore the environment build representations and learn effective actions? What makes it successful learner in changing environments?

10. How to Motivate a Machine ?  Although artificial curiosity helps to explore the environment, it leads to learning without a specific purpose.  We suggest that the hostility of the environment, required for EI, is the most effective motivational factor.  Both are needed - hostility of the environment and intelligence that learns how to reduce the pain. Fig. englishteachermexico.wordpress.com/

11. Motivated Learning  Definition*: Motivated learning (ML) is pain based motivation, goal creation and learning in embodied agent.  It uses externally defined pain signals.  Machine is rewarded for minimizing the primitive pain signals.  Machine creates abstract goals based on the primitive pain signals.  It receives internal rewards for satisfying its abstract goals.  ML applies to EI working in a hostile environment. *J. A. Starzyk, Motivation in Embodied Intelligence, Frontiers in Robotics, Automation and Control, I-Tech Education and Publishing, Oct. 2008, pp. 83-110.

12. Neural self-organizing structures in ML Goal creation scheme an abstract pain is introduced by solving lower level pain Motivations and selection of a goal WTA competition selects motivation another WTA selects implementation a primitive pain is directly sensed thresholded curiosity based pain

13. Building representation through motivated learning Experiments…

14. Base Task Specification Environment  Environment consist of six different categories of resources.  Five of them have limited availability.  One, the most abstract resource is inexhaustible. FoodBankOfficeSandboxGrocerySchool The least abstract The most abstract

15. Agent uses resources performing proper actions. There are 36 possible actions but only six of them are meaningful and at a given situation (environment’s and agent’s state) there is usually one best action to perform. The problem is: determine which action should be performed renewing in time the most needed resource. Meaningful sensory-motor pairs and their effect on the environment: Base Experiment - Task Specification IdSENSORYMOTORINCREASESDECREASESPAIR Id 0FoodEatSugar levelFood supplies0 1GroceryBuyFood suppliesMoney at hand7 2BankWithdrawMoney at handSpending limits14 3OfficeWorkSpending limitsJob opportunities21 4SchoolStudyJob opportunitiesMental state28 5SandboxPlayMental state-36

16. How to simulate complexity and hostility of environment FoodFood BankBank OfficeOffice FeastFeast GroceryGrocery SchoolSchool 1 2 1. Complexity Different resources are available in the environment. Agent should learn dependencies between resources and its actions to operate properly. 2. Hostility Function which describes the probability of finding resources in the environment. Mild environment Harsh environment

17. Base Experiment Results 2 RL agent (left side) can learn dependencies between only few basic resources. In contrast ML agent is able to learn dependencies between all resources. In a harsh environment ML agent is able to control its environment (and limit its ‘primitive pain’) but RL agent cannot RLML 1

18. ML agent in „Normal” vs. „Graded” Environment  Two kinds of environments - “normal” (1) and “graded” (2).  “Graded” environment corresponds to gradual development and representation building  Simulations in four environments with: 6, 10, 14 and 18 different hierarchy levels each one representing different resource. 1 Time Resources … Time Resources 2 …

19.  ML agent learns more effectively in the ”graded” environments with gradually increasing complexity.  In a complex environment this difference becomes more significant.  “gradual” learning is beneficial to mental development ML agent in „Normal” vs. „Graded” Environment

20. ML agent vs. RL agent in „Graded” Environment.  The second group of experiments compares effectiveness of ML and RL based agents.  In this simulation we have used “graded” environments with gradually increasing complexity.  We simulated environments with: 6, 10, 14, 18 levels of hierarchy. Time Resources …

21. 6 levels of hierarchy  Initially ML agent experiences similar primitive pain signal P p as RL agent.  ML agent converges quickly to a stable performance. 10 levels of hierarchy  Initially RL agent experiences lower primitive pain signal P p than ML agent.  RL agent’s pain increases when environment is more hostile. ML agent vs. RL agent in „Graded” Environment.

22. 14 levels of hierarchy ML agent keeps learning while RL agent exploits early knowledge In effect, RL doesn’t learn all dependencies it time to survive 18 levels of hierarchy Similar results to 10 and 14 levels ML agent vs. RL agent in „Graded” Environment.

23. RL action state reward Future work action state GC reward GOALS (motivations) RL

24. References: Starzyk J.A., Raif P., Ah-Hwee Tan, Motivated Learning as an Extension of Reinforcement Learning, Fourth International Conference on Cognitive Systems, CogSys 2010, ETH Zurich, January 2010. Starzyk J.A., Raif P., Motivated Learning Based on Goal Creation in Cognitive Systems, Thirteenth International Conference on Cognitive and Neural Systems, Boston University, May 2009. J. A. Starzyk, Motivation in Embodied Intelligence, Frontiers in Robotics, Automation and Control, I-Tech Education and Publishing, Oct. 2008, pp. 83-110.

25. Questions?