A Mechanism for Learning, Attention Switching, and Cognition School of Electrical Engineering and Computer Science, Ohio University, USA Dagstuhl Seminar, March 27- April 1, Cathedral of Applied Information Systems University of Information Technology and Management Poland Janusz Starzyk

Motivated Learning Various pains, internal, and external signals compete for attention. Attention switching results from competition. Cognitive perception is aided by attention switching. Definition: Motivated learning (ML) is pain based motivation, goal creation and learning in embodied agent. ML applies to EI working in a hostile environment. Machine creates abstract goals based on the pain signals. It receives internal rewards for satisfying its goals

Reinforcement Learning Motivated Learning External rewards Predictable Objectives set by designer Maximizes the reward Potentially unstable Learning effort increases with complexity Always active Internal rewards Unpredictable Sets its own objectives Solves minimax problem Always stable Learns better in complex environment than RL Acts when needed

Primitive Goal Creation -+ Pain Dry soil Primitive level open tank sit on garbage refill faucet w. can water Dual pain Reinforcing a proper action

Abstract Goal Hierarchy Abstract goals are created to reduce abstract pains and to satisfy the primitive goals A hierarchy of abstract goals is created to satisfy the lower level goals Activation Stimulation Inhibition Reinforcement Difference Need Expectation -+ + Dry soil Primitive Level Level I Level II faucet - w. can open water + Sensory pathway (perception, sense) Motor pathway (action, reaction) Level III tank - refill

Drought Reservoir Irrigate Thirsty Water Drink Water Primitive Needs Dirty Wash in Water Abstract Needs Primitive needs

Drought Reservoir Public Money Irrigate Spend Money to Build Thirsty Water Drink Water Spend Money to Buy Primitive Needs Well Draw own Water Dirty Wash in Water Abstract Needs Abstract needs

Drought Reservoir Public Money Tourists' Attractions Irrigate Spend Money to Build Build Ecotourism Thirsty Water Drink Water Spend Money to Buy Primitive Needs Build Water Recreation Wealthy Taxpayers Rise Taxes Well Draw own Water Dirty Wash in Water Well Building Dig a Well Abstract Needs Ground Water Water Supply Abstract needs

Drought Reservoir Public Money Tourists' Attractions Irrigate Spend Money to Build Build Ecotourism Thirsty Water Drink Water Spend Money to Buy Primitive Needs Build Water Recreation Policy Develop Infrastructure Wealthy Taxpayers Rise Taxes Well Draw own Water Dirty Wash in Water Well Building Dig a Well Abstract Needs Employment Opportunities Ground Water Water Supply Receive Salary Resource Management and Planning Management Regulate Use Planning Abstract needs

6 levels of hierarchy Initially ML agent experiences similar primitive pain signal P p as RL agent. ML agent converges quickly to a stable performance. 10 levels of hierarchy Initially RL agent experiences lower primitive pain signal P p than ML agent. RL agents pain increases when environment is more hostile. ML vs. RL agents in hierarchical environments J.A. Starzyk, P. Raif, and A.-H. Tan, Mental Development and Representation Building through Motivated Learning, WCCI Special Session on Mental Architecture and Representation, Barcelona, Spain, July 18-23, 2010.Mental Development and Representation Building through Motivated Learning

Grid world problem Four kinds of resources distributed over 25 x 25 grid. P. Raif, J.A. Starzyk, Motivated Learning In Autonomous Systems, submitted to IJCNN Special Session on Autonomous learning of object representation and control, San Jose, CA, July 31-Aug. 5, 2011.

Intelligence Central executive Attention and attention switching Mental saccades Cognitive perception Cognitive action control Consciousness Photo:

Computational Model of Conscious Machine Semantic memory Sensory processors Data encoders/ decoders Sensory units Motor skills Motor processors Data encoders/ decoders Motor units Emotions, rewards, and sub-cortical processing Attention switching Action monitoring Motivation and goal processor Planning and thinking Episodic memory Queuing and organization of episodes Episodic Memory & Learning Central Executive Sensory-motor Inspiration: human brain Photo (brain):

Attention switching Action monitoring Motivation and goal processor Planning and thinking Central Executive Tasks o cognitive perception o attention o attention switching o motivation o goal creation and selection o thoughts o planning o learning, etc. Central Executive

Interacts with other units for o performing its tasks o gathering data o giving directions to other units No clearly identified decision center Decisions are influenced by o competing signals representing motivations, pains, desires, plans, and interrupt signals need not be cognitive or consciously realized o competition can be interrupted by attention switching signal Attention switching Action monitoring Motivation and goal processor Planning and thinking Central Executive

Attention Switching ! Dynamic process resulting from competition between representations related to motivations sensory inputs internal thoughts including spurious signals (like noise). blog.gigoo.org/.../

Input image A B C D A B C D A B C D WhatWhere Visual Saccades

Mental Saccades This in turn activates memory traces in the global workspace area that will be used for mental searches (mental saccades). Selected part of the image resulting from an eye saccade. Perceived input activates object recognition and associated areas of semantic and episodic memory.

Mental saccades in a conscious machine Perceptual saccadesChanging perception Changing environment Associative memory No Action control Loop 5 Loop 2 Perceptual saccadesChanging perception Changing environment Associative memory No Action control Advancement of a goal? Yes Learning Advancement of a goal? Advancement of a goal? Yes Learning Attention spotlight Mental saccades Continue search? Yes Loop 1 Attention spotlight Mental saccades Continue search? Continue search? Yes Loop 1 Plan action? No Yes Action? Yes No Changing motivation Loop 3 Loop 4 Plan action? No Yes Action? Yes No Changing motivation Loop 3 Loop 4 Loop 5 Loop 2

Action and subgoal planning Intended action Induced pain Dual pain Perception Pain reduction Next mental saccade Perform action Learning Pain Environment Decide action Attention spotlight Desired item Memory

Action control Predicted changes known Pain increase Predicted changes Intended action Associative memory Cognitive action control Lower level action control Action? Cognitive abort

A Mechanism for Learning, Attention Switching, and Cognition: Summary A mechanism of switching attention is fundamental for building cognitive machines. Attention switching is a dynamic process resulting from competition between goals, representations, sensory inputs, and internal thoughts. Motivated learning provides a mechanism for creation of abstract goals and continuous goal oriented motivation Mental saccades of the working memory are fundamental for cognitive thinking, attention switching, planning, and action monitoring

Motivations for actions are physically distributed o competing pain (need) signals are generated in various parts of machines mind Before a winner is selected, machine does not interpret the meaning of the competing signals Cognitive processing is predominantly sequential o winner of the internal competition is an instantaneous director of the cognitive thought process Top down supervision of perception, planning, internal thought or motor functions o results in conscious experience decision of what is observed and where is it planning how to respond o a train of such experiences constitutes consciousness A Mechanism for Learning, Attention Switching, and Cognition: Summary

Conclusions 1.Consciousness is computational 2.Motivated intelligent machines can be conscious

Questions ?? Photo:

References P.A.O. Haikonen, The cognitive approach to conscious machines. UK: Imprint Academic, J. Bach, Principles of Synthetic Intelligence PSI: An Architecture of Motivated Cognition, Oxford Univ. Press, B. J. Baars A cognitive theory of consciousness, Cambridge Univ. Press, A. Sloman, "Developing concept of consciousness," Behavioral and Brain Sciences, vol. 14 (4), pp , Dec J. Schmidhuber, Curious model-building control systems, Proceedings Int. Joint Conf. Neural Networks, Singapore, vol. 2, pp. 1458–1463, B. Bakker and J. Schmidhuber, Hierarchical Reinforcement Learning with Subpolicies Specializing for Learned Subgoals, in Proc. of the 2nd Int. Conf. on Neural Networks & Computational Intelligence, Switzerland, pp , A. Barto, S. Singh, and N. Chentanez, Intrinsically motivated learning of hierarchical collections of skills, Proc. 3rd Int. Conf. Development Learn., San Diego, CA, pp. 112–119, J. A. Starzyk, "Motivation in Embodied Intelligence" in Frontiers in Robotics, Automation and Control, Oct. 2008, pp J.A. Starzyk, Motivated Learning for Computational Intelligence, in Computational Modeling and Simulation of Intellect. ed. B. Igelnik, IGI Publ, Photo:

Embodied Intelligence –Mechanism: biological, mechanical or virtual agent with embodied sensors and actuators –EI acts on environment and perceives its actions –Environment hostility is persistent and stimulates EI to act –Hostility: direct aggression, pain, scarce resources, etc –EI learns so it must have associative self-organizing memory –Knowledge is acquired by EI Definition Embodied Intelligence (EI) is a mechanism that learns how to minimize hostility of its environment

Embodiment of a Mind Embodiment is a part of the environment that EI controls to interact with the rest of the environment It contains intelligence core and sensory motor interfaces under its control Necessary for development of intelligence Not necessarily constant or in the form of a physical body Boundary transforms modifying brains self- determination