1. EE141 Emergence of the Embodied Intelligence How to Motivate a Machine ? Janusz Starzyk School of Electrical Engineering and Computer Science, Ohio University, USA www.ent.ohiou.edu/~starzyk Cognitive Neuroscience and Embodied Intelligence

2. EE141  Traditional Artificial Intelligence  Embodied Intelligence (EI)  Embodiment of Mind  EI Interaction with Environment  How to Motivate a Machine  Goal Creation Hierarchy  GCS Experiment  Motivated Learning  Challenges of EI  We need to know how to organize it  We need means to implement it  We need resources to build and sustain its operation  Promises of EI  To economy  To society Outline

3. EE141  “…Perhaps the last frontier of science – its ultimate challenge- is to understand the biological basis of consciousness and the mental process by which we perceive, act, learn and remember..” from Principles of Neural Science by E. R. Kandel et al.  E. R. Kandel won Nobel Price in 2000 for his work on physiological basis of memory storage in neurons.  “… The question of intelligence is the last great terrestrial frontier of science...” from Jeff Hawkins On Intelligence.  Jeff Hawkins founded the Redwood Neuroscience Institute devoted to brain research Intelligence AI’s holy grail From Pattie Maes MIT Media Lab

4. EE141 Is what is intelligence?

5. EE141 Various Definitions of Intelligence  The American Heritage Dictionary: l The capacity to acquire and apply knowledge. l The faculty of thought and reason.  Webster Dictionary: l The act or state of knowing; the exercise of the understanding. l The capacity to know or understand; readiness of comprehension;  Wikipedia – The Free Encyclopedia: l The capacity to reason, plan, solve problems, think abstractly, comprehend ideas and language, and learn.reasonplansolve problemsabstractlylanguagelearn  Kaplan & Sadock: l The ability to learn new things, recall information, think rationally, apply knowledge and solve problems.  On line dictionary dict.die.netdict.die.net l The ability to comprehend; to understand and profit from experience  The classical behavioral/biologists: l The ability to adapt to new conditions and to successfully cope with life situations.  Dr. C. George Boeree, professor in the Psychology Department at Shippensburg University:Psychology DepartmentShippensburg University l A person's capacity to (1) acquire knowledge (i.e. learn and understand), (2) apply knowledge (solve problems), and (3) engage in abstract reasoning.  Stanford University Professor of Computer Science Dr. John McCarthy, a pioneer in AI: l The computational part of the ability to achieve goals in the world.  Scientists in Psychology: l Ability to remember and use what one has learned, in order to solve problems, adapt to new situations, and understand and manipulate one’s reality.understandreality

6. EE141 From http://www.indiana.edu/~intell/map.shtml Mainstream Science on Intelligence December 13, 1994: An Editorial With 52 Signatories, by Linda S. Gottfredson, University of Delaware Intelligence is a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience. Intelligence

7. EE141 Animals’ Intelligence  Defining intelligence through humans is not appropriate to design intelligent machines: –Animals are intelligent too  Dog IQ test:  Dogs can learn 165 words (similar to 2 year olds)  Average dog has the mental abilities of a 2-year-old child (or better)  They would beat a 3- or 4-year-old in basic arithmetic,  Dogs show some basic emotions, such as happiness, anger and disgust  “The social life of dogs is very complex - more like human teenagers - interested in who is moving up in the pack, who is sleeping with who etc,“ says professor Stanleay Coren from University of British Columbia  Border collies, poodles, and german shepards are the smartest dogs

8. EE141 Traditional AI Embodied Intelligence  Abstract intelligence  attempt to simulate “highest” human faculties: –language, discursive reason, mathematics, abstract problem solving  Environment model  Condition for problem solving in abstract way  “brain in a vat”  Embodiment  knowledge is implicit in the fact that we have a body –embodiment supports brain development  Intelligence develops through interaction with environment  Situated in environment  Environment is its best model

9. EE141 Design principles of intelligent systems Design principles  synthetic methodology  time perspectives  emergence  diversity/compliance  frame-of-reference  complete agent principle from Rolf Pfeifer “Understanding of Intelligence” From: www.spectrum.ieee.org/.../biorobot11f-thumb.jpg

10. EE141 Design principles of intelligent systems from Rolf Pfeifer “Understanding of Intelligence”, 1999  Interaction with complex environment  ecological balance  redundancy principle  parallel, loosely coupled processes  asynchronous  sensory-motor coordination  value principle  cheap design Agent Drawing by Ciarán O’Leary- Dublin Institute of Technology

11. EE141 The principle of “cheap design”  intelligent agents: “cheap”  exploitation of ecological niche  economical (but redundant)  exploitation of specific physical properties of interaction with real world

12. EE141 Principle of “ecological balance”  balance / task distribution between  morphology  neuronal processing (nervous system)  materials  environment  balance in complexity  given task environment  match in complexity of sensory, motor, and neural system

13. EE141 The redundancy principle  redundancy prerequisite for adaptive behavior  partial overlap of functionality in different subsystems  sensory systems: different physical processes with “information overlap”

14. EE141 Generation of sensory stimulation through interaction with environment  multiple modalities  constraints from morphology and materials  generation of correlations through physical process  basis for cross- modal associations

15. EE141 The principle of sensory-motor coordination  self-structuring of sensory data through interaction with environment  physical process — not „computational“  prerequisite for learning Holk Cruse no central control only local neuronal communication global communication through environment neuronal connections

16. EE141 The principle of parallel, loosely coupled processes  Intelligent behavior emergent from agent-environment interaction  Large number of parallel, loosely coupled processes  Asynchronous  Coordinated through agent’s –sensory-motor system –neural system –interaction with environment

17. EE141 So what is an Embodied Intelligence ?

18. EE141 Embodied Intelligence  Definition  Embodied Intelligence (EI) is a mechanism that learns how to survive in a hostile environment –Mechanism: biological, mechanical or virtual agent with embodied sensors and actuators –EI acts on environment and perceives its actions –Environment hostility is persistent and stimulates EI to act –Hostility: direct aggression, pain, scarce resources, etc –EI learns so it must have associative self-organizing memory –Knowledge is acquired by EI

19. EE141  EI mimics biological intelligent systems, extracting general principles of intelligent behavior and applying them to design intelligent agents.  Knowledge is not entered into such systems, but rather is a result of their successful interaction with the environment.  Embodied intelligent systems adapt to unpredictable and dynamic situations in the environment by learning, which gives them a high degree of autonomy.  Learning in such systems is incremental, with continuous prediction of the input associations based on the emerging models - only new information is registered in the memory. Embodied Intelligence

20. EE141 What is Embodiment of a Mind?

21. EE141 Embodiment of a Mind  Embodiment of a mind is a part of environment under control of the mind  It contains intelligence core and sensory motor interfaces to interact with environment  It is necessary for development of intelligence  It is not necessarily constant or in the form of a physical body  Boundary of embodiment transforms modifying brain’s self-determination

22. EE141  Brain learns own body’s dynamic  Self-awareness is a result of identification with own embodiment  Embodiment can be extended by using tools and machines  Successful operation is a function of correct perception of environment and own embodiment Embodiment of Mind

23. EE141 Requirements for Embodied Intelligence  State oriented  Learns spatio-temporal patterns  Situated in time and space  Learning  Perpetual learning  Screening for novelty  Value driven  Pain detection  Pain management  Goal creation  Competing goals  Emergence  artificial evolution  self-organization

24. EE141 INPUTOUTPUT Simulation or Real-World System Task Environment Agent Architecture Long-term Memory Short-term Memory Reason Act Perceive RETRIEVALLEARNING EI Interaction with Environment From Randolph M. Jones, P : www.soartech.com

25. EE141 Kandel Fig. 23-5 Sensory Inputs Coding How do we process and represent sensory information? Richard Axel, 1995 Foot Hip Trunk Arm Hand Face Tongue Larynx Kandel Fig. 30-1 Visual, auditory, olfactory, tactile, smell -> motor

26. EE141 Challenges of Embodied Intelligence

27. EE141 Challenges of Embodied Intelligence  Development of sensory interfaces  Active vision  Speech processing  Tactile, smell, taste, temperature, pressure sensing  Additional sensing –Infrared, radar, lidar, ultrasound, GPS, etc. –Can too many senses be less useful?  Development of pain sensors  Energy, temperature, pressure, acceleration level  Teacher input  Development of motor interfaces  Arms, legs, fingers, eye movement Intelligence core Embodiment Sensors Actuators Environment

28. EE141 Challenges of Embodied Intelligence (cont.)  Finding algorithmic solutions for  Association, memory, sequence learning, invariance building, representation, anticipation, value learning, goal creation, planning  Development of circuits for neural computing  Determine organization of artificial minicolumn  Self-organized hierarchy of minicolumns for sensing and motor control  Self-organization of goal creation pathway

29. EE141  V. Mountcastle argues that all regions of the brain perform the same computational algorithm V. Mountcastle  Groups of neurons (minicolumns) connected in a pseudorandom way  Same structure  Minicolumns organized in macrocolumns VB Mountcastle (2003). Introduction [to a special issue of Cerebral Cortex on columns]. Cerebral Cortex, 13, 2-4. Human Intelligence – Cortex Uniform Structure

30. EE141 “ The basic unit of cortical operation is the minicolumn … It contains of the order of 80-100 neurons except in the primate striate cortex, where the number is more than doubled. The minicolumn measures of the order of 30-50  m in transverse diameter, separated from adjacent minicolumns by vertical, cell-sparse zones … The minicolumn is produced by the iterative division of a small number of progenitor cells in the neuroepithelium. ” (Mountcastle) Mini Columns Copyright © 2006-2008, all rights reserved, Visualbiotech

31. EE141  Sensory neurons are responsible for representing environment  receive inputs from sensors or sensory neurons on lower level  represent the environment  receive feedback input from motor and higher level neurons  help to activate motor and reinforcement neurons  Motor neurons are responsible for actions and skills  are activated by reinforcement and sensory neurons  activate actuators or provide an input to lower level motor neurons  provide planning inputs to sensory neurons  Reinforcement neurons are responsible for building the value system, goal creation, learning, and exploration  receive inputs from reinforcement neurons on the lower level  receive inputs from sensory neurons  provide inputs to motor neurons  initiate learning and force exploration Artificial Minicolumn Organization

32. EE141  Sensory and motor pathways are interconnected on different hierarchical levels outputs internal representations Sensory-motor Coordination inputs

33. EE141 Sensory-motor Coordination R: representation E: expectation A: association D: direction P: planning

34. EE141 How to Motivate a Machine ? A fundamental question is what motivates an agent to do anything, and in particular, to enhance its own complexity? What drives an agent to explore the environment and learn ways to effectively interact with it?

35. EE141 How to Motivate a Machine ?  Pfeifer claims that an agent’s motivation should emerge from the developmental process.  He called this the “motivated complexity” principle.  Chicken and egg problem? An agent must have a motivation to develop while motivation comes from development?  Steels suggested equipping an agent with self-motivation.  “Flow” experienced when people perform their expert activity well would motivate to accomplish even more complex tasks.  Humans get internal reward for activities that are slightly above their level of development (Csikszentmihalyi).  But what is the mechanism of “flow”?  Oudeyer proposed an intrinsic motivation system.  Motivation comes from a desire to minimize the prediction error.  Similar to “artificial curiosity” presented by Schmidhuber.

36. EE141 How to Motivate a Machine ? Can a machine that only implements externally given goals be intelligent? If not how these goals can be created? There is a need for a hierarchy of values. Not all values can be predetermined by the designer. There is a need for motivation to act, explore and learn. As machine makes new observations about the environment, there is a need to relate them to goals and values and create new goals and values.

37. EE141 How to Motivate a Machine ?  Exploration is needed in order to learn and to model the environment.  But is this mechanism the only motivation we need to develop intelligence?  Can “flow” ideas explain goal oriented learning?  Can we find a more efficient mechanism for learning?  I suggest a simpler mechanism to motivate a machine.  Although artificial curiosity helps to explore the environment, it leads to learning without a specific purpose.  It may be compared to exploration in reinforcement learning.  internal reward motivates the machine to perform exploration.

38. EE141 How to Motivate a Machine ?  I suggest that it is the hostility of the environment, in the definition of EI that is the most effective motivational factor.  It is the pain we receive that moves us.  It is our intelligence determined to reduce this pain that motivates us to act, learn, and develop.  Both are needed - hostility of the environment and intelligence that learns how to reduce the pain.  Thus pain is good.  Without pain there would be no intelligence.  Without pain we would not be motivated to develop. Fig. englishteachermexico.wordpress.com/

39. EE141 Motivated Learning  I suggest a goal-driven mechanism to motivate a machine to act, learn, and develop.  A simple pain based goal creation system is explained next.  It uses externally defined pain signals that are associated with primitive pains.  Machine is rewarded for minimizing the primitive pain signals.  Definition: Motivated learning (ML) is learning based on the self-organizing system of goal creation in embodied agent.  Machine creates higher level (abstract) goals based on the primitive pain signals.  It receives internal rewards for satisfying its goals (both primitive and abstract).  ML applies to EI working in a hostile environment.

40. EE141 Pain-center and Goal Creation for ML  Simple Mechanism  Creates hierarchy of values  Leads to formulation of complex goals  Pain comparators release reinforcement neurotransmitter: Pain increase - inhibitory Pain decrease - excitatory  Forces exploration + - Environment Sensor Motor Pain level Dual pain level Pain increase Pain decrease (-) (+) Excitation (-) (+)

41. EE141 Pain-center and Goal Creation for ML + - Sensor Motor Pain detection Dual pain memory Pain increase Pain decrease (-) (+) Stimulation (-) (+) activation need Sensory neuron Motor neuron Pain detection/goal creation center Reinforcement neurotransmitter Missing objects inhibition expectation

42. EE141 Abstract Goal Creation for ML  The goal is to reduce the primitive pain level  Abstract goals are created if they satisfy the primitive goals Expectation Association Inhibition Reinforcement Connection Planning -+ PainDual pain Food refrigerator -+ Stomach Abstract pain (Delayed memory of pain) “food”becomes a sensory input to abstract pain center Sensory pathway (perception, sense) Motor pathway (action, reaction) Primitive Level Level I Level II Eat Open

43. EE141 Abstract Goal Hierarchy  Hierarchy of abstract goals is created if they satisfy the primitive goals Activation Stimulation Inhibition Reinforcement Echo Need Expectation -+ + Sugar level Primitive Level Level I Level II Money - Food Spend Eat + Sensory pathway (perception, sense) Motor pathway (action, reaction) Level III Job - Work

44. EE141 The Three Pathways Combined  Goal creation, sensory and motor pathways interact on different hierarchy levels  Pain driven goal creation sets goal priorities Pain tree I Pain tree II Motor pathway Sensory pathway Pain center to motor Sensor to motor Sensor to pain center

45. EE141 EI Interaction with Environment EI machine interacts with environment using its three pathways

46. EE141 Goal Creation Experiment in ML Sensory-motor pairs and their effect on the environment PAIR #SENSORYMOTORINCREASESDECREASES 1FoodEatsugar levelfood supplies 8GroceryBuyfood suppliesmoney at hand 15BankWithdrawmoney at handspending limits 22OfficeWorkspending limitsjob opportunities 29SchoolStudyjob opportunities -

47. EE141 Goal Creation Experiment in ML Goal Creating Neural Network

48. EE141 Goal Creation Experiment in ML Trainable connections between pain, bias, and goal neurons

49. EE141 Goal Creation Experiment in ML Pain signals in CGS simulation

50. EE141 Goal Creation Experiment in ML Action scatters in 5 CGS simulations

51. EE141 Goal Creation Experiment in ML The average pain signals in 100 CGS simulations

52. EE141 Goal Creation Experiment in ML Comparison between GCS and RL

53. EE141 Compare RL (TDF) and ML (GCS) Mean primitive pain P p value as a function of the number of iterations: - green line for TDF -blue line for GCS. Primitive pain ratio with pain threshold 0.1

54. EE141  Comparison of execution time on log-log scale  TD-Falcon green  GCS blue  Combined efficiency of GCS 1000 better than TDF Compare RL (TDF) and ML (GCS) Problem solved Conclusion: embodied intelligence, with motivated learning based on goal creation system, effectively integrates environment modeling and decision making – thus it is poised to cross the chasm

55. EE141 Reinforcement Learning Motivated Learning  Single value function  Measurable rewards  Can be optimized  Predictable  Objectives set by designer  Maximizes the reward  Potentially unstable  Learning effort increases with complexity  Always active  Multiple value functions  One for each goal  Internal rewards  Cannot be optimized  Unpredictable  Sets its own objectives  Solves minimax problem  Always stable  Learns better in complex environment than RL  Acts when needed

56. EE141 How can we make human level intelligence?  We need to know how  We need means to implement it  We need resources to build and sustain its operation

57. EE141 From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006 Resources – Evolution of Electronics

58. EE141 By Gordon E. Moore

59. EE141

60. From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006 Clock Speed (doubles every 2.7 years)

61. EE141 Doubling (or Halving) times  Dynamic RAM Memory “Half Pitch” Feature Size5.4 years  Dynamic RAM Memory (bits per dollar)1.5 years  Average Transistor Price1.6 years  Microprocessor Cost per Transistor Cycle1.1 years  Total Bits Shipped1.1 years  Processor Performance in MIPS1.8 years  Transistors in Intel Microprocessors2.0 years  Microprocessor Clock Speed2.7 years From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006

62. EE141 From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006

63. EE141 From Hans Moravec, Robot, 1999

64. EE141 Software or hardware?  Sequential  Error prone  Require programming  Low cost  Well developed programming methods  Concurrent  Robust  Require design  Significant cost  Hardware prototypes hard to build SoftwareHardware

65. EE141 Future software/hardware capabilities Human brain complexity

66. EE141 Why should we care? Source: SEMATECH

67. EE141 Percent of die area that must be occupied by memory to maintain SOC design productivity Design Productivity Gap  Low-Value Designs? Source = Japanese system-LSI industry

68. EE141 Self-Organizing Learning Arrays SOLAR  Integrated circuits connect transistors into a system -millions of transistors easily assembled -first 50 years of microelectronic revolution  Self-organizing arrays connect processors into a system -millions of processors easily assembled -next 50 years of microelectronic revolution  * Self-organization  * Sparse and local interconnections  * Dynamically reconfigurable  * Online data-driven learning

69. EE141 Promises of embodied intelligence  To society  Advanced use of technology –Robots –Tutors –Intelligent gadgets  Intelligence age follows –Industrial age –Technological age –Information age  Society of minds –Superhuman intelligence –Progress in science –Solution to societies’ ills  To industry  Technological development  New markets  Economical growth ISAC, a Two-Armed Humanoid Robot Vanderbilt University

70. EE141 2002201020202030 Biomimetics and Bio-inspired Systems Impact on Space Transportation, Space Science and Earth Science Mission Complexity Biological Mimicking Embryonics Extremophiles DNA Computing Brain-like computing Self Assembled Array Artificial nanopore high resolution Mars in situ life detector Sensor Web Biological nanopore low resolution Skin and Bone Self healing structure and thermal protection systems Biologically inspired aero-space systems Space Transportation

71. EE141 Sounds like science fiction  If you’re trying to look far ahead, and what you see seems like science fiction, it might be wrong.  But if it doesn’t seem like science fiction, it’s definitely wrong. From presentation by Feresight Institute

72. EE141 Embodied Artificial Intelligence Based on: [1] E. R. Kandel et al. Principles of Neural Science, McGraw-Hill/Appleton & Lange; 4 edition, 2000. [2] F. Inda, R. Pfeifer, L. Steels, Y. Kuniyoshi, “Embodied Artificial Intelligence,” International seminar, Germany, July 2003. [3] R. Chrisley, “Embodied artificial intelligence, ” Artificial Intelligence, vol. 149, pp.131-150, 2003. [4] R. Pfeifer and C. Scheier, Understanding Intelligence, MIT Press, Cambridge, MA, 1999. [5] R. A. Brooks, “Intelligence without reason,” In Proc. IJCAI-91. (1991) 569-595. [6] R. A. Brooks, Flesh and Machines: How Robots Will Change Us, (Pantheon, 2002). [7] R. Kurzweil The Age of Spiritual Machines: When Computers Exceed Human Intelligence, (Penguin, 2000).

73. EE141 Questions?