1. EE141 How to Motivate Machines to Learn and Help Humans in Making Water Decisions? Janusz Starzyk School of Electrical Engineering and Computer Science, Ohio University, USA www.ent.ohiou.edu/~starzyk

2. EE141  Embodied Intelligence (EI)  Embodiment of Mind  EI Interaction with Environment  How to Motivate a Machine  Goal Creation Hierarchy  Goal Creation Experiment  Promises of EI  To economy  To society Outline

3. EE141  “…Perhaps the last frontier of science – its ultimate challenge- is to understand the biological basis of consciousness and the mental process by which we perceive, act, learn and remember..” from Principles of Neural Science by E. R. Kandel et al.  E. R. Kandel won Nobel Price in 2000 for his work on physiological basis of memory storage in neurons.  “… The question of intelligence is the last great terrestrial frontier of science...” from Jeff Hawkins On Intelligence.  Jeff Hawkins founded the Redwood Neuroscience Institute devoted to brain research Intelligence AI’s holy grail From Pattie Maes MIT Media Lab

4. EE141 Traditional AI Embodied Intelligence  Abstract intelligence  attempt to simulate “highest” human faculties: –language, discursive reason, mathematics, abstract problem solving  Environment model  Condition for problem solving in abstract way  “brain in a vat”  Embodiment  knowledge is implicit in the fact that we have a body –embodiment is a foundation for brain development  Intelligence develops through interaction with environment  Situated in a specific environment  Environment is its best model

5. EE141 Design principles of intelligent systems from Rolf Pfeifer “Understanding of Intelligence”, 1999  Interaction with complex environment  cheap design  ecological balance  redundancy principle  parallel, loosely coupled processes  asynchronous  sensory-motor coordination  value principle Agent Drawing by Ciarán O’Leary- Dublin Institute of Technology

6. EE141 Embodied Intelligence Definition  Embodied Intelligence (EI) is a mechanism that learns how to survive in a hostile environment –Mechanism: biological, mechanical or virtual agent with embodied sensors and actuators –EI acts on environment and perceives its actions –Environment hostility is persistent and stimulates EI to act –Hostility: direct aggression, pain, scarce resources, etc –EI learns so it must have associative self-organizing memory –Knowledge is acquired by EI

7. EE141 Embodiment of a Mind  Embodiment contains intelligence core and sensory motor interfaces under its control to interact with environment  Necessary for development of intelligence  Not necessarily constant or in the form of a physical body  Boundary transforms modifying brain’s self- determination

8. EE141  Brain learns own body’s dynamic  Self-awareness is a result of identification with own embodiment  Embodiment can be extended by using tools and machines  Successful operation is a function of correct perception of environment and own embodiment Embodiment of a Mind

9. EE141 INPUTOUTPUT Simulation or Real-World System Task Environment Agent Architecture Long-term Memory Short-term Memory Reason Act Perceive RETRIEVALLEARNING EI Interaction with Environment From Randolph M. Jones, P : www.soartech.com

10. EE141 How to Motivate a Machine ? The fundamental question is how to motivate a machine to do anything, in particular to increase its “brain” complexity? How to motivate it to explore the environment and learn how to effectively work in this environment? Can a machine that only implements externally given goals be intelligent? If not how these goals can be created?

11. EE141  I suggest that hostility of environment motivates us.  It is the pain that moves us.  Our intelligence that tries to minimize this pain motivates our actions, learning and development  We need both the environment hostility and the mechanism that learns how to reduce inflicted by the environment pain How to Motivate a Machine ?  I propose based on the pain mechanism that motivates the machine to act, learn and develop.  So the pain is good.  Without the pain there will be no intelligence.  Without the pain there will be no motivation to develop.

12. EE141 Pain-center and Goal Creation  Simple Mechanism  Creates hierarchy of values  Leads to formulation of complex goals  Reinforcement : Pain increase Pain decrease  Forces exploration + - Environment Sensor Motor Pain level Dual pain level Pain increase Pain decrease (-) (+) Excitation (-) (+) Wall-E’s goal is to keep his plants from dying

13. EE141 Primitive Goal Creation -+ Pain Dry soil Primitive level open tank sit on garbage refill faucet w. can water Dual pain

14. EE141 Abstract Goal Creation  The goal is to reduce the primitive pain level  Abstract goals are created to reduce abstract pains in order to satisfy the primitive goals  Abstract pain center -+ Pain Dual pain + Dry soil Abstract pain “water can” – sensory input to abstract pain center Sensory pathway (perception, sense) Motor pathway (action, reaction) Primitive Level Level I Level II faucet - w. can open water Activation Stimulation Inhibition Reinforcement Echo Need Expectation

15. EE141 Abstract Goal Hierarchy  A hierarchy of abstract goals is created - they satisfy the lower level goals Activation Stimulation Inhibition Reinforcement Echo Need Expectation -+ + Dry soil Primitive Level Level I Level II faucet - w. can open water + Sensory pathway (perception, sense) Motor pathway (action, reaction) Level III tank - refill

16. EE141 GCS vs. Reinforcement Learning Actor-critic design Goal creation system Case study: “How can Wall-E water his plants if the water resources are limited and hard to find?” Sensory pathway Motor pathway GCS Environment Pain States Gate control Desired action &state Action decision Action

17. EE141 Goal Creation Experiment Sensory-motor pairs and their effect on the environment PAIR # SENSORYMOTORINCREASESDECREASES 1water canwater the plantmoisturewater in can 8faucetopenwater in canwater in tank 15tankrefillwater in tankreservoir water 22pipeopenreservoir waterlake water 29rainfalllake water-

18. EE141 Results from GCS scheme 0100200300400500 600 0 2 4 pain Dry soil 0100200300400500600 0 1 2 pain No water in can 0100200300400500600 0 1 2 pain No water in tank 0100200300400500 600 0 0.5 1 pain No water in reservoir 0100200300400500 600 0 2 4 pain No water in lake

19. EE141 Averaged performance over 10 trials: GCS: RL: Machine using GCS learns to control all abstract pains and maintains the primitive pain signal on a low level in demanding environment conditions. GCS vs. Reinforcement Learning

20. EE141 Goal Creation Experiment Action scatters in 5 CGS simulations

21. EE141 Goal Creation Experiment The average pain signals in 100 CGS simulations 0100200300400500600 0 0.5 Primitive pain – dry soil Pain 0100200300400500600 0 0.1 0.2 Lack of water in can Pain 0100200300400500600 0 0.1 0.2 Lack of water in tank Pain 0100200300400500600 0 0.1 0.2 Lack of water in reservoir Pain 0100200300400500600 0 0.05 0.1 Lack of water in lake Pain Discrete time

22. EE141 Promises of embodied intelligence  To society  Advanced use of technology –Robots –Tutors –Intelligent gadgets  Intelligence age follows –Industrial age –Technological age –Information age  Society of minds –Superhuman intelligence –Progress in science –Solution to societies’ ills  To industry  Technological development  New markets  Economical growth ISAC, a Two-Armed Humanoid Robot Vanderbilt University

23. EE141 2002201020202030 Biomimetics and Bio-inspired Systems Impact on Space Transportation, Space Science and Earth Science Mission Complexity Biological Mimicking Embryonics Extremophiles DNA Computing Brain-like computing Self Assembled Array Artificial nanopore high resolution Mars in situ life detector Sensor Web Biological nanopore low resolution Skin and Bone Self healing structure and thermal protection systems Biologically inspired aero-space systems Space Transportation Memristors

24. EE141 Sounds like science fiction  If you’re trying to look far ahead, and what you see seems like science fiction, it might be wrong.  But if it doesn’t seem like science fiction, it’s definitely wrong. From presentation by Feresight Institute

25. EE141Questions?