We Have Not Yet Begun to Learn Rich Sutton AT&T Labs

We Have Not Yet Begun to Learn None of our ML and RL systems learn anything like the things that people animals know and use everyday

Intelligence can be defined as using knowledge flexibly to achieve goals/purposes A working definition that matches our intuitions Sufficiently sufficient

AI systems Chess-playing programs Planners Heuristic search Dynamic programming SOAR Calendar agents Reactive robots Navigating robots Theorem Provers CYC Quality of knowledge  amount  scope  relevance/utility  accuracy Flexibility with which knowledge is used Kinds of knowledge  policy  transition  implication  value Knowledge

So What Holds AI Back? For the problems of interest, it is hard to get the knowledge right  people must manually tune it to make it right  and often it’s not quite clear what “right” means –may be “right” mainly because of what it causes to happen The complex web of knowledge becomes unwieldy  brittle  difficult to change  unreliable More responsibility needs to be given to the machine To autonomously maintain and verify its knowledge

Reliable Knowledge Requires Verification To know something reliably, robustly, you have to be able to tell, by yourself, whether it is correct  policies do they get reward?  predictionsdo they come true?  theoremsdo they have valid proofs?  plansdo they achieve their goals?

Conclusion: Reliable Knowledge Requires Verification We can distinguish 1. Having knowledge 2. Having the ability to verify knowledge I.e., there is something beyond having knowledge which we might call understanding its meaning and which is key in practice to building powerful AIs

Let’s Focus on Transition Knowledge Projective/predictive knowledge of what follows what  Strips Operators  Action models  Physics, dynamics, causation The key kind of knowledge in planners/search systems  chess players, state-based planning, SOAR A paradigm case in knowledge that can be verified and learned from experience

Verifying Transition Knowledge Must have experience Knowledge must be expressed in term of experience Verification must be in terms of experience “Dyna” and successors did this for 1-step transitions But 1-step predictions are not expressive enough Need predictions of arbitrary experiments  a closed-loop policy  with closed-loop, temporally-flexible termination Need something like option models + options

Anatomy of a Super-Prediction 1 Predictor (option model) Recognizes the conditions, makes the prediction 2 Experiment (option) - policy - termination condition - measurement function(s) knowledge verifier

“Reliable Knowledge Requires Verification” is an Example of Purposive Design “Purposive” = control by consequences  Fixed ends achieved by variable means Widely seen as the hallmark of mind  “the mark and criterion of mentality” William James, 1890  “Purposive Behavior in Animals and Man” Edward Tolman, 1932 Achieving systems vs Purposive systems achieve the fixed end achieve the fixed end more often, by varying means. requires ability to verify/recognize the fixed end

There is a tension between achievement and purposive design It’s always easier to build than to meta-build  easier to write a policy than a policy-learner  easier to plan than to write a planner  easier to add knowledge than to add its verifier Even at the policy level, many advocate direct building  reactive systems  simple expert systems At the knowledge level, it is easier to rely on human interpretation than to write explicit verifiers  can talk at human level - objects, times, properties, space  rather than experience level - actions, observations, rewards

Conclusion: Reliable Knowledge Requires Verification We can distinguish 1. Having knowledge 2. Having the ability to verify knowledge I.e., there is something beyond having knowledge which we might call understanding its meaning and which is key in practice to building powerful AIs