Transfer Learning and Intelligence: an Argument and Approach Matthew E. Taylor Joint work with: Gregory Kuhlmann and Peter Stone Learning Agents Research Group Department of Computer Sciences The University of Texas at Austin

Result Summary: AGI-08 Help select source task for given target Transfer a search heuristic Target Task Size Hand-coded Heuristic? Target Task Random Start State? Target Task Random Goal State? Transfer Percentage 8x8YesNo 75 8x8No 73 10x10No 66 8x8NoYesNo69 8x8NoYesYes (anywhere)92 8x8NoYesYes (SW quadrant)70

AGI & Learning Why Learn? 1.Better solutions 2.On-line adaptation Current Problems: 1.Commonly applied to simple tasks 2.Algorithms often data-inefficient 3.Need substantial amounts of human knowledge One possible answer: Transfer Learning

Transfer Learning (related to Lifelong Learning or Multi-task Learning) Learn across multiple tasks: Learn faster Harder tasks become tractable Learn with less human input Prerequisite for AGI?

Transfer Examples Learn difficult tasks faster – Learn a set of simple tasks – Eventually learn target task – Total time reduction Autonomous transfer – Explore the world, learning – Transfer autonomously – Effectively use past knowledge

Transfer in Reinforcement Learning Environment Agent Action State Reward Source Task Environment Agent Action StateReward Target Task

Representative Transfer Results

Policy: π(s) → a Action-value function: Q(s,a) → R Model of the environment: T(s, a) → s’ Rules / Advice Higher-level information – Search heuristic – Learning rates – Appropriate features What to transfer? Environment Agent Action StateReward

How to transfer? Human design (engineering task) Construct a sequence of tasks Provide learner with mappings between tasks Fully autonomous (not yet achieved) Learn if tasks are related Learn how tasks are related ? ?

Result Summary: AGI-08 Help select source task for given target Transfer a search heuristic General Game Playing task W13: Transfer Learning for Complex Tasks