1. From Exploration to Planning Cornelius Weber and Jochen Triesch Frankfurt Institute for Advanced Studies Goethe University Frankfurt, Germany 18 th International Conference on Artificial Neural Networks 3 d - 6 th September 2008, Prague

2. Reinforcement Learning valueactor units fixed reactive system that always strives for the same goal Trained Weights

3. reinforcement learning does not use the exploration phase to learn a general model of the environment that would allow the agent to plan a route to any goal so let’s do this

4. Learning actor state space randomly move around the state space learn world models: ● associative model ● inverse model ● forward model

5. Learning: Associative Model weights to associate neighbouring states use these to find any possible routes between agent and goal

6. Learning: Inverse Model weights to “postdict” action given state pair use these to identify the action that leads to a desired state Sigma-Pi neuron model

7. Learning: Forward Model weights to predict state given state-action pair use these to predict the next state given the chosen action

8. Planning

22. goal actor units agent

23. Planning

26. Discussion - reinforcement learning... if no access to full state space - previous work... AI-like planners assume links between states - noise... wide “goal hills” will have flat slopes - shortest path... not taken; how to define? - biological plausibility... Sigma-Pi neurons; winner-take-all - to do: embedding... learn state space from sensor input - to do: embedding... let the goal be assigned naturally - to do: embedding... hand-designed planning phases

27. Acknowledgments Collaborators: Jochen Triesch FIAS J-W-Goethe University Frankfurt Stefan Wermter University of Sunderland Mark Elshaw University of Sheffield