1. Kernelized Value Function Approximation for Reinforcement Learning Gavin Taylor and Ronald Parr Duke University

2. Overview Solve for value function given kernelized model Solve for model as in GPRL Kernelized Model Kernel: k(s,s’) Training Data: (s,r,s’),(s,r,s’) (s,r,s’)… Solve for value directly using KLSTD or GPTD V=Kw Kernelized Value Function

3. Overview - Contributions Construct new model-based VFA Equate novel VFA with previous work Decompose Bellman Error into reward and transition error Use decomposition to understand VFA reward error transition error Bellman Error SamplesModelVFA

4. Outline Motivation, Notation, and Framework Kernel-Based Models –Model-Based VFA –Interpretation of Previous Work Bellman Error Decomposition Experimental Results and Conclusions

5. Markov Reward Processes M =( S, P, R,  ) Value: V(s) =expected, discounted sum of rewards from state s Bellman equation: Bellman equation in matrix notation:

6. Kernels Properties: –Symmetric function between two points: –PSD K -matrix Uses: –Dot-product in high-dimensional space (kernel trick) –Gain expressiveness Risks: –Overfitting –High computational cost

7. Outline Motivation, Notation, and Framework Kernel-Based Models –Model-Based VFA –Interpretation of Previous Work Bellman Error Decomposition Experimental Results and Conclusions

8. Kernelized Regression Apply kernel trick to least-squares regression t : target values K : kernel matrix, where k(x) : column vector, where : regularization matrix

9. Kernel-Based Models Approximate reward model Approximate transition model –Want to predict k(s’) (not s’ ) –Construct matrix K’, where SamplesModelVFA

10. Model-based Value Function SamplesModelVFA

11. Model-based Value Function SamplesModelVFA Unregularized: Regularized: Whole state space:

12. Previous Work Kernel Least-Squares Temporal Difference Learning (KLSTD) [Xu et. al., 2005] –Rederive LSTD, replacing dot products with kernels –No regularization Gaussian Process Temporal Difference Learning (GPTD) [Engel, et al., 2005] –Model value directly with a GP Gaussian Processes in Reinforcement Learning (GPRL) [Rasmussen and Kuss, 2004] –Model transitions and value with GPs –Deterministic reward SamplesModelVFA

13. Equivalency MethodValue FunctionModel-based Equivalent KLSTD GPTD GPRL Model- based [T&P `09] : GPTD noise parameter : GPRL regularization parameter SamplesModelVFA

14. Outline Motivation, Notation, and Framework Kernel-Based Models –Model-Based VFA –Interpretation of Previous Work Bellman Error Decomposition Experimental Results and Conclusions

15. Model Error Error in reward approximation: Error in transition approximation: : expected next kernel values : approximate next kernel values

16. Bellman Error reward error transition error Bellman Error a linear combination of reward and transition errors

17. Outline Motivation, Notation, and Framework Kernel-Based Models –Model-Based VFA –Interpretation of Previous Work Bellman Error Decomposition Experimental Results and Conclusions

18. Experiments Version of two room problem [Mahadevan & Maggioni, 2006] Use Bellman Error decomposition to tune regularization parameters REWARD

19. Experiments

20. Conclusion Novel, model-based view of kernelized RL built around kernel regression Previous work differs from model-based view only in approach to regularization Bellman Error can be decomposed into transition and reward error Transition and reward error can be used to tune parameters

21. Thank you!

22. What about policy improvement? Wrap policy iteration around kernelized VFA –Example: KLSPI –Bellman error decomposition will be policy dependent –Choice of regularization parameters may be policy dependent Our results do not apply to SARSA variants of kernelized RL, e.g., GPSARSA

23. What’s left? Kernel selection –Kernel selection (not just parameter tuning) –Varying kernel parameters across states –Combining kernels (See Kolter & Ng ‘09) Computation costs in large problems –K is O(#samples) –Inverting K is expensive –Role of sparsification, interaction w/regularization

24. Comparing model-based approaches Transition model –GPRL: models s’ as a GP –T&P: approximates k(s’) given k(s) Reward model –GPRL: deterministic reward –T&P: reward approximated with regularized, kernelized regression

25. Don’t you have to know the model? For our experiments & graphs: Reward, transition errors calculated with true R, K’ In practice: Cross-validation could be used to tune parameters to minimize reward and transition errors

26. Why is the GPTD regularization term asymmetric? GPTD is equivalent to T&P when Can be viewed as propagating the regularizer through the transition model – –Is this a good idea? –Our contribution: Tools to evaluate this question

27. What about Variances? Variances can play an important role in Bayesian interpretations of kernelized RL –Can guide exploration –Can ground regularization parameters Our analysis focuses on the mean Variances a valid topic for future work

28. Does this apply to the recent work of Farahmand et al.? Not directly All methods assume ( s,r,s’ ) data Farahmand et al. include next states ( s’’ ) in their kernel, i.e., k(s’’,s) and k(s’’,s’) Previous work, and ours, includes only s’ in the kernel: k(s’,s)

29. How is This Different from Parr et al. ICML 2008? Parr et al. considers linear fixed point solutions, not kernelized methods Equivalence between linear fixed point methods was fairly well understood already Our contribution: –We provide a unifying view of previous kernel-based methods –We extend the equivalence between model-based and direct methods to the kernelized case