1. SARSOP Successive Approximations of the Reachable Space under Optimal Policies Devin Grady 4 April 2013

2. Outline 1.Review – POMDP – Point-based Value Iteration – Heuristic Search [point-based] Value Iteration 2.SARSOP – Add a new heuristic and pruning method to HSVI – Results

3. REVIEW (Thanks Ryan)

4. POMDP Solving a POMDP is very similar to an MDP The similarities: – State transitions are still stochastic – Value function is a function of our current “state” – We still perform Bellman backups to compute V The differences: – We have a probability distribution of where we are – We can make (stochastic) observations of our current belief

5. Let’s solve a POMDP! measurementsaction u 3 state x 2 payoff measurements actions u 1, u 2 payoff state x 1

6. Make a decision… NOW

7. HEY! You said POMDP… Geez. OK. We don’t know that we observed z 1. We must compute expected value.

8. HEY! You said POMDP… Geez. OK. We don’t know that we observed z 1. We must compute expected value.

9. Belief Update

10. Lather, Rinse, Repeat We just did a full backup for T=1! Repeat for T=2.

11. The POMDP Hex POMDPs have both the curse of dimensionality and the curse of history. Scholars maintain that history is the truly unmanageable part. O (|V| x |A| x |Ω| x |S| 2 + |A| x |S| x |V| |Ω| ) Belief Update (taking an action) Value Backups (sensor measurements)

12. Point-based Value Iteration Maintains a fixed set of belief points, B The value function is computed only over B – B only contains reachable beliefs The number of constraints (|V|) is fixed at |B| – Value updates are now PTIME PBVI provides an anytime solution that converges to optimal value function The error in the value function is bounded

13. What does it Mean? Start with a small number of reachable beliefs Point-based backup instead of Bellman backup – Implicitly prunes value simplex Increase # beliefs until timeout O (|V| x |A| x |Ω| x |S| 2 + |B| x |A| x |S| x |Ω|)

14. HSVI Value Function When search ceases, perform full Bellman backups in reverse order Just insert the constraint vector for the l.b. Update u.b. based on expected value Both bounds are uniformly improvable

15. Properties of HSVI Upper and lower bounds monotonically converge to optimal value function Local updates preserve improvability Maximum regret is ε Finite search depth Finite depth → finite Bellman updates

16. SARSOP (New stuff here!)

17. Like HSVI… Except, only look at reachable OPTIMAL belief points instead of all reachable points

18. Algorithm Outline 1.Sample 2.Backup 3.Prune

19. Algorithm Outline

20. “Selective Deep Sampling”

21. Algorithm Outline 1.Sample 2.Backup – See HSVI, standard Bellman backup 3.Prune

22. Algorithm Outline

23. Rock-Sample Deterministic motions Noisy sensor for Rock-goodness +10 for sampling good -10 for sampling bad +10 for exiting No other cost/reward

24. Rock-Sample Deterministic motions Noisy sensor for Rock-goodness +10 for sampling good -10 for sampling bad +10 for exiting No other cost/reward

25. New method… is worse? The robot position in RockSample is fully observed The rock goodness values are also completely independent HSVI search for value function cuts that mask others is particularly suited to this

26. HomeCare

27. SARSOP Overall It’s a heuristic method – There will always be bad corner cases – It works well overall most of the time It has become very popular in the past year for POMDP literature – Because it performs well and is fairly straightforward if you understand HSVI?

28. References Kurniawati, Hanna, David Hsu, and Wee Sun Lee. "SARSOP: Efficient point- based POMDP planning by approximating optimally reachable belief spaces."Proc. Robotics: Science and Systems. Vol. 62. 2008. SARSOP available for download at http://bigbird.comp.nus.edu.sg/pmwiki/farm/appl/ http://bigbird.comp.nus.edu.sg/pmwiki/farm/appl/ G. Shani, J. Pineau and R. Kaplow. A Survey of Point-based POMDP Solvers. Autonomous Agents and Multi-agent Systems. 2012. T. Smith and R. Simmons. Heuristic Search Value Iteration for POMDPs. Uncertainty in Artificial Intelligence. 2004. J. Pineau, G. Gordon and S. Thrum. Point-based Value Iteration: An anytime algorithm for POMDPs. Int’l Joint Conferences in Artificial Intelligence. 2003. S. Thrun, W. Burgard. and D. Fox. Probabilistic Robotics. MIT Press. 2006.