1 Black Box and Generalized Algorithms for Planning in Uncertain Domains Thesis Proposal, Dept. of Computer Science, Carnegie Mellon University H. Brendan McMahan

2 Outline The Problem and Approach  Motivating Examples  Goals and Techniques  MDPs and Uncertainty Example Algorithms Proposed Future Work

3 Mars Rover Mission Planning Human control not realistic Collect data while conserving power and bandwidth First Experiments in the Robotic Investigation of Life in the Atacama Desert of Chile. D. Wettergreen, et al Recent Progress in Local and Global Traversability for Planetary Rovers. S. Singh, et al

4 Autonomous Helicopter Control 6+ continuous state dimensions Complex, non-linear dynamics High failure cost Inverted Autonomous Helicopter Flight via Reinforcement Learning A. Ng, et al. Autonomous Helicopter Control using Reinforcement Learning Policy Search Methods J. Bagnell and J. Schneider

5 Online Shortest Path Problem Getting from my (old) house to CMU each day:

6 Other Domains

7 Goal Planning multiple decisions over time to achieve goals or minimize cost in Uncertain Domains NOT deterministic, fully observable, perfectly modeled

8 The Black Box Approach Fast Existing Algorithm New Algorithm Hard Planning Problem Easier Problems Solutions Solution

9 The Generalization Approach Hard Planning Problem Solution Generalization of Existing Algorithm Fast Existing Algorithm

10 Two Examples Black Box Approach MDP Alg (e.g., value iteration) Used as a Black Box Oracle Algorithms (MDPs with unknown costs) Generalize To Algorithms for Stochastic Shortest Paths Dijkstra’s Alg (Shortest Paths) Generalization Approach

11 Benefits of using Black Boxes Use fast/optimized/mature implementations Pick implementation for specific domain Will be able to use algorithms not even invented yet Theoretical advantages

12 Benefits of Generalization New intuitions Some performance guarantees for free

13 Markov Decision Processes An MDP (S, A, P, c) … S is a finite set of states A is a finite set of actions dynamics P(y | x, a) costs c(x,a) Goal: New idea! No New Ideas Hungry A = {eat, wait, work} $1.00 $0.10 $4.75 A Research MDP

14 Simple Example Domain Robot path planning problem: Actions = {8 neighbors} Cost: Euclidean Distance Prob. p of random action

15 Types of Uncertainty Outcome Uncertainty (MDPs) Partial Observability (POMDPs) Model Uncertainty (families of MDPs, RL) Modeling Other Agents (Agent Uncertainty?)

16 The Curse of Dimensionality The size of |S| is exponential in the number of state variables: <x,y, vx, vy, battery_power, door_open, another_door_open, goal_x, goal_y, bob_x, bob_y, …>

17 Outline The Problem and Approach Example Algorithms  MDPs with Unknown Costs  Generalizing Dijkstra’s Algorithm Proposed Future Work

18 Unknown Costs, Offline Version A game with two players: The Planner chooses a policy for a MDP with known dynamics The Sentry chooses a cost function from a set K = {c 1,…,c k } of possible cost functions.

19 Avoiding Detection by Sensors The Planner (robot) picks policies (paths): The Sentry picks cost functions (sensor placements):

20 Matrix Game Formulation Matrix game M: Planner (rows) selects a policy  Sentry (columns) selects a cost c M ( , c) = [total cost of  under costs c] Goal: Find a minimax solution to M An optimal mixed strategy for the planner is a distribution over deterministic polices (paths).

21 Interpretations Model Uncertainty: → unknown cost function Partial Observability: → fixed, unobservable cost function Agent Uncertainty: → an adversary picks the cost function

22 How to Solve It Problem: Matrix M is exponentially big Solution: Can be represented compactly as a Linear Program (LP) Problem: LP still takes much too long to solve Solution: The Single Oracle Algorithm, taking advantage of fast black box MDP algorithms

23 Single Oracle Algorithm F is a small set of policies M’ is the matrix game where the Planner must play from F. We can solve M’ efficiently, it is only |F| x |K| in size! |F| = 2

24 Single Oracle Algorithm If only … we knew it was sufficient for the Planner to randomize among a small set of strategies and we could find that set of strategies.

25 Single Oracle Algorithm 1.Use an MDP algorithm to find an optimal policy  against the fixed cost function c. 2.Add  to F 3.Solve M’ and let c be the expected cost function under the Sentry’s optimal mixed strategy.

26 Example Run: Initialization Fix policy (blue path) Solve M’ to find red sensor field (cost vector), fix this as c

27 Iteration 1: Best Response Solve for the best response policy  (new blue line) Add  to F Red: Fixed cost vector (expected field of view) Blue: Shortest path given costs

28 Iteration 1: Solve the Game Solve M’ Minimax Equilibrium: Red:Mixture of Costs Blue:Mixture of Paths from F

29 Iteration 2: Best Response Solve for the best response policy  (new blue line) Add  to F Red: Fixed cost vector (expected field of view) Blue: Shortest path given costs

30 Iteration 2: Solve the Game Solve M’ Minimax Equilibrium: Red:Mixture of Costs Blue:Mixture of Paths from F

31 Iteration 6: Convergence Solution to M’ Best Response

32 Unknown Costs, Online Version Go from my house to CMU each day Model as a graph

33 A Shortest Path Problem? If we knew all the edge costs, it would be easy! But, traffic, downed trees → uncertainty

34 Limited Observations Each day, observe the total length of the path we actually took to get to CMU BGA Algorithm: Keep estimates of edge lengths Most days, follow FPL 1 algorithm: pick shortest path with respect to estimated lengths plus a little noise. Occasionally, play a “random” path in order to make sure we have good estimates of the edge lengths. 1 [Kalai and Vempala, 2003]

35 Dijkstra's Algorithm G x1x1 x2x2 x3x3 x4x4 v'= 0 v'= ∞ v'=3 v'=2 v'=1 v'=5 v'=6 v'=7 v'=2 Keeps states on a priority queue Pops states in order of increasing distance, updates predecessors Prioritized Sweeping 1,2 has a similar structure, but doesn’t reduce to Dijkstra’s algorithm 1 [A. Moore, C. Atkeson 1993] 2 [D. Andre, et al. 1998]

36 Prioritized Sweeping When we pop a state x, backup x, update priorities of predecessors w y1y1 y2y2 y3y3 w1w1 w2w2 x1x1 Values of red states updated based on value of purple states.

37 Improved Prioritized Sweeping When we pop a state x, its value has already been updated Update values and priorities of predecessors w y1y1 y2y2 y3y3 w1w1 w2w2 x1x1 Values of red states updated based on value of purple states.

38 Priority Function Intuitions Update the state: with lowest value (closest to goal) whose value is most accurately known  For Dijkstra’s algorithm, the updated (popped) state’s optimal value is known  This is the state whose value will change the least in the future. whose value has changed the most since it was last updated.

39 Comparison IPS, deterministic domain: PS, same problem: Dark red indicates recently popped from queue, lighter means less recently.

40 Outline The Problem and Approach Example Algorithms Proposed Future Work  Bounded RTDP and extensions  Large action spaces  Details of proposed contributions

41 Bounded RTDP RTDP: Fixed start state means many states are irrelevant Sample, backup along start → goal trajectories BRTDP adds: performance guarantees, much faster convergence (often better than HDP, LRTDP, and LAO*)

42 Dijkstra and BRTDP Dijkstra-style scheduling of backups for BRTDP Sample multiple trajectories Use priority queue to schedule backups of states on all trajectories

43 Dijkstra, BRTDP, and POMDPs HSVI 1 is like BRTDP, but for POMDPs The same trick should apply But more benefit, because backups are more expensive Piecewise linear belief- space value function x1x1 x2x2 1 [T. Smith and R. Simmons ]

44 Large Action Spaces (Prioritized) Policy Iteration already has an advantage Better tradeoff between policy evaluation, policy improvement? Structured sets of actions? Application of Experts/Bandits algorithms?

45 Details: Proposed Contributions Discussion of algorithms already developed: Oracle Algorithms, BGA, IPS, BRTDP, and several others. At least two significant new algorithmic contributions:  BRTDP + Dijkstra algorithm, extension to POMDPs  Improved version of PPI to handle large action spaces  Something else: generalizations of conjugate-gradient linear solvers to MDPs, extensions of the technique for finding upper bounds introduced in the BRTDP paper, algorithms for efficiently solving restricted classes of POMDPs...

46 Details: Proposed Contributions At least one significant new theoretical contribution:  Approximation algorithm for Canadian Traveler’s Problem or Stochastic TSP  Results connecting online algorithms / MDP techniques to stochastic optimization  New contributions on bandit-style online algorithms, perhaps applications to MDPs

47 Summary Motivating Problems Black Boxes: MDPs with unknown Costs Generalization: Reducing to Dijkstra Future Work: BRTDP + Dijkstra, Large action spaces

48 Questions?

49 Relationships of Algorithms Discussed

50 Iteration 3: Best Response Solve for the best response policy  (new blue line) Add  to F Red: Fixed cost vector (expected field of view) Blue: Shortest path given costs

51 Representations, Algorithms Simulation dynamics model Factored Representation (DBNs, etc) STRIPS-style languages Policy Search, … Generalizations of Value Iteration, …