Alborz Geramifard Logic Programming and MDPs for Planning Winter 2009

MDP+ Logic + Programming MDP Index 2 Introduction Logic Programming

Why do we care about planning? 3

Sequential Decision Making 4 ✙ ?? Desired Property? Feasible: Logic (Focuses on Feasibility) Optimal: MDPs (Scaling is Hard)

MDP+ Logic + Programming MDP Index 5 Introduction Logic Programming

Logic 6 Goal Directed Search StartGoal State: Set of propositions (e.g. ¬Quite, Garbage, ¬Dinner) Feasible Plan: Sequence of actions from start to goal

Actions 7 Clean Hands Ingredients Cook Dinner Ready PreconditionActionEffect STRIPS, Graph Plan (16.410/16.413)

8 Graph Plan or Forward Search Not easily scalable GOLOG: ALGOL in LOGIC Restrict action space with programs Scales easier Results are dependent on high-level programs Logic Programming [Levesque, et. al. 97]

Situation Calculus (Temporal Logic) 9 Situation: S0, do(A,S) Example: do(putdown(A), do(walk(L), do(pickup(A), S0))) Relational Fluent: relations with values depending on situations Example: is_carrying(robot, p, s) Functional Fluent: situation dependent functions Example: loc(robot, s) More expressive than LTL

10 GOLOG: Syntax [Scott Sanner - ICAPS08 Tutorial] ?? Aren’t we hard-coding the whole solution?

Blocks World 11 Actions: pickup(b), putOnTable(b), putOn(b1,b2) Fluents: onTable(b, s), on(b1, b2, s) GOLOG Program: while ( ∃ b) ¬onTable(b) then (pickup(b), putOnTable(b)) endWhile ?? What does it do?

S4S4 Given Start S0, Goal S’, and program δ Call Do(δ,S0,S’) First-order logic proof system Prolog S0S0 S1S1 S2S2S3S3 GOLOG Execution 12 Non-deterministic

MDP+ Logic + Programming MDP Index 13 Introduction Logic Programming

MDP 14 B.S., Working, +60, B.S. Studying, -50, M.S.,...

MDP 15 Policy π(s): How to act in each state Value Function V(s): How good is it to be in each state?

MDP 16 Goal Find the optimal policy (π*), which maximizes the value function for all states

MDP 17 ?? Some Algorithms Value Iteration (Dynamic Programing) Policy Iteration

Example (Value Iteration) Reward: Actions:

MDP+ Logic + Programming MDP Index 19 Introduction Logic Programming

DT-GOLOG: Decision Theoretic GOLOG 20 MDP Scaling is challenging GOLOG No Optimization All non-deterministic actions are optimized using MDP concepts. Uncertainty, Reward [Boutilier, et. al 00] S4S4S4S4 S4S4S4S4 S0S0S0S0 S0S0S0S0 S1S1S1S1 S1S1S1S1 S2S2S2S2 S2S2S2S2 S3S3S3S3 S3S3S3S3

Decision Theoretic GOLOG 21 Programming(GOLOG) Planning(MDP) Not enough grasping Low-level Structure Known Solution High-level Structure

Example 22 Build a tower with least effort Use which block for base? In which order pick up the blocks Pick a block as base Stack all other blocks on top of it GOLOG MDP ?? Guaranteed to be optimal?

S4S4S0S0S1S1 S2S2S3S3 DT-GOLOG Execution 23 Given Start S0, Goal S’, and program δ Add rewards Formulate the problem as an MDP ( ∃ b) ¬onTable(b), b ∈ {b1,..., bn} ¬onTable(b1) ∨ ¬onTable(b2) ∨... ∨ ¬onTable(bn)

24 First Order Dynamic Programming Idea: Logical Structure Abstract Value Function Avoid curse of dimensionality! [Sanner 07] Resulting MDP can still be intractable.

25 Symbolic Dynamic Programming (Deterministic) Tabular: Symbolic ? Representation of Reward and Values Adding rewards and values Max Operator Find S

26 Reward and Value Representation Case Representation ∃ b, b≠b1, on(b,b1) 10 ∄ b, b≠b1, on(b,b1) 0 b1

27 Stochastic Actions b=b10.8 b≠b10.9 P(MoveS(b) | Move(b)) = b=b10.2 b≠b10.1 P(MoveF(b) | Move(b)) = b1 Nature’s Choice

28 Add Symbolically Similarly defined for and [Scott Sanner - ICAPS08 Tutorial] ⊕ A10 ¬A20 B1 ¬B2 A∧BA∧B 11 A ∧ ¬B 12 ¬A ∧ B 21 ¬A ∧ ¬B 22 =

29 Max operator max Operator Φ110 Φ25 Φ33 Φ40 max = Φ110 ¬Φ1 ∧ Φ2 5 ¬Φ1 ∧ ¬Φ2 ∧ Φ3 3 ¬Φ1 ∧ ¬Φ2 ∧ ¬Φ3 ∧ Φ4 0 [Scott Sanner - ICAPS08 Tutorial]

30 Max operator max Operator Φ110 Φ25 Φ33 Φ40 max_a = Φ110 ¬Φ1 ∧ Φ2 5 ¬Φ1 ∧ ¬Φ2 ∧ Φ3 3 ¬Φ1 ∧ ¬Φ2 ∧ ¬Φ3 ∧ Φ4 0 a_1 a_2 [Scott Sanner - ICAPS08 Tutorial]

31 Find s? Isn’t it obvious? sss’s’ a Dynamic Programming: Given V(s’) find V(s) In MDPs, we have s explicitly. In symbolic representation we have it implicitly so we have to build it.

32 Find s = Goal Regression b1 B B regress( Φ1,a) Weakest relation that ensures Φ1 after taking a b1 A A Clear(b1) ∧ B≠b1 On(A,b1) Φ1=clear(b1) put(A,B) ∨ ?

33 Symbolic Dynamic Programming (Deterministic) Tabular: Symbolic: ?

34 Classical Example BoxTruckCity Goal: Have a box in Paris ∃ b, BoxIn(b,Paris) 10 else0

35 Classical Example Actions: drive(t,c1,c2), load(b,t), unload(b,t), noop load and unload have 10% chance of failure Fluents: BoxIn(b,c), BoxOn(b,t), TruckIn(t,c) Assumptions: All cities are connected. ϒ =.9

36 Example [Sanner 07] ∃ b, BoxIn(b,Paris) noop else, ∃ b,t TruckIn(t,Paris) ∧ BoxOn(b,t) unload(b,t) else, ∃ b,c,t BoxOn(b,t) ∧ TruckIn(t,c) drive(t,c,paris) else, ∃ b,c,t BoxIn(b,c) ∧ TruckIn(t,c) 72load(b,t) else, ∃ b,c1,c2,t BoxIn(b,c1) ∧ TruckIn(t,c2) drive(t,c2,c1) else0noop ?? What did we gain by going through all of this? sV*(s)π*(s)

Conclusion 37 Logic Programming Situation Calculus GOLOG Planning MDP Review Value Iteration MDP+ Logic + Programming DT-GOLOG Symbolic DP

References 38 Levesque, H.,Reiter, R., Lespérance, Y., Lin, F., and Scherl, R. “ GOLOG: A Logic Programming Language for Dynamic Domains “, Journal of Logic Programming, 31:59--84, 1997 Richard S. Sutton, Andrew G. Barto, “Reinforcement Learning: An Introduction”, MIT Press, Cambridge, 1998 Craig Boutilier, Raymond Reiter, Mikhail Soutchanski, Sebastian Thrun, “Decision-Theoretic, High-Level Agent Programming in the Situation Calculus”. AAAI/IAAI 2000: S. Sanner, and K. Kersting, ”Symbolic dynamic programming”. Chapter to appear in C. Sammut, editor, Encyclopedia of Machine Learning, Springer-Verlag, 2007

References 39 Craig Boutilier, Ray Reiter and Bob Price, “Symbolic Dynamic Programming for First-order MDPs”, Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence (IJCAI), Seattle, pp (2001).

40 Questions ?