1. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 1 Chapter 17 Planning Based on Model Checking Dana S. Nau University of Maryland 9:10 PM April 12, 2015 Lecture slides for Automated Planning: Theory and Practice

2. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 2 Motivation Actions with multiple possible outcomes  Action failures »e.g., gripper drops its load  Exogenous events »e.g., road closed Nondeterministic systems are like Markov Decision Processes (MDPs), but without probabilities attached to the outcomes  Useful if accurate probabilities aren’t available, or if probability calculations would introduce inaccuracies a c b grasp(c) a c b Intended outcome ab c Unintended outcome

3. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 3 Nondeterministic Systems Nondeterministic system: a triple  = (S, A,  )  S = finite set of states  A = finite set of actions   : S  A  2 s Like in the previous chapter, the book doesn’t commit to any particular representation  It only deals with the underlying semantics  Draw the state-transition graph explicitly Like in the previous chapter, a policy is a function from states into actions  π: S  A Notation: S π = {s | (s,a)  π}  In some algorithms, we’ll temporarily have nondeterministic policies »Ambiguous: multiple actions for some states  π: S  2 A, or equivalently, π  S  A  We’ll always make these policies deterministic before the algorithm terminates

4. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 4 Goal Start 2 Robot r1 starts at location l1 Objective is to get r1 to location l4 π 1 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)) } π 2 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)), (s5, move(r1,l3,l4)) } π 3 = { (s1, move(r1,l1,l4)) } Example

5. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 5 Goal Start 2 Robot r1 starts at location l1 Objective is to get r1 to location l4 π 1 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)) } π 2 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)), (s5, move(r1,l3,l4)) } π 3 = { (s1, move(r1,l1,l4)) } Example

6. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 6 Goal Start 2 Robot r1 starts at location l1 Objective is to get r1 to location l4 π 1 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)) } π 2 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)), (s5, move(r1,l3,l4)) } π 3 = { (s1, move(r1,l1,l4)) } Example

7. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 7 Goal Start 2 Execution Structures Execution structure for a policy π:  The graph of all of π’s execution paths Notation:  π = (Q,T)  Q  S  T  S  S π 1 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)) } π 2 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)), (s5, move(r1,l3,l4)) } π 3 = { (s1, move(r1,l1,l4)) } s2 s1 s5 s3 s4

8. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 8 Execution Structures Execution structure for a policy π:  The graph of all of π’s execution paths Notation:  π = (Q,T)  Q  S  T  S  S π 1 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)) } π 2 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)), (s5, move(r1,l3,l4)) } π 3 = { (s1, move(r1,l1,l4)) } s2 s1 s5 s3 s4

9. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 9 Goal Start 2 Execution Structures Execution structure for a policy π:  The graph of all of π’s execution paths Notation:  π = (Q,T)  Q  S  T  S  S π 1 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)) } π 2 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)), (s5, move(r1,l3,l4)) } π 3 = { (s1, move(r1,l1,l4)) } s2 s1 s5 s3 s4

10. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 10 Execution Structures Execution structure for a policy π:  The graph of all of π’s execution paths Notation:  π = (Q,T)  Q  S  T  S  S π 1 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)) } π 2 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)), (s5, move(r1,l3,l4)) } π 3 = { (s1, move(r1,l1,l4)) } s2 s1 s5 s3 s4

11. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 11 Goal Start 2 Execution Structures Execution structure for a policy π:  The graph of all of π’s execution paths Notation:  π = (Q,T)  Q  S  T  S  S π 1 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)) } π 2 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)), (s5, move(r1,l3,l4)) } π 3 = { (s1, move(r1,l1,l4)) } s1 s4

12. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 12 Execution Structures Execution structure for a policy π:  The graph of all of π’s execution paths Notation:  π = (Q,T)  Q  S  T  S  S π 1 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)) } π 2 = { (s1, move(r1,l1,l2)), (s2, move(r1,l2,l3)), (s3, move(r1,l3,l4)), (s5, move(r1,l3,l4)) } π 3 = { (s1, move(r1,l1,l4)) } s1 s4

13. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 13 Weak solution: at least one execution path reaches a goal Strong solution: every execution path reaches a goal Strong-cyclic solution: every fair execution path reaches a goal  Don’t stay in a cycle forever if there’s a state-transition out of it s0 s1 s3 Goal a0 a1 a2 s2 a3 s0 s1 s3 Goal a0 a1 a2 s2 s0 s1 s3 Goal a0 a1 a2 s2 Goal Types of Solutions a3

14. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 14 Finding Strong Solutions Backward breadth-first search StrongPreImg (S) = {(s,a) :  (s,a) ≠ ,  (s,a)  S}  all state-action pairs for which all of the successors are in S PruneStates (π,S) = {(s,a)  π : s  S}  S is the set of states we’ve already solved  keep only the state-action pairs for other states MkDet (π')  π' is a policy that may be nondeterministic  remove some state-action pairs if necessary, to get a deterministic policy

15. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 15 π = failure π' =  S π' =  S g  S π' = { s4 } Goal Start s4 2 Example

16. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 16 Start π = failure π' =  S π' =  S g  S π' = { s4 } π''  PreImage = {( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } Goal s5 s3 s4 2 Example

17. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 17 Start π = failure π' =  S π' =  S g  S π' = { s4 } π''  PreImage = {( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } π  π' =  π'  π' U π'' = {( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } Goal s5 s3 s4 2 Example

18. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 18 Example Goal Start π =  π' = {( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } S π' = { s3,s5 } S g  S π' = { s3,s4,s5 } s5 s3 s4 2

19. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 19 π =  π' = {( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } S π' = { s3,s5 } S g  S π' = { s3,s4,s5 } PreImage  { (s2,move(r1,l2,l3)), (s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)), (s3,move(r1,l4,l3)), (s5,move(r1,l4,l5)) } π''  { (s2,move(r1,l2,l3)) } π  π' = { (s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } π'  {(s2,move(r1,l2,l3), (s3,move(r1,l3,l4)), (s5,move(r1,l5,l4))} Goal Start s5 s3 s4 s2 2 Example

20. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 20 π = {( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } π' = { (s2,move(r1,l2,l3)), ( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } S π' = { s2,s3,s5 } S g  S π' = { s2,s3,s4,s5 } Goal Start s2 s5 s3 s4 2 Example

21. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 21 Goal Start s2 s5 s3 s4 s1 2 Example π = {( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } π' = { (s2,move(r1,l2,l3)), ( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } S π' = { s2,s3,s5 } S g  S π' = { s2,s3,s4,s5 } π''  { (s1,move(r1,l1,l2)) } π  π' = { (s2,move(r1,l2,l3)), ( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } π'  { (s1,move(r1,l1,l2)), (s2,move(r1,l2,l3)), (s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) }

22. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 22 π = { (s2,move(r1,l2,l3)), ( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } π' = { (s1,move(r1,l1,l2)), (s2,move(r1,l2,l3)), ( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } S π' = { s1,s2,s3,s5 } S g  S π' = { s1,s2,s3,s4,s5 } Goal Start s2 s5 s3 s4 s1 2 Example

23. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 23 π = { (s2,move(r1,l2,l3)), ( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } π' = { (s1,move(r1,l1,l2)), (s2,move(r1,l2,l3)), ( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } S π' = { s1,s2,s3,s5 } S g  S π' = { s1,s2,s3,s4,s5 } S 0  S g  S π' MkDet (π') = π' Goal Start s2 s1 s5 s3 s4 2 Example

24. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 24 Finding Weak Solutions Weak-Plan is just like Strong-Plan except for this: WeakPreImg (S) = {(s,a) :  (s,a) i S ≠  }  at least one successor is in S Weak

25. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 25 π = failure π' =  S π' =  S g  S π' = { s4 } Example Goal Start s4 2 Weak

26. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 26 Example Start π = failure π' =  S π' =  S g  S π' = { s4 } π'' = PreImage = {( s1,move(r1,l1,l4)), ( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } π  π' =  π'  π' U π'' = {( s1,move(r1,l1,l4)), ( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } Goal s5 s3 s4 2 s1 Weak

27. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 27 Example Goal Start π =  π' = {( s1,move(r1,l1,l4)), ( s3,move(r1,l3,l4)), (s5,move(r1,l5,l4)) } S π' = { s1,s3,s5 } S g  S π' = { s1,s3,s4,s5 } S 0  S g  S π' MkDet (π') = π' s5 s3 s4 2 s1 Weak

28. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 28 Finding Strong-Cyclic Solutions Begin with a “universal policy” π' that contains all state-action pairs Repeatedly, eliminate state-action pairs that take us to bad states  PruneOutgoing removes state-action pairs that go to states not in S g ∪ S π »PruneOutgoing (π,S) = π – {(s,a)  π :  (s,a)  S ∪ S π  PruneUnconnected removes states from which it is impossible to get to S g »Start with π' = , compute fixpoint of π'  π ∩ WeakPreImg (S g ∪ S π’ )

29. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 29 Finding Strong-Cyclic Solutions Once the policy stops changing, If it’s not a solution, return failure RemoveNonProgress removes state-action pairs that don’t go toward the goal  implement as backward search from the goal MkDet makes sure there’s only one action for each state Goal Start s5 s3 s4 s2 2 s1 s6 at(r1,l6)

30. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 30 π   π'  {(s,a) : a is applicable to s} Goal Start s5 s3 s4 s2 2 s1 s6 at(r1,l6) Example 1

31. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 31 π   π'  {(s,a) : a is applicable to s} π  {(s,a) : a is applicable to s} PruneOutgoing(π',S g ) = π' PruneUnconnected(π',S g ) = π' RemoveNonProgress(π') = ? Goal Start s5 s3 s4 s2 2 s1 s6 at(r1,l6) Example 1

32. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 32 π   π'  {(s,a) : a is applicable to s} π  {(s,a) : a is applicable to s} PruneOutgoing(π',S g ) = π' PruneUnconnected(π',S g ) = π' RemoveNonProgress(π') = as shown Start s5 s3 s4 s2 2 s1 s6 Goal at(r1,l6) Example 1

33. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 33 π   π'  {(s,a) : a is applicable to s} π  {(s,a) : a is applicable to s} PruneOutgoing(π',S g ) = π' PruneUnconnected(π',S g ) = π' RemoveNonProgress(π') = as shown MkDet(…) = either {(s1,move(r1,l1,l4), (s2,move(r1,l2,l3)), (s3,move(r1,l3,l4), (s4,move(r1,l4,l6), (s5,move(r1,l5,l4)} or {(s1,move(r1,l1,l2), (s2,move(r1,l2,l3)), (s3,move(r1,l3,l4), (s4,move(r1,l4,l6), (s5,move(r1,l5,l4)} Start s5 s3 s4 s2 2 s1 s6 at(r1,l6) Goal Example 1

34. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 34 π   π'  {(s,a) : a is applicable to s} Goal Start s3 s4 s2 2 s1 s6 at(r1,l6) Example 2: no applicable actions at s5 s5

35. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 35 π   π'  {(s,a) : a is applicable to s} π  {(s,a) : a is applicable to s} PruneOutgoing(π',S g ) = … Goal Start s3 s4 s2 2 s1 s6 at(r1,l6) s5 Example 2: no applicable actions at s5

36. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 36 π   π'  {(s,a) : a is applicable to s} π  {(s,a) : a is applicable to s} PruneOutgoing(π',S g ) = π' Goal Start s3 s4 s2 2 s1 s6 at(r1,l6) s5 Example 2: no applicable actions at s5

37. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 37 π   π'  {(s,a) : a is applicable to s} π  {(s,a) : a is applicable to s} PruneOutgoing(π',S g ) = π' PruneUnconnected(π',S g ) = as shown Goal Start s3 s4 s2 2 s1 s6 at(r1,l6) Example 2: no applicable actions at s5

38. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 38 π   π'  {(s,a) : a is applicable to s} π  {(s,a) : a is applicable to s} PruneOutgoing(π',S g ) = π' PruneUnconnected(π',S g ) = as shown π'  as shown Goal Start s3 s4 s2 2 s1 s6 at(r1,l6) Example 2: no applicable actions at s5

39. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 39 π'  as hown π  π' PruneOutgoing(π',S g ) = π' PruneUnconnected(π',S g ) = π' so π = π' RemoveNonProgress(π') = … Goal Start s3 s4 s2 2 s1 s6 at(r1,l6) Example 2: no applicable actions at s5

40. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 40 π'  shown π  π' PruneOutgoing(π',S g ) = π' PruneUnconnected(π'',S g ) = π' so π = π' RemoveNonProgress(π') = as shown MkDet(shown) = no change Goal Start s3 s4 s2 2 s1 s6 at(r1,l6) Example 2: no applicable actions at s5

41. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 41 Planning for Extended Goals Here, “extended” means temporally extended  Constraints that apply to some sequence of states Examples:  want to move to l3, and then to l5  want to keep going back and forth between l3 and l5 Goal Start move(r1,l2,l1) wait 2

42. Dana Nau: Lecture slides for Automated Planning Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License: http://creativecommons.org/licenses/by-nc-sa/2.0/ 42 Planning for Extended Goals Context: the internal state of the controller Plan: (C, c 0, act, ctxt)  C = a set of execution contexts  c 0 is the initial context  act: S  C  A  ctxt: S  C  S  C Sections 17.3 extends the ideas in Sections 17.1 and 17.2 to deal with extended goals  We’ll skip the details