Winning by being lazy: Hierarchy, Abstraction and Least- commitment in the new-age planning Subbarao Kambhampati Arizona State University rakaposhi.eas.asu.edu/yochan.html Invited Talk at NIPS-98 Workshop on Reinforcement Learning

Reinforcing “small chunk at a time” behavior --Theory of inverted reinforcement

Objective A quick overview of the ideas of abstraction, hierarchy, reuse and least- commitment in planning –With special emphasis on new-age planners Share some (hopefully portable) lessons...

Overview Planning -- Then and Now Abstraction/hierarchy in planning –Detail Abstraction Least Commitment –Task decomposition –Experiential abstraction (reuse/replay) Lessons

Planning: The problem GStates are modeled in terms of (binary) state-variables (factored rep.) -- Complete initial state, partial goal state GActions are modeled as state transformation functions -- Syntax: ADL language (Pednault) GPlans are sequences of actions Load(o 1 ) In(o 1 ) At(o 1,l 1 ), At(R,l 1 ) At(R,E) Fly() At(R,M), ¬At(R,E)  x In(x)  At(x,M) & ¬At(x, E) Unload(o 1 ) In(o 1 ) ¬In(o 1 ) Earth At(A,E), At(B,E),At(R,E) At(A,M),At(B,M) ¬In(A), ¬In(B) Effects Preconditions Appolo 13

Refinement Planning: The idea Narrowing sets of action sequences to progress towards sets of solutions Partial plans Refinements Remove non-solutions All action sequences All Solutions  P P’ All Sol P P’ All Seq. Refine [AIMAG-97]

Existing Refinement Strategies PSR 0 1:Unload(A)   2:Fly() 3:Unload(A) 0  At(A,M)  ¬At(A,M) 0 1: Unload(A)  2: Load(A) 0  2: Load(B) 0  2: Fly() 0  1: Unload(A) At(A,E) At(B,E) At(R,E) progression Regression 1: Transport(A) 0  At(A,E) At(A,M) 1: Load(A) 0  At(A,E) At(A,M) 2: Fly() 3: Unload(A) HTN Extend Prefix Extend Suffix Add in the middle Decompose State-Space Plan-Space

Search in the space of conjunctive partial plans –Disjunction split into the search space –Solution extraction is trivial Examples: –STRIPS & Prodigy –SNLP & UCPOP –NONLIN & SIPE Search in the space of disjunctive partial plans –Disjunction handled explicitly –Solution extraction is non- trivial CSP/SAT methods Examples: –Graphplan –SATPLAN Conjunctive plannersDisjunctive planners [AIMag-97;IJCAI-97] Then Now

1: Load(A) 2 : Load(B) 0  3 : Fly(R) or Refining disjunctive plans 1: Load(A) 0 1: Load(B) 0 1: Fly(R) 0 2: Load(B) 2: Load(A) 1: Load(A) 0 1: Load(B) 0 2: Unload(A,E) 2: Unload(B,E) 1: Load(A) 0 1: Load(B) 0 2: Fly(R)        Refining conjunctive plans

Detail Abstraction Idea –Abstract some details of the problem or actions. –Solve the abstracted version. –Extend the solution to the detailed version Precondition Abstraction –Work on satisfying important preconditions first Importance judged by: –Length of plans for subgoals [ABSTRIPS, PABLO] –Inter-goal relations [ALPINE] –Distribution-based [HighPoint] –Strong abstractions (with downward refinement property) are rare –Effectiveness is planner-dependent Clashes with other heuristics such as “most constrained first”

Abstracting Resources (Teasing apart Planning and Scheduling) Most planners thrash by addressing planning and scheduling considerations together –Eg. Blocks world, with multiple robot hands Idea: Abstract resources away during planning –Plan assuming infinite resources –Do a post-planning resource allocation phase –Re-plan if needed (with Biplav Srivastava)

Least Commitment (Detail Postponement) Postpone commitments unless forced –Big idea in conjunctive refinement planning Partial-order planners: UCPOP, SNLP –Interacts with precondition abstraction –Becomes a non-issue in disjunctive planning There is very little commitment to begin with Encodings based on partial order planning can actually be worse off [Mali, 98] –Exception: Variablized (“lifted”) representations PSR 0 1:Unload(A)   2:Fly() 3:Unload(A) 0  At(A,M)  ¬At(A,M)

Task Decomposition (HTN) Planning Domain model contains non-primitive actions, and schemas for reducing them Reduction schemas are given by the designer –Can be seen as encoding user-intent Two notions of completeness: –Schema completeness (Partial Hierarchicalization) –Planner completeness

Modeling Action Reduction

Dual views of HTN planning Capturing hierarchical structure of the domain –Motivates top-down planning Start with abstract plans, and reduce them Many technical headaches –Respecting user-intent, maintaining systematicity and minimality [AAAI-98] Phantomization, filters, promiscuity, downward- unlinearizability.. Capturing expert advice about desirable solutions –Motivates bottom-up planning Ensure that each partial plan being considered is “legal” with respect to the reduction schemas Connection to efficiency is not obvious Relative advantages are still unclear... [Barrett, 97]

HTN planning in the new-age The ideas of top-down and bottom-up HTN planning can be ported to disjunctive planners [AIPS-98] –Abstract actions can be seen as disjunctive constraints Add constraints to the SAT/CSP encodings of the planning problem to ensure that: –Abstract actions are related to primitive actions through the reduction schemas [Top-down version] OR –Each primitive actions must be part of some task reduction schema [Bottom-up version] Puzzle: How can increasing encoding sizes lead to efficient planning? –New constraints support simplification of the original constraints [with Amol Mali]

HTNSAT: Some results [with Mali, AIPS-98] 40x speedup

Experiential Abstraction: Macrops, Reuse, Replay Structures being reused –Opaque vs. Modifiable –Solution vs. Solving process (derivation) Acquisition of structures to be reused –Human given vs. Automatically acquired Mechanics of reuse –Phased vs. simultaneous Costs –Storage & Retrieval costs; Solution quality

Case-study: DerSNLP Modifiable derivational traces were reused Traces were automatically acquired during problem solving –Analyze the interactions among the parts of a plan, and store plans for non-interacting subgoals separately Reduces retrieval cost –Use of EBL failure analysis to detect interactions All relevant trace fragments were retrieved and replayed before the control is given to from-scratch planner –Extension failures are traced to individual replayed traces, and their storage indices are modified appropriately Improves retrieval accuracy (with Ihrig, JAIR 97)

DerSNLP: Results Performance with increased Training % Solvability with increased traning (JAIR, 97) Library Size 5 3 1

Reuse in Disjunctive Planning Harder to make a disjunctive planner commit to extending a specific plan first Options: –Support opaque macros along with primitive actions –Modify the problem/domain specification so the old plan’s constraints will be respected in any solution –MAX-SAT formulations of reuse problem [with Amol Mali]

Reachability/Relevance minimizations Reachability analysis –Analyze which actions cannot be executed together and which propositions cannot be made together at particular time steps Graphplan mutual exclusions Domain invariants Relevance analysis –Analyze which actions are relevant and must occur together Greedy Regression (RIFO) –Operator Graphs Inseperability constraints Explicate which parts of a disjunctive structure cannot be part of a solution (focusing)

General Lessons Dual views: Detail reduction vs. Expert advice –Detail reduction => hierarchical solving with promise of improved efficiency –Expert advice implies further constraints on the solutions Strong abstractions are rare –Must take abstractions as advice that can be overridden The interaction between abstraction and search mechanism Emphasis on automatic generation of abstractions –Need to consider utility issues Emphasis on satisficing solutions –Few quantitative guarantees on solution quality