Pat Langley School of Computing and Informatics Arizona State University Tempe, Arizona Institute for the Study of Learning and Expertise Palo Alto, California Robust Reasoning and Learning About Goal-Directed Activities Thanks to T. Konik, D. Choi, U. Kutur, and D. Nau for their contributions. This talk reports work funded by grants from DARPA, which is not responsible for its contents.

Abductive Plan Understanding We can state the task of abductive plan understanding as: Given: A set of generalized conditional hierarchical plans; Given: A partial sequence of observed actions or events; Find: An explanation of these events in terms of other agents goals and intentions. We can also state a related task that involves plan learning: Given: A set of primitive action models (plan operators); Given: A partial sequence of action/event sequences with associated goals; Find: A set of generalized conditional hierarchical plans that explain these and future behaviors.

Learning Plan Knowledge from Demonstration Plan Knowledge If Impasse Problem ? Initial State goal LIGHT Demonstration Traces Background knowledge Reactive Executor Learned plan knowledge Concept definitions Action model States and actions HTNs Expert

Primitive Concept assigned-mission (?patient ?mission) Nonprimitive Concept patient-form-filled (?patient) Inputs to LIGHT: Conceptual Knowledge Conceptual kowledge is cast as Horn clauses that specify relevant relations in the environment – Hierarchically organized in memory – Divided into primitive and non-primitive predicates

Inputs to LIGHT: Action Models Effects Concept arrival-time(?patient) Precondition Concept patient(?p) and travel-from(?p ?from) and travel-to(?p ?to) Action get-arrival-time (?patient ?from ?to) Operators describe low-level actions that agents can execute directly in the environment – Preconditions: legal conditions for action execution – Effects: expected changes when action is executed

Inputs to LIGHT: Expert Traces and Goals Expert demonstration traces – actions the expert takes and the resulting belief state State: set of concept instances Goal is a concept instance in the final state – LIGHT learns generalized skills that achieves similar goals Action instance get-arrival-time(P2) Concept instance assigned-flight (P1 M1) State Goal Concept all-patients-arranged

Outputs of LIGHT: HTN Methods Methods decompose goals into subgoals – if you have a goal and its precondition is satisfied, then apply its submethods or its operators Similar to regular HTNs but methods indexed by goals achieved precondition concept operator HTN method HTN goal concept HTN method subgoal

Learning HTNs by Trace Analysis concepts actions

Action Chaining Learning HTNs by Trace Analysis

Concept Chaining concepts actions Learning HTNs by Trace Analysis

Explanation Structure dest-airport patient1 SFO arrival-time NW32, 1pm query-arrival-time Scheduled NW32 location patient1 SFO 1pm assigned patient1 NW32 Flight-available assign patient1, NW32 transfer-hospital patient1, hospital2 arrange-ground-transportation SFO, hospital2, 1pm close-airport hospital2, SFO Time:1Time:2 Time:3

Hierarchical Task Network dest-airport ?patient ?loc arrival-time ?flight ?time query-arrival-time Scheduled ?flight location ?patient ?loc ?time assigned ?patient ?flight Flight-available assign ?patient ?flight transfer-hospital ?patient ?hospital arrange-ground-transportation ?loc ?hospital ?time close-airport ?hospital ?loc

Adapting HTNs to Plan Understanding HTNs and methods for learning them (like LIGHT) are typically designed for generating and executing plans. To adapt HTNs to plan understanding, we must revise the framework to support abductive inference when: actions and events are only partially observed; some goals and plans are more likely than others; observations of others behaviors are inherently noisy. These characteristics require extensions to our representation, performance methods, and learning mechanisms.

Markov Task Networks To this end, we have designed a new representatonal formalism for plan knowledge – Markov task networks – that include: A set of goal-indexed HTN methods, each with; a prior probability,, on the method s goal a conditional probability,, for its precondition a conditional probability, for each subgoal A set of Horn clauses, each with: a prior probability,, of the clause s head a conditional probability,, for each condition This framework appears better suited to abductive inference about goal-directed behavior than Markov logic.

Markov Task Networks A Markov task network is a goal-indexed HTN with probabilities for: goals the agent may aim to achieve subgoals he may pursue when using a given method preconditions that suggest he is using the method constraints among the subgoal orders It also includes probabilistic information about relevant relational concepts. P(Precondition| Method) P(Goal)P(Subgoal| Method) P(Goal) P(Precondition| Method) P(Subgoal| Method)

Inference Over Markov Task Networks We can estimate the posterior probability of each goal in a Markov task network given a sequence of observed states by computing: Where when is a primitive relation that occurs in. To obtain actual probabilities, we normalize using the expressions: This is a variant on cascaded Bayesian classifiers (Provan et al., 1996). )

Learning in Markov Task Networks Like other probabilistic frameworks, Markov task networks require two forms of learning: Parameter estimation occurs either: by simple counting, as in naïve Bayes, in the fully supervised case where all goals/subgoals are given by expectation maximization in the partly supervised case where only the top-level goal is provided Structure learning occurs as in LIGHT, except that: Explanation takes advantage of methods learned earlier This process finds the most probable account of events Both forms of learning should be efficient computationally and require few training cases.

Missing concepts Missing actions Learning Markov Task Networks by Trace Analysis Trace analysis proceeds as before, but guided by probabilistic inference that allows for: Missing conceptual relations in states Missing actions that connect states When an existing method is used to explain a trace, probabilities are updated accordingly.

Plans for Future Research To evaluate the framework of Markov task networks, we must: Implement the performance and learning algorithms Design tasks in realistic simulators like OneSAF and MadRTS Use these simulators to generate sequences of observed states Provide background knowledge about these domains Measure accuracy of goal inference given handcrafted task networks Measure ability of learned task networks to produce similar results Experimental results of this sort will suggest ways to improve our formulation and its methods for inference and learning.

Related Work on Abduction and Learning Our approach incorporates ideas from a number of traditions: Hierarchical task networks (Nau et al., 1999; Choi & Langley, 2005) Logical methods for abductive inference (Ng & Mooney, 1990) Relational Bayesian classifiers (Flach & Lachiche, 1999) Cascaded Bayesian classifiers (Provan, Langley, & Binford, 1996) Explanation-based learning from expert traces (Segre, 1987) Statistical relational learning (Muggleton, 1996; Domingos, 2004) However, it adapts and combines them in ways appropriate to the task of abductive plan understanding and learning.

End of Presentation

Hierachical Concepts (in-rightmost-lane ?self ?clane) :percepts (self ?self) (segment ?seg) (line ?clane segment ?seg) :relations (driving-well-in-segment ?self ?seg ?clane) (last-lane ?clane) (not (lane-to-right ?clane ?anylane)) (driving-well-in-segment ?self ?seg ?lane) :percepts (self ?self) (segment ?seg) (line ?lane segment ?seg) :relations (in-segment ?self ?seg) (in-lane ?self ?lane) (aligned-with-lane-in-segment ?self ?seg ?lane) (centered-in-lane ?self ?seg ?lane) (steering-wheel-straight ?self) (in-lane ?self ?lane) :percepts (self ?self segment ?seg) (line ?lane segment ?seg dist ?dist)) :tests (> ?dist -10) (<= ?dist 0) Primitive Concepts Nonprimitive Concepts

(in-rightmost-lane ?self ?line) :percepts (self ?self) (line ?line) :start (last-lane ?line) :subgoals (driving-well-in-segment ?self ?seg ?line) (driving-well-in-segment ?self ?seg ?line) :percepts (segment ?seg) (line ?line) (self ?self) :start (steering-wheel-straight ?self) :subgoals (in-segment ?self ?seg) (centered-in-lane ?self ?seg ?line) (aligned-with-lane-in-segment ?self ?seg ?line) (steering-wheel-straight ?self) (in-segment ?self ?endsg) :percepts (self ?self speed ?speed) (intersection ?int cross ?cross) (segment ?endsg street ?cross angle ?angle) :start (in-intersection-for-right-turn ?self ?int) :actions( steer 1) Hierarchical Methods Primitive Skill Nonprimitive Skill