Software Multiagent Systems: Lecture 13 Milind Tambe University of Southern California

Teamwork When agents act together

Understanding Teamwork  Ordinary traffic  Driving in a convoy  Two friends A & B together drive in a convoy  B is secretly following A  Pass play in Soccer  Contracting with a software company  Orchestra

Understanding Teamwork Not just a union of simultaneous coordinated actions Different from contracting Together Joint Goal Co-labor  Collaborate

Why Teamwork? Why not: Master-Slave? Contracts?

Why Teams Robust organizations Responsibility to substitute Mutual assistance Information communicated to peers Still capable of structure (not necessarily flat) Subteams, subsubteams Variations in capabilities and limitations

Approach Theory Practical teamwork architectures

Taking a step back…

Key Approaches in Multiagent Systems Market mechanisms Auctions Distributed Constraint Optimization (DCOP) x1 x2 x3x4 Belief-Desire-Intention (BDI) Logics and Psychology (JPG  p (MB   p) ۸ (MG   p) ۸ (Until [(MB  p) ۷ (MB    p)] (WMG   p)) Distributed POMDP Hybrid DCOP/ POMDP/ AUCTIONS/ BDI Essential in large-scale multiagent teams Synergistic interactions

Key Approaches for Multiagent Teams    Markets BDI Dis POMDPs Local interactions UncertaintyLocal utility Human usability & plan structure DCOP      Markets BDI Dis POMDPs Local interactions UncertaintyLocal utility Human usability & plan structure DCOP BDI-POMDP Hybrid

Distributed POMDPs Three papers on the web pages: What to read: Ignore all the proofs Ignore complexity results JAIR article: the model and the results at the end Understand fundamental principles

Domain: Teamwork for Disaster Response

Multiagent Team Decision Problem (MTDP) MTDP: S: s1, s2, s3… Single global world state, one per epoch A: domain-level actions; A = {A1, A2, A3,…An} Ai is a set of actions for each agent i Joint action

MTDP P: Transition function: P(s’ | s, a1, a2, …an) R A : Reward R(s, a1, a2,…an) One common reward; not separate Central to teamwork

MTDP (cont’d)  : observations Each agent: different finite sets of possible observations  O: probability of observation O(destination-state, joint-action, joint-observation) P(o1,o2..om | a1, a2,…am, s’)

Simple Scenario Cost of action: -0.2 Must fight fires together Observe own location and fire status

MTDP Policy  he problem: Find optimal JOINT policies One policy for each agent  i : Action policy Maps belief state into domain actions (Bi  A) for each agent Belief state: sequence of observations

MTDP Domain Types Collectively partially observable: general case, no assumptions Collectively observable: Team (as a whole) observes state For all joint observations, there is a state s, such that, for all other states s’ not equal to s, Pr (o1,o2…on | s’) = 0 Pr (o1, o2, …on | s ) = ? Pr (s | o1,o2..on) = ? Individually observable: each agent observes the state For all individual observations, there is a state s, such that for all other states s’ not equal to s, Pr (oi | s’) = 0

From MTDP to COM-MTDP Two separate actions: communication vs domain actions Two separate reward types: Communication rewards and domain rewards Total reward: sum two rewards Explicit treatment of communication Analysis

Communicative MTDPs(COM-MTDPs)  : communication capabilities, possible “speech acts” e.g., “I am moving to fire1.” R  : communication cost (over messages) e.g., saying, “I am moving to fire1,” has a cost R   Why ever communicate?

Two Stage Decision Process Agent World Observes Actions SE1 P1 b1 P2 SE2 b2 Communications to and from P1: Communication policy P2: Action policy Two state estimators Two belief State updates

COM-MTDP Continued  Belief state (each Bi history of observations, Communication) Two stage belief update Stage 1: Pre-communication belief state for agent i (updates just from observations)  i 0      i 1      i t-1   t-1   i t   Stage 2: Post-communication belief state for i (updates from observations and communication)  i 0      i 1      i t-1   t-1   i t   t  Cannot create probability distribution over states

COM-MTDP Continued  he problem: Find optimal JOINT policies One policy for each agent   : Communication policy Maps pre-communication belief state into message (Bi   for each agent  A : Action policy Maps post-communication belief state into domain actions (Bi  A) for each agent

More Domain Types General Communication: no assumptions on R  Free communication: R  (s,  ) = 0 No communication: R  (s,  ) is negatively infinite

Teamwork Complexity Results Individual observability Collective observability Collective Partial obser. No communication P-complete NEXP complete NEXP complete General communication P-complete NEXP complete NEXP complete Full communication P-complete PSPACE complete

Classifying Different Models Individual observability Collective observability Collective Partial obser. No communication MMDP DEC-POMDP POIPSG General communication XUAN-LESSER COM-MTDP Full communication

True or False If agents communicated all their observations at each step then the distributed POMDP would be essentially a single agent POMDP In distributed POMDPs, each agent plans its own policy Solving Distributed POMDPs with two agents is of same complexity as solving two separate individual POMDPs

Algorithms

NEXP-complete No known efficient algorithms Brute force search 1. Generate space of possible joint policies 2. For each policy in policy space 3.Evaluate over finite horizon T Complexity: No. of policies Cost of evaluation

Locally optimal search Joint equilibrium based search for policies JESP

Nash Equilibrium in Team Games Nash equilibrium vs Global optimal reward for the team 3,67,1 5,18,2 6,06,2 x y z uv A B x y z uv A B

JESP: Locally Optimal Joint Policy x y z uv A B w Iterate keeping one agent’s policy fixed More complex policies the same way

Joint Equilibrium-based Search Description of algorithm: 1. Repeat until convergence 2.For each agent i 3.Fix policy of all agents apart from i 4.Find policy for i that maximizes joint reward Exhaustive-JESP: brute force search in policy space of agent I Expensive

JESP: Joint Equilibrium Search (Nair et al, IJCAI 03) Repeat until convergence to local equilibrium, for each agent K: Fix policy for all except agent K Find optimal response policy for agent K Optimal response policy for K, given fixed policies for others in MTDP: Transformed to a single-agent POMDP problem: “Extended” state defined as not as Define new transition function Define new observation function Define multiagent belief state Dynamic programming over belief states Fast computation of optimal response

Extended State, Belief State Sample progression of beliefs: HL and HR are observations a2: Listen

Run-time Results Method Exhaustive-JESP DP-JESP

Is JESP guaranteed to find the global optimal? Random restarts

Not All Agents are Equal Scaling up Distributed POMDPs for Agent Networks

Runtime

POMDP vs. distributed POMDP Distributed POMDPs more complex Joint transition and observation functions Better policy Free communication = POMDP Less dependency = lower complexity

BDI vs. distributed POMDP BDI teamworkDistributed POMDP teamwork Explicit joint goalExplicit joint reward Plan/organization hierarchiesUnstructured plans/teams Explicit commitmentsImplicit commitments No costs / uncertaintiesCosts & uncertainties included