1. Yifeng Zeng Aalborg University Denmark
Twenty Second Conference on Artificial Intelligence
(AAAI’07)
Approximate Solutions of Interactive Dynamic Influence Diagrams Using Model Clustering
Yifeng Zeng
Aalborg University
Denmark
Prashant Doshi
Univ. of Georgia
USA
Qiongyu Chen
National University of Singapore

2. Outline Interactive Dynamic Influence Diagrams (I-DIDs)
Curses of History and Dimensionality
Model Clustering
Computational Savings and Error Bound
Experimental Results

3. Interactive Dynamic Influence Diagrams (I-DIDs). (Doshi et al
Interactive Dynamic Influence Diagrams (I-DIDs) (Doshi et al. AAMAS’07)
Graphical models for decision-making in multiagent settings
Sequential decision-making over multiple time steps in multiagent settings
Generalize dynamic IDs to multiagent domains
Differ from MAIDs (Koller&Milch01) and NIDs (Gal&Pfeffer04)
Online solutions to I-POMDPs (Gmytrasiewicz&Doshi, JAIR’05)
Allow nested modeling of agents

4. Overview of I-ID Ri Ai
A generic level l Interactive-ID (I-ID) for agent i situated with one other agent j
Model Node: Mj,l-1
Models of agent j at level l-1
Policy link: dashed line
Distribution over the other agent’s actions given its models
Beliefs on Mj,l-1
P(Mj,l-1|s)
Update? Aj
Mj,l-1
Level l I-ID S Oi

5. Details of the Model Node
Members of the model node
Different chance nodes are solutions of models mj,l-1
Mod[Mj] represents the different models of agent j
CPT of the chance node Aj is a multiplexer
Assumes the distribution of each of the action nodes (Aj1, Aj2) depending on the value of Mod[Mj]
Mj,l-1 Aj S
Mod[Mj]
mj,l-11
Aj1
mj,l-11, mj,l-12 could be I-IDs or IDs
mj,l-12
Aj2

6. Interactive Dynamic Influence Diagrams (I-DIDs)Ri
Ait+1 Ri
Oit+1
St+1
Ajt+1
Mj,l-1t+1
Ait
Ajt St
Oit
Mj,l-1t
Model Update Link

7. Semantics of Model Update Link
Ajt+1
Mj,l-1t+1
Ajt
st+1
Mj,l-1t
Mod[Mjt+1] st
mj,l-1t+1,1
Aj1
Mod[Mjt]
mj,l-1t+1,2 Oj
Aj2
mj,l-1t+1,3
mj,l-1t,1
Aj3
Aj1
Oj1
mj,l-1t+1,4
mj,l-1t,2
Aj4
Aj2
Oj2
These models differ in their initial beliefs, each of which is the result of j updating its beliefs due to its actions and possible observations

8. Curses of History and Dimensionality
Primary complexity of solving I-DIDs is due to the large number of models that must be solved over time Curse of dimensionality
At time step t:
Nested property of modeling
More Agents
N+1 agent setting: (NM)l models (M is bounded # of models at each level)
Curse of history of agent j

9. Model Clustering Idea:
Prune the model space to K representative models from M candidate models, K << M, at each time step
Approach
Cluster Models
k-means clustering method (MacQueen67)
Note: k is not equal to K
Clusters contain models that are likely behaviorally equivalent
Select K representative models from the clusters

10. Selection of Initial Means
Facilitate clustering of behaviorally equivalent models
Behaviorally equivalent regions
Prescribe the same optimal behavior for j
[0,0.1], [0.1,0.9], [0.9,1]
Select region boundary points as initial means
0, 0.1, 0.9, 1 10 -1
Value L OL OR
0.1
0.9 1
P(TR)
Sensitivity points

11. Selection of Initial Means
Sensitivity points
Models that induce policies that are different from those by surrounding models
Vertices of the belief simplex
One dimension: 0, 1
Two dimensions: [0,0], [0,1],[1,0], and [1,1]

12. LP for Computing Sensitivity Points
SPs are non-dominated
points on intersections
between value functions SP
Non-dominated
Intersection

13. Example of Iterative Clustering
P(TR)
0.1
0.9 1
Initial Means
Iteration 1 . .
Iteration n
Select K=10

14. K Model Selection Algorithm Compute SPs Clustering
Select Initial Means
Selection
Compute SPs
Cluster
models
Re-compute
means
Select K
nearest models

15. Approximate Solution of I-DID
Exact algorithm
Expansion phase
Expand all M models over time
Look-ahead phase
Approximation – Modify exact algorithm
Prune model space using KModelSelection
Maintain only K models over time

16. Computational Savings and Error Bound
(NM)l V.S. (NK)l
M grows exponentially over time
Retain K models (Mk) and discard M-K models (M/K)
Error bounded by finding the model among the K retained models that is the closest to the discarded one (PBVI; Pineau et al. 03)

17. Error Bound Let Error bound for agent j
Expected error bound for agent i

18. Empirical Results Two Problem Domains Comparison with Measure
Multiagent tiger
Multiagent machine maintenance
Comparison with
Exact solution of I-DID for different M
Interactive particle filtering on I-DID
Measure
Average rewards solving the level 1 I-DIDs
Variance over 50 runs
Run time

21. Run Time Comparison Slower than the I-PF Solve I-DIDs up to 8 horizons
Reason: convergence step
Solve I-DIDs up to 8 horizons
Pro.
Tiger
Machine
Exact
83.6s
99.2s
K=20
K=50 MC
3.8s
10.5s
6.2s
18.7s
I-PF
3.9s
9.5s
4.3s
10.8s

22. Future Work Variants of model clustering Application domains
Compose our package for I-DIDs

23. Thank You!

24. Together: I-ID Ri Ai S Aj Oi
Mod[Mj]
Aj1
Aj2
mj,l-11
mj,l-12

25. Notes
Updated set of models at time step (t+1) will have at most models
:number of models at time step t
:largest space of actions
:largest space of observations
New distribution over the updated models uses
original distribution over the models
probability of the other agent performing the action, and
receiving the observation that led to the updated model

26. Exact Solution

27. Oit
Ait Ri St
Ait+1 Ri
Oit+1
St+1
Ajt Oj
Ajt+1
Mod[Mjt]
Mod[Mjt+1]
mj,l-1t+1,1
Aj1
mj,l-1t+1,2
Aj2
mj,l-1t+1,3
mj,l-1t,1
Aj2
Aj1
Oj1
mj,l-1t+1,4
mj,l-1t,2
Aj2
Aj2
Oj2

28. One Example

29. K Model Selection Initial Means Iteration Selection
Sensitivity points + Vertices of the belief simplex
Iteration
Re-compute the cluster mean
Assign new models to clusters
Selection
Select K models
Kn: In proportion to the size of cluster n