1. Exploiting Graphical Structure in Decision-Making
Ben Van Roy
Stanford University

2. Overview Graphical models in decision-making
Singly-connected  efficient computation
General decision problems  intractable
Sparsity  reduction in computation?
Sequential decision-making
Curse of dimensionality
Sparsity or other graphical structure  reduced computational requirements?
Structured value functions and/or policies?
Preliminary results and research directions

3. Graphical Models in Inference
Conditional independencies simplify inference
Singly-connected graphs (trees)
General sparse graphs
Preprocessing
Approximations? x1 x2 x3 x4

4. Graphical Models in Decision-Making
Deterministic dynamic programming
Nonserial dynamic programming
General sparse graphs
Preprocessing
Approximations? x1 x2 x3 x4

5. Sequential Decision-Making
u(t)
state
x(t)
system
strategy
Bellman’s equation
J*(x(t)) = max E[g(x(t), u(t)) + a J*(x(t+1)) | x(t)]

6. The Curse of Dimensionality
# states is exponential in # variables
The value function encodes one value per state
Storage is intractable
Computation is intractable
Research objective: exploit sparsity and other special graphical structure to reduce computational requirements of sequential decision problems

7. Dynamic Bayesian Networks
x1(t)
x1(t+1)
x2(t)
x2(t+1)
x3(t)
x3(t+1)
x4(t)
x4(t+1)

8. Example Multiclass Queueing Networks

9. Can We Exploit Proximity?
Idea: variables that are “far” from each don’t interact much
Does this allow us to decompose the problem?

10. Yes… The value function decomposes
N(i) = a neighborhood; i.e. a set of nodes within some “distance” of i
Complexity: O(nd)  O(dnN)
…but there’s a problem here…

11. Optimal Decisions Depend on Global State information
u1(t+1)
x1(t)
x1(t+1)
x2(t)
x2(t+1)
x3(t)
x3(t+1)
x4(t)
x4(t+1)

12. Things Still Work Out… Conjecture: Consequence:
If decision ui influences only xi
Then near-optimal decisions can be made based only on variables “near” xi
Consequence:
u1(t+1)
x1(t)
x1(t+1)
x2(t)
x2(t+1)
x3(t)
x3(t+1)
x4(t)
x4(t+1)

13. The Underlying Problemx7 x2 x3 x1 x6 x4 x5
Which fij’s do I need to know to choose a near-optimal uk (without coordination)?

14. A Simple Case Let N(i) = nodes within r stepsx1 x2 x3 x4 x5 x6
Let N(i) = nodes within r steps
Result: loss of optimality ~ O(1/r)
Note: amount of information required is independent of the graph size
(Rusmevichientong and Van Roy, 2000)

15. Future Work Extending this result to general graphs
Exploring practical implications
Expected practical utility: reduction of complexity in approximation algorithms
Problem is no longer O(nd)
May instead be O(dnr)
Still computationally prohibitive, but not exponential in problem size
Simplification of decision-supporting information?

16. More Future Work Current work exploits proximity
Many graphs arising in practical problems pose additional special structure (e.g., symmetries, multiple “layers” of relationships, etc.)
Can we also exploit such structure? (e.g., are there sometimes appropriate hierarchical representations?)