Exploiting Graphical Structure in Decision-Making

1
Exploiting Graphical Structure in Decision-Making

• Ben Van Roy
• Stanford University

2
Overview

• Graphical models in decision-making
• Singly-connected ? efficient computation
• General decision problems ? intractable
• Sparsity ? reduction in computation?
• Sequential decision-making
• Curse of dimensionality
• Sparsity or other graphical structure ? reduced
  computational requirements?
• Structured value functions and/or policies?
• Preliminary results and research directions

3
Graphical Models in Inference

• Conditional independencies simplify inference
• Singly-connected graphs (trees)
• General sparse graphs
• Preprocessing
• Approximations?

x1
x2
x4
x3
4
Graphical Models in Decision-Making

• Deterministic dynamic programming
• Nonserial dynamic programming
• General sparse graphs
• Preprocessing
• Approximations?

x1
x2
x4
x3
5
Sequential Decision-Making
state x(t)
decision u(t)
system
strategy

• Bellmans equation

J(x(t)) max Eg(x(t), u(t)) a J(x(t1))
x(t)
6
The Curse of Dimensionality

• states is exponential in variables
• The value function encodes one value per state
• Storage is intractable
• Computation is intractable
• Research objective exploit sparsity and other
  special graphical structure to reduce
  computational requirements of sequential decision
  problems

7
Dynamic Bayesian Networks
x1(t)
x1(t1)
x2(t)
x2(t1)
x3(t)
x3(t1)
x4(t)
x4(t1)
8
Example Multiclass Queueing Networks
u1
u2
x1
x3
x2
x5
x4
9
Can We Exploit Proximity?

• Idea variables that are far from each dont
  interact much
• Does this allow us to decompose the problem?

10
Yes

• The value function decomposes
• N(i) a neighborhood i.e. a set of nodes within
  some distance of i
• Complexity O(nd) ? O(dnN)
• but theres a problem here

11
Optimal Decisions Depend on Global State
information
u1(t1)
x1(t)
x1(t1)
x2(t)
x2(t1)
x3(t)
x3(t1)
x4(t)
x4(t1)
12
Things Still Work Out

• Conjecture
• If decision ui influences only xi
• Then near-optimal decisions can be made based
  only on variables near xi
• Consequence

u1(t1)
x1(t)
x1(t1)
x2(t)
x2(t1)
x3(t)
x3(t1)
x4(t)
x4(t1)
13
The Underlying Problem
x7
x2
x3
x1
x6
x4
x5

• Which fijs do I need to know to choose a
  near-optimal uk (without coordination)?

14
A Simple Case
x1
x2
x3
x4
x5
x6

• Let N(i) nodes within r steps
• Result loss of optimality O(1/r)
• Note amount of information required is
  independent of the graph size
• (Rusmevichientong and Van Roy, 2000)

15
Future Work

• Extending this result to general graphs
• Exploring practical implications
• Expected practical utility reduction of
  complexity in approximation algorithms
• Problem is no longer O(nd)
• May instead be O(dnr)
• Still computationally prohibitive, but not
  exponential in problem size
• Simplification of decision-supporting information?

16
More Future Work

• Current work exploits proximity
• Many graphs arising in practical problems pose
  additional special structure (e.g., symmetries,
  multiple layers of relationships, etc.)
• Can we also exploit such structure? (e.g., are
  there sometimes appropriate hierarchical
  representations?)