A PREDICTIVE THEORY OF GAMES

1
A PREDICTIVE THEORY OF GAMES David H.
Wolpert NASA Ames Research Center David.H.Wolpe
rt_at_nasa.gov http//ti.arc.nasa.gov/people/dhw/ NA
SA-ARC-05-097
2
ROADMAP
1) Review statistics and game theory
2) Apply statistics to games (as opposed to
within games)
3) E.g., Coupled players and Quantal Response Eq.
4) New mathematical tools rationality functions,
cost of computation, varying numbers of
players, etc.
3
ONLY IDEA IN THIS TALK
Human beings are physical objects
- Physical object that are goal-directed
though, whatever that means ...
4
REVIEW OF GAME THEORY
N independent players, each with possible
moves, xi ? Xi Each i has a distribution
qi(xi) q(x) ?iqi(xi) N utility functions
ui(x) player i wants maximal Eq(ui) Eq(ui)
depends on q but i only sets qi
Equilibrium concept mapping from ui ? q
Strawman Only equilibrium q can arise with
humans. All we must do is find the right
equilibrium concept.
5
REVIEW OF GAME THEORY - 2
Ex. 1 Nash Equilibrium (NE) q For all
players i, Eq(ui) cannot rise by changing qi Ex.
2 Quantal Response Equilibrium (QRE) q
Simultaneously for all i, qi(xi) ? exp?iE(ui
xi) Crude model of bounded rationality.
In fair agreement with experiment. Phase
transitions for finite systems.
6
REVIEW OF STATISTICS

• Probability theory is the only consistent
  calculus of
• uncertainty for making predictions about
  physical world
• 2) In particular consistency forces Bayes
  Theorem
• P(truth z knowledge ?) ? P(? z) P(z)
• 3) Given a P(z ?) and a loss function L(truth
  z, prediction y),
• the Bayes-optimal prediction is argminy
  EPL(., y) (Savage).
• 4) argmaxz P(z ?) is an approximation the MAP
  prediction

Probability theory to reason about physical
objects. Minimize expected loss to distill P(z)
to a single z.
7
EXAMPLE OF STATISTICS
1) Let the random variable we wish to predict
itself be a probability distribution, z
q(x). 2) Information theory tells us to use the
Entropic prior where S(q) is the
Shannon entropy of q, and ? ? ? 3) Let the
knowledge ? about q be Eq(H) h for some H(x)

• P(q) ? exp? S(q)

P(q ?) ? exp? S(q) ?Eq(H) - h
8
EXAMPLE OF STATISTICS - 2
4) So MAP q maximizes S(q') over the q' obeying
Eq'(H) h. 5) Let x be phase space position of
a physical system with H(x) the Hamiltonian.
The MAP q gives the Canonical Ensemble where
? is a Lagrange parameter (it equals 1 /
temperature) 6) If the numbers of particles of
various types also varies stochastically, the MAP
q is the Grand Canonical Ensemble.

• q(x) ? exp-?H(x)

9
ROADMAP
1) Review statistics and game theory
2) Apply statistics to games (as opposed to
within games)
3) E.g., Coupled players and Quantal Response Eq.
4) New mathematical tools rationality functions,
cost of computation, varying numbers of
players, etc.
10
ONLY IDEA IN THIS TALK
Human beings are physical objects
I.e., you are a Scientist playing against Nature.
Whether Nature is i) humans in a game, or
ii) interacting physical objects You should
infer Natures mixed strategy the same way.
11
GAME EQUILIBRIA
1) Humans are physical objects to reason about
the mixed strategy q of a game we must use
distributions over q N.b., bounded
rationality automatic with distributions over
q. 2) To distill a distribution over game
outcomes to a single prediction, can use a loss
function L assessing the prediction L
associated with the external scientist, not with
the players. No need for refinements
equilibrium q is unique.
Game theory equilibrium strawman is deficient
Predictive Equilibrium of a game
meaningless without a loss function.
12
GAME EQUILIBRIA - 2
3) Alternative way to distill P(q ?) to a
single distribution over x P(x ?) ?dq
P(q ?)P(x q, ?) ?dq P(q ?)q(x)
(I.e., marginalize over qs) 4) Problematic.
Even if support P(q ?) of is restricted to the
NE, if there are multiple NE q P(x ?) is
not a product distribution even though
players are independent under each NE q, to us
they are coupled. P(xi ?) is
not best-response to P(x-i ?).
13
THE POSTERIOR IN GAMES

• So the positive problem is to infer the
  posterior
• 2) Assume an entropic prior over q, P(q) ? exp?
  S(q)
• 3) How set the likelihood function P(? q) when
  data ? is the utility functions of the players?

P(joint mixed strategy q ui i 1, N)
14
ROADMAP
1) Review statistics and game theory
2) Apply statistics to games (as opposed to
within games)
3) E.g., Coupled players and Quantal Response Eq.
4) New mathematical tools rationality functions,
cost of computation, varying numbers of
players, etc.
15
COUPLED PLAYERS LIKELIHOOD

• Say players are statistically coupled.
• E.g., they have interacted before current
  game.
• Game outcome varies each time game is run. But
  how smart the players are is invariant. How
  formalize that?
• Define the (xi-indexed) vector
  
• Game instance Joint strat. Expected payoffs
  How smart
• 1 q(1)
  Uiq(1) Constant
• 2 q(2)
  Uiq(2) Constant
• . .
  . Constant
• . .
  . Constant

Ui(xi) E(ui xi)
16
COUPLED PLAYERS LIKELIHOOD - continued

• 4) Require single-valued map ?i(.) relates Uiq-i
  to Eqi(Uiq-i)
• Recall that VonNeumann-Morgenstern utility
  theory presumes that player i is only interested
  in Eqi(Uiq-i)
• details underlying are Uiq-i irrelevant.
• As q varies between games instances (i.e., by
  sampling P(q)), Ui does. And so ?i(Ui) and
  Eqi(Uiq-i) also change.
• But the relation of (4) always holds

Eqi(Uiq-i) ?i(Uiq-i)
17
COUPLED PLAYERS - continued
COUPLED PLAYERS LIKELIHOOD - continued

• 6) Example Nash equilibrium has ?i(Ui)
  maxxiUi(xi)
• 7) Example Uniform random play has ?i(Ui) lt
  Ui gt,
• the uniform average of Ui(xi) over xi.
• 8) Eqi(Ui) qi Ui
• So for any Ui(.), ?i(.), many qi obey Eqi(Ui)
  ?i(Ui).

Eqi(Uiq-i) ?i(Uiq-i)
Our requirement is very weak
18
COUPLED PLAYERS - continued
COUPLED PLAYERS LIKELIHOOD - continued
Eqi(Ui) ?i(Ui) ... but how set ?i ?

• Game against nature Gedanken experiment (N
  1)
• i) So q-i doesnt vary over game instances.
• ii) So ?i(Uiq-i) doesnt.
• iii) So (by blue) Eqi(Ui) doesnt - its fixed
  to some value.
• Whatever that value
• Assume entropic prior over qi.
• Then the MAP qi is a Boltzmann distribution.

19
COUPLED PLAYERS - continued
COUPLED PLAYERS LIKELIHOOD - continued

• 10) Under that MAP qi for that game against
  Nature,
• 11) Player i doesnt know if Ui set by Nature or
  by other players. So (10) holds in general this
  sets ?i(.)
• 12) Example qi(xi) ? exp?iUi(xi) (MAP for
  game against Nature) solves Eqi(Ui) ?i(Ui).
• Many other qi also solve it.
• ? In this example, a smart player - high
  ?i, so low Ti - is
• literally cold. And a dumb player is hot.

Eqi(Ui) ? ?x'i exp?iUi(x'i) Ui(x'i)
20
QUANTAL RESPONSE EQUILIBRIUM
1) So Bayes theorem says that with the entropic
prior over q, All ?i ? ? the support of
P(q ?) is the Nash equilibria. 2) Locally MAP
qs - local maxima of P(q ?) - are
approximated by a set of coupled equations
Quantal Response Eq.
P(q ?) ? exp? S(q) ?i?Eqi(Uiq-i) -
?i?i(Uiq-i)
qi(xi) ? exp?i Uiq-i (xi)
21
ROADMAP
1) Review statistics and game theory
2) Apply statistics to games (as opposed to
within games)
3) E.g., Coupled players and Quantal Response Eq.
4) New mathematical tools rationality functions,
cost of computation, varying numbers of
players, etc.
22
QUANTIFYING A PLAYERS RATIONALITY
Want a way to quantify how rational an
(arbitrary!) qi is, for an (arbitrary) effective
utility Ui. KL rationality is one solution to
some natural desiderata 1) Use
Kullbach-Leibler distance KL(p, p') to measure
distance between distributions p and p'.
2) KL rationality is the ?i minimizing the KL
distance from the associated Boltzmann
distribution to qi

• ?KL(Ui, qi) argmin ?i KL(qi, exp(?i Ui))

23
GAMES WITH VARIABLE NUMBERS OF PLAYERS
1) Recall The MAP q for physical systems where
the numbers of particles of various types
varies stochastically is the Grand Canonical
Ensemble (GCE). Intuition Players with types
particles with types 2) So MAP q for a game
with varying numbers of players is governed
by the GCE i) Corrections to replicator
dynamics, ii) New ways to analyze firms
(varying numbers of employees of
various types), etc.
24
INDEPENDENT PLAYERS

• Say players are statistically independent.
• E.g., they have never interacted before
  current game.
• Game outcome varies each time game is run. But
  how smart the players are against
  counterfactual opponents is invariant.
• This invariance is how common knowledge arises.
  Player i behaves like a player with
  temperature ?i playing against her prediction
  of her opponents behavior.
• That prediction is for a (counterfactual) coupled
  players game. The I know that you know ...
  regress is in that game.

25
INDEPENDENT PLAYERS - 2

• 1) To formalize this, use a usual likelihood
  constraint
• i) ??ii(.) is the usual game against Nature
  function.
• ii) Now though Uic is a counterfactual
  coupled-players environment for i
• where just like in the coupled players
  formulation,
• 2) Intuitively, player i imitates how we behave
  when we make a prediction for a coupled players
  scenario

Eqi(Uic) ??ii(Uic)
Uic(xi) ?dqc-i P(qc-i)Eqc-i(ui xi),
P(qc-i) ? ? dqci expac S(qc) ?i?Eqci(Uiqc-i)
- ?bii(Uiqc-i)
26
FUTURE WORK

• Apply to cooperative game theory - issue of what
  equilibrium concept to use rendered moot.
• Apply to mechanism design - bounded rational
  mechanism design, corrections to incentive
  compatibility criterion, etc.
• Extend (1, 2) to games with varying numbers of
  players.
• Investigate alternative choices of P(? q) and
  P(q), e.g., to reflect Allais paradox.
• Integrate (predictive) game theory with the field
  of user modeling (i.e., with modeling real people
  as Bayes nets).

27
CONCLUSION

• Probability theory governs outcome of a game
  there is a
• distribution over mixed strat.s, not a
  single equilibrium.
• 2) To predict a single mixed strategy must use
  our loss function
• (external to the games players).
• 3) Provides a quantification of any strategys
  rationality.
• 4) Prove rationality falls as cost of computation
  rises
• (for players who have not previously
  interacted).
• 5) All extends to games with varying numbers of
  players.

28
QRE and BAYES OPTIMALITY

• 1) Unimodal P(q ?)
• The QRE approximates a q (the MAP),
• which in turn approximates
• the Bayes-optimal q.
• How good an approximation depends on
  loss function.
• Multimodal P(q ?), non-exchangeable modes. All
  ?i ? ? (full rationality). P(x) ?dq P(q)P(xq)
  ?dq P(q)q(x).

• For non-exchangeable Nash equilibria,
• P(x) may not even be a product distribution.
• Never mind a Nash equilibrium.

29
1st EXAMPLE OF PROBABILITY THEORY

• 1) ? is a game with joint move x, joint mixed
  strategy q(x).
• 2) Meta-game of a scientist (S) playing against
  nature (N).
• 3) Ns move is the q of ?. Ss move, qS, is a
  prediction of q.
• 4) Ss utility function, u(qS, q), is negative
  loss.

S maximizing expected utility a Bayes-optimal
scientist.
Central problem of real-world game theory (i.e.,
for S) What is Ns mixed strategy, P(q)?
30
TWO TYPES OF GAMES

• Type I games
• i) Each player i consciously chooses a move
  (pure strategy) xi
• ii) i s choice varies with her mood,
  attentiveness, etc.
• (Physically, xi varies with is
  neurotransmitter levels, etc.)
• iii) So xi is set by a (physical) probability
  distribution qi(xi )
• iv) Our (!) inference of qi is encapsulated in
  P(qi).
• 2) Type II games
• i) Each player i consciously chooses a mixed
  strategy qi
• ii) So qi is set by a (physical) probability
  distribution ?i(qi )
• iii) Our inference of ?i is encapsulated in
  P(?i).

31
THE LIKELIHOOD FUNCTION IN GAMES
1) Consider N 1 (a game against Nature), so
Ui is constant i) Use desiderata to fix a
parameterized map from Ui to qi qi
F?i(Ui) ii) Since desiderata are always
partially wrong, relax them Only
require that what ultimately interests the
player, qi ? Ui, is consistent with
the parameterization Only require that qi
obeys qi ? Ui ? F?i(Ui) ? Ui ? ??i(Ui). iii)
?i characterizes how smart player i
is 2) Generalize For N players, likelihood is
P(q ui) ?i ???i(Ui) - qi ? Ui