Application of Game Theory

1
Application of Game Theory to Radio Resource
Management
James Neel January 15, 2004 Qualifier Presentation
2
Presentation Overview

• Radio Resource Management
• Game Theory
• Application of Game Theory to RRM
• Summary
• Research Directions

3
Radio Resource Management

• Definition, Categorization, Issues with
  Distributed RRM

4
Radio Resource Management

• Fundamentally an optimization problem
• Definition from an optimization perspective
• Given a particular infrastructure deployment
  (constraints), allocate resources (variables) in
  a manner that (ideally) max(min)imize some
  operational parameter(s) (objective functions).

5
Radio Resource Management
RRM
Resource Design
Resource Allocation
Dynamic Allocation
Fixed Allocation
Distributed
Centralized
6
Fixed Resource Design

• Determine system-wide network parameters based on
  expected operational parameters and performance
  and economic criteria.
• Occurs in initial planning stages
• Commonly addressed issues
• Total system bandwidth
• Number of access points (base stations)
• Operational waveform
• Frequency reuse factor
• Antenna heights
• Acceptable power levels
• May include specification of performance criteria

7
Fixed Resource Allocation

• Determine (initial) how the system wide resources
  are initially divided up among the various system
  components based on expected operational
  parameters and performance criteria.
• Generally occurs in initial planning stages
• Commonly addressed issues
• How many access points are needed (coverage vs
  capacity vs cost)?
• How are resources allocated among the access
  points (channel assignments)?
• How is user mobility handled (how many channels
  are reserved for handoffs)?
• How much resources does each user receive
  (bandwidth, target SINRs)?

8
Dynamic Resource Management

• Because of variations, dynamic resource
  allocation can outperform fixed resource
  allocation
• But dynamic resource allocation almost
  necessarily incurs some overhead penalty.

Figure 10.2 J. Zander, S. Kim, Radio Resource
Management for Wireless Networks, Artech House
Publishers, Boston, 2001.
9
CSMA/CA Scenario

• To limit collisions, nodes typically announce
  intentions and reserve channel
• Reduces collisions, but system throughput less
  than ideal as other nodes in network must wait
  for transmitting nodes to finish

10
CSMA/CA Scenario

• Various authors have suggested using power
  control to limit the propagation of packets
  Agarwal01, Monks00, Lin03
• Significant improvements in performance possible
  Agarwal01, Monks01

11
Approaches to Dynamic Resource Allocation

• Centralized
• Single decision maker
• Finds jointly optimal allocation
• Advantages
• Predictable
• Best system wide solution
• Disadvantages
• High overhead
• Infrastructure constraints
• Examples
• Single Cell Closed Loop Power Control
• Round robin scheduling
• Beamforming

• Distributed
• Multiple decision makers
• Finds locally optimal allocations
• Advantages
• Minimal overhead
• Normally scales well
• Disadvantages
• Interactive decision process (analysis harder),
  potentially undesirable operation
• Examples
• Most ad-hoc RRM algorithms
• Multi-cell power control

12
Analyzing Distributed Dynamic Behavior

• Dynamic Benefits
• Improved Spectrum Utilization
• Improve QoS
• Many decisions may have to be localized
• Distributed Behavior
• Adaptations of one radio can impact adaptations
  of others
• Interactive Decisions
• Difficult to Predict Performance

13
Key Distributed RRM Issues

• Steady State Existence
• Steady State Optimality
• Convergence
• Stability
• Scalability

Convergence How do initial conditions impact
the system steady state? How long does it
take to reach the steady state?
Stability How does system variations impact
the system? Do the steady states change?
Is convergence affected
Steady State Existence Is it possible to
predict behavior in the system? How many
different outcomes are possible?
Optimality Are these outcomes desirable?
Do these outcomes maximize the system target
parameters?
Scalability As the number of devices
increases, How is the system impacted?
Do previously optimal steady states remain
optimal?
14
Game Theory

• Definition, Key Concepts, Important Models

15
Game Theory

• Game Theory is a part of (applied) mathematics
  that describes and studies interactive decision
  problems.
• In an interactive decision problem the decisions
  made by each decision maker affect the outcomes
  and, thus, the resulting situation for all
  decision makers involved.
• The study of mathematical models of conflict and
  cooperation between intelligent rational
  decision-makers Myerson (1997)

16
Mathematical Models

• Modeling some interactive process
• Necessitates some abstraction (sometimes quite a
  lot of abstraction)
• Many different models exist to analyze the same
  situation
• Formal analysis greatly aided by rigorous
  mathematical descriptions of models

17
Key Concepts

• Decision Makers
• Agents who can choose an action
• Must be multiple decision makers
• Conflict and Cooperation
• Key is interaction, i.e., actions of one agent
  (player) must impact others either negatively or
  positively
• Effect of interaction is dependent on selected
  model

18
Intelligent Rational Decision Making

• Actions taken by an agent must be in that agents
  self-interest
• Clearly defined objectives
• Expectation on how actions impact objectives

19
Game

• A game is model (mathematical representation) of
  an interactive decision situation.
• Its purpose is to create a formal framework that
  captures the relevant information in such a way
  that is suitable for analysis.
• Different situations indicate the use of
  different game forms.

20
Critical Components of a Game

• A (well-defined) set of 2 or more players
• A set of actions for each player.
• A set of preference relationships for each player
  for each possible action tuple.

More elaborate games exist with more components
but these three must always be there. Some also
introduce an outcome function which maps action
tuples to outcomes which are then valued by the
preference relations. Games with just these three
components (or a variation on the preference
relationships) are said to be in Normal form or
Strategic Form
21
Set of Players

• N set of n players consisting of players
  named 1, 2, 3,,i, j,,n
• Note the n does not mean that there are 14
  players in every game.
• Other components of the game that belong to a
  particular player are normally indicated by a
  subscript.
• Generic players are most commonly written as i or
  j.
• Usage N is the SET of players, n is the COUNT of
  players.
• N \ i 1,2,,i-1, i1 ,, n All players in N
  except for i

22
Actions
Example Two Player Action Space
Ai Set of available actions for player i ai A
particular action chosen by i, ai ? Ai A Action
Space, Cartesian product of all Ai AA1? A2?
? An a Action tuple a point (vector) in the
Action Space A-i Another action space A formed
from A-i A1? A2? ?Ai-1 ? Ai1 ? ?
An a-i A point from the space A-i A Ai ? A-i
A1 A2 0 ?)
AA1? A2
A2 A-1
A1 A-2
a1 a-2
23
Preference Relations
Preference Relation expresses an individual
players desirability of one outcome over another
(A binary relationship)
Preference Relationship (prefers at least as much
as)
a is preferred at least as much as a by player i
Strict Preference Relationship (prefers strictly
more than)
iff
but not
Indifference Relationship (prefers equally)
iff
and
24
Utility Functions
A mathematical description of preference
relationships.
Maps action space to set of real numbers.
Preference Relation then defined as
iff
iff
iff
25
Nash Equilibrium
A steady-state where each player holds a correct
expectation of the other players behavior and
acts rationally. - Osbourne
An action vector from which no player can
profitably unilaterally deviate.
Definition
An action tuple a is a NE if for every i ? N
for all bi ?Ai.
Note showing that a point is a NE says nothing
about the process by which the steady state is
reached. Nor anything about its uniqueness. Also
note that we are implicitly assuming that only
pure strategies are possible in this case.
26
Friend or Foe Example
(Friend, Friend)??
No
(Friend, Foe)??
(Foe, Friend)??
Yes
(Foe, Foe)??
Yes
27
How do the players find the Nash Equilibrium?

• Preplay Communication
• Before the game, discuss their options. Note
  only NE are suitable candidates for coordination
  as one player could profitably violate any
  agreement.
• Rational Introspection (Best Response)
• Based on what each player knows about the other
  players, reason what the other players would do
  in its own best interest. Points where everyone
  would be playing correctly are the NE.
• Focal Point
• Some distinguishing characteristic of the tuple
  causes it to stand out. The NE stands out
  because its every players best response.
• Trial and Error (Better Response with Errors)
• Starting on some tuple which is not a NE a player
  discovers that deviating improves its payoff.
  This continues until no player can improve by
  deviating. Only guaranteed to work for Potential
  Games (later).

28
Repeated Games

• A specialized form of an extensive form game
  wherein at each stage, the same game is played.
• A particular stage of the repeated game is called
  a stage game.
• In each stage game, a subset of the players
  modify their behavior according to some
  predefined rule.
• Repeated Game may continue indefinitely
  Infinite Horizon
• Repeated Game may end after an expected number of
  stages Finite Horizon

29
Comments on Repeated Game Components

• Player Set
• Remains the same in each stage
• Actions
• Action space is the same in each stage
• Players play an action in each stage (need not
  change from previous stage)
• Determination of action in each stage is based on
  each players strategy, observations, insights
  about other players, and possibly negotiations
  (threats, promises), may learn
• Utility Function
• Players have the same utility function for each
  stage.
• The utility for the entire repeated game is some
  combination of the stage game utilities.
• Time Discounting More weight for more immediate
  payoffs

30
Comments on Play

• Myopic Processes
• Players have no knowledge about utility
  functions, or expectations about future play,
  typically can observe or infer current actions
• Best response dynamic maximize individual
  performance presuming other players actions are
  fixed
• Better response dynamic improve individual
  performance presuming other players actions are
  fixed

31
Example Repeated Game

• Stage Game
• 2 Players
• 2 Actions
• Stage Game Utility as Shown
• 3 Stages
• Repeated Game Utility sum of utility at each stage

Foe
Friend
Friend
-10,1000
500, 500
Foe
0,0
1000,-10
32
Better Response Dynamic
Foe
Friend
Game starts in (Friend, Friend)
Friend
-10,1000
500, 500
Foe
0,0
1000,-10
In stage 2, Player 2 improves his payoff (Friend,
Friend)
Note Convergence would have happened from any
point The convergence of the better response
dynamic is due to special properties of the stage
game.
In stage 3, Player 1 improves his payoff (Foe,
Foe)
33
More Complex Play

• Strategies that punish or reward other players to
  influence their actions
• Grim Trigger
• Once a player deviates from agreed upon action
  tuple, remaining players attempt to minimize
  deviating players utility for remainder of game
• Tit-for-Tat
• Reward good behavior, punish bad behavior

34
Repeated Games and Complex Play

• Depending on structure and decision updating
  algorithm, play may or may not lead to NE of the
  stage game.
• By Folk Theorem, play can be forced to any
  feasible payoff vector with proper selection of
  punishment strategy and discount factor.

35
Potential Games

• Key Properties
• Nash Equilibrium Exists
• Better Response (Myopic) Dynamic Converges
• Why We Care
• Steady state exists
• Virtually every decision updating process
  converges
• Minimal level of network complexity
• Once Modeled, Steady States Easy to Identify
  (Potential Function Maximizers)
• How to Identify

Or find an ordinal transformation for which this
works.
36
Supermodular Games

• Key Properties
• Best Response (Myopic) Dynamic Converges
• Nash Equilibrium Generally Exists
• Why We Care
• Most decision updating processes converge
• Low level of network complexity
• How to Identify

37
Application of Game Theory to RRM

• Under what conditions can game theory be applied?
• What capabilities does this imply?
• How to formally model a RRM algorithm?
• What kinds of information can be gleaned from
  this application?

38
Conditions for Applying Game Theory to RRM

• Conditions for Rationality
• Well defined decision making processes
• Expectation of how changes impacts performance
• Conditions for a Nontrivial Game
• Multiple interactive decision makers
• Nonsingleton action sets
• Conditions generally satisfied by distributed
  dynamic RRM schemes

39
Example Application Appropriateness

• Inappropriate Applications
• Cellular Downlink power control (single cell)
• Site Planning
• Most unadulterated random access schemes
• Appropriate Applications
• Distributed power control on non-orthogonal
  waveforms
• Ad-hoc power control
• Cell breathing
• Adaptive MAC
• Network formation (localized objectives)

40
Implied Radio Functionalities

• Minimum capabilities
• Observation Process
• Observation Valuation
• Decision Updating Process
• Similar to Level 3 Cognitive Radio

Modified from Table 4-1, J. Mitola, III
Cognitive Radio An Integrated Agent
Architecture for Software Defined Radio, PhD
Dissertation Royal Institute of Technology,
Sweden, May 2000.
41
Modeling a Network as a Game
Network
Game
Nodes
Players
Power Levels
Actions
Algorithms
Utility Functions Learning
42
Physical Layer Model Parameters
43
SINR Power Control Games
Assume that there is a radio network wherein each
radio can alter their power.
Assume each radio reacts to some separable
function of SINR, e.g. log ratio
Each radio would also like to minimize power
consumption
Decentralized Power Control Using a dB Metric
Thus game is a potential game and convergence is
assured and we can quickly find steady states.
44
Example Power Control Game

• Parameters
• Single Cluster
• DS-SS multiple access
• Pi Pj 0, Pmax ? i,j ?N
• Utility target BER

Also a potential game.
45
Snapshot inner outer loop power control

• Parameters
• Single Cluster
• DS-SS multiple access
• Pi Pj 0, Pmax ? i,j ?N
• Utility target SINR
• Supermodular best response convergence

46
Game Models, Convergence, and Complexity

• Determining the kind of game required to
  accurately model a RRM algorithm yields
  information about what updating processes are
  appropriate and thus indicates expected network
  complexity.
• In Neel04 the following relation between power
  control algorithms, game models, and network
  complexity was observed.

47
Steady State Modification

• Suppose during the algorithm design stage, it is
  discovered that a is a steady state, but a is
  desired.
• If network can be modeled as an exact potential
  the steady state can be modified as follows
• Solve for the Network Cost function, NC, from
  the following equation.
• Simplest approach may be to form correspondence
  from Cartesian product of single player cost
  functions.
• Lower order cost functions (such as a plane) can
  limit the number of new NE that are introduced.

48
Summary

• Distributed dynamic resource allocations have the
  potential to provide performance gains with
  reduced overhead, but introduce a potentially
  problematic interactive decision process.
• Game theory is not always applicable.
• Can generally be applied to distributed radio
  resource management schemes.

49
Summary

• Introduced set of conditions for applying game
  theory to RRM
• Multiple rational interactive decision makers
• Described required node functionality
• Formal model for physical layer games
• Examined a handful of applications
• Game theory good for identifying steady states
  and establishing convergence criteria

50
Research Directions

• Primary focus on power control algorithms
• Steady-state solutions for common ad-hoc and
  distributed cellular power control algorithms
• Convergence criteria for these algorithms
• Determination of how scaling impacts solutions
• Determination of how stochastic channel models
  impact solutions
• Development of techniques to improve steady-state
  performance
• Power control algorithms that incorporate rate
  adaptation.