ESI 4313 Operations Research 2

1
ESI 4313Operations Research 2

• Markov Chains
• Lecture 39 April 13, 2006

2
Ergodic Markov chains

• If all states in a Markov chain communicate with
  each other and are
• Recurrent
• Aperiodic
• then the chain is called ergodic
• Example
• Smallvilles weather Markov chain
• but not
• Gambling Markov chain

3
Limiting behavior

• If a Markov chain is ergodic, then there exists
  some vector
• such that

4
Limiting behavior

• Interpretation
• After many transitions, the probability that we
  are in state i is approximately equal to ?i,
  independent of the starting state
• The values ?i are called the steady-state
  probabilities, and the vector ? is called the
  steady-state (probability) distribution or
  equilibrium distribution

5
Limiting behavior

• Can you explain why a steady-state distribution
  exists for ergodic Markov chains, but not for
  other chains?
• Consider the case of
• Periodic chains
• non-communicating pairs of states
• In the following, we will assume we have an
  ergodic Markov chain

6
Limiting behavior

• How do we compute the steady-state probabilities
  ??
• Recall the following relationship between the
  multi-step transition probabilities
• The theoretical result says that, for large n

7
Limiting behavior

• So, for large n,
• becomes
• In matrix notation,

8
Limiting behavior

• But
• Has as a solution
• In fact, it has an infinite number of solutions
  (because P is singular why?)
• Fortunately, we know that ? should be a
  probability distribution, so not all solutions
  are meaningful!
• We should ensure that

9
Example 3

• Recall the 1st Smalltown example
• 90 of all sunny days are followed by a sunny day
• 80 of all cloudy days are followed by a cloudy
  day
• Find the steady-state probabilities

10
Example 3

• We need to solve the system
• or
• We can ignore one of the first 2 equations! (Why?)

11
Example 3

• Solving
• gives
• Compare with