ESI 4313 Operations Research 2

1
ESI 4313Operations Research 2

• Markov Chains
• Lecture 20

2
Initial state

• Often, we do not know the state of the Markov
  chain at time 0
• Suppose that we do have some information
  regarding the initial state, in the form of a
  probability distribution for that state, i.e.,

3
Initial state

• In that case, what is the probability
  distribution of the state of the Markov chain at
  time n?
• We have to condition on the initial state
• The distribution of Xn can be found by computing
  qTPn

4
Example 3

• Recall the 1st Smalltown example
• 90 of all sunny days are followed by a sunny day
• 80 of all cloudy days are followed by a cloudy
  day

5
Example 3

• Suppose that on day 0 it is sunny with 50
  probability and cloudy with 50 probability
• The probability distribution of the weather on
  day 3 is then given by

6
Example 3

• Alternatively, suppose that on day 0 it is sunny
  with 2/3 probability and cloudy with 1/3
  probability
• The probability distribution of the weather on
  day 1 is then given by
• Can you explain this?

7
Mean first passage time

• Consider an ergodic Markov chain
• Suppose we are currently in state i
• What is the expected number of transitions until
  we reach state j ?
• This is called the mean first passage time from
  state i to state j and is denoted by mij
• For example, in Smallvilles weather example, m12
  would be the expected number of days until the
  first cloudy day, given that it is currently
  sunny
• How can we compute these quantities?

8
Mean first passage time

• We are currently in state i
• In the next transition, we will go to some state
  k
• If kj, the first passage time from i to j is 1
• If k?j, the mean first passage time from i to j
  is 1mkj
• So

9
Mean first passage time

• We can thus find all mean first passage times by
  solving the following system of equations
• What is mii?
• The mean number of transitions until we return to
  state i
• This is equal to 1/?i !

10
Example 3

• Recall the 1st Smalltown example
• 90 of all sunny days are followed by a sunny day
• 80 of all cloudy days are followed by a cloudy
  day
• The steady-state probabilities are ?12/3 and
  ?21/3

11
Example 3

• Thus
• m11 1/?1 1/(2/3) 1½
• m22 1/?2 1/(1/3) 3
• And m12 and m21 satisfy

12
Absorbing chains

• While many practical Markov chains are ergodic,
  another common type of Markov chain is one in
  which
• some states are absorbing
• the others are transient
• Examples
• Gambling Markov chain
• Work-force planning

13
Example

• State College admissions office has modeled the
  path of a student through State College as a
  Markov chain
• States
• 1Freshman, 2Sophomore, 3Junior, 4Senior,
  5Quits, 6Graduates
• Based on past data, the transition probabilities
  have been estimated

14
Example

• Transition probability matrix

15
Example

• Clearly, states 5 and 6 are absorbing states, and
  states 1-4 are transient states
• Given that a student enters State College as a
  freshman, how many years will be spent as
  freshman, sophomore, junior, senior before
  entering one of the absorbing states?

16
Absorbing chains

• While many practical Markov chains are ergodic,
  another common type of Markov chain is one in
  which
• some states are absorbing
• the others are transient
• Examples
• Gambling Markov chain
• Work-force planning

17
Example

• State College admissions office has modeled the
  path of a student through State College as a
  Markov chain
• States
• 1Freshman, 2Sophomore, 3Junior, 4Senior,
  5Quits, 6Graduates
• Based on past data, the transition probabilities
  have been estimated

18
Example

• Transition probability matrix

19
Example

• Clearly, states 5 and 6 are absorbing states, and
  states 1-4 are transient states
• Given that a student enters State College as a
  freshman, how many years will be spent as
  freshman, sophomore, junior, senior before
  entering one of the absorbing states?

20
Example

• Consider the expected number of years spent as a
  freshman before absorption (quitting or
  graduating)
• Definitely count the first year
• If the student is still a freshman in the 2nd
  year, count it also
• If the student is still a freshman in the 3rd
  year, count it also
• Etc.

21
Example

• Recall the transition probability matrix

matrix Q
matrix R
matrix I
22
Example

• The number of years spent in states 1-4 when
  entering State College as a freshman can now be
  found as the first row of the matrix (I-Q)-1
• If a student enters State College as a freshman,
  how many years can the student expect to spend
  there?
• Sum the elements in the first row of the matrix
  (I-Q)-1

23
Example

• In this example
• The expected (average) amount of time spent at
  State College is 1.11 0.99 0.99 0.88
  3.97 years

24
Example

• What is the probability that a freshman
  eventually graduates?
• We know the probability that a starting freshman
  will have graduated after n years from the n-step
  transition probability matrix
• Its the (1,6) element of Pn !
• The probability that the student will eventually
  graduate is thus
• the (1,6) element of limn??Pn !

25
Example
26
Example

• So the probability that a student that starts as
  a freshman will eventually graduate is
• the (1,6-42) element of (I-Q)-1R
• In the example