ST3236: Stochastic Process Tutorial 5

1
ST3236 Stochastic ProcessTutorial 5

• TA Mar Choong Hock
• Email g0301492_at_nus.edu.sg
• Exercises 6

2
Question 1

• Consider the MC with transition probability
  matrix
• Starting in state 1, determine the mean time that
  the
• process spends in state 1 prior to absorption and
  
• the mean time that the process spends in state 2
• prior to absorption. Verify that the sum of these
  is
• the mean time to the absorption.

3
Question 1
Denote by wi1 the mean time the process spends in
state 1 starting in state i prior to the
absorption. We have, w01 0 w11 1 0.1w01
0.2w11 0.5w21 0.2w31 w21 0 0.1w01
0.2w11 0.6w21 0.1w31 w31 0 The solution
is w01 0, w11 1.8182, w21 0.9091, w31 0.
4
Question 1
Similarly, denote by wi2 the mean time the
process spends in state 2 starting in state i
prior to the absorption. We have, w02 0 w12
0 0.1w02 0.2w12 0.5w22 0.2w32 w22 1
0.1w02 0.2w12 0.6w22 0.1w32 w32 0 The
solution is w02 0, w12 2.2727, w22 3.6364,
w32 0.
5
Question 1
Denote by vi the mean time to the absorption
starting in state i prior to the absorption. We
have, v0 0 v1 1 0.1v0 0.2v1 0.5v2
0.2v3 v2 1 0.1v0 0.2v1 0.6v2 0.1v3 v3
0 The solution is v0 0, v1 4.0909, v2
4.5455, v3 0. We have verified that, v1 w11
w12
6
Question 2

• Consider the MC with transition probability
  matrix
• Starting in state 1, determine the mean time that
  the
• process spends in state 1 prior to absorption and
  
• the mean time that the process spends in state 2
• prior to absorption. Verify that the sum of these
  is
• the mean time to the absorption.

7
Question 2
Denote by wi1 the mean time the process spends in
state 1 starting in state i prior to the
absorption. We have, w01 0 w11 1 0.5w01
0.2w11 0.1w21 0.2w31 w21 0 0.2w01
0.1w11 0.6w21 0.1w31 w31 0 The solution
is, w01 0, w11 1.290, w21 0.3225, w31 0.
8
Question 2
Similarly, denote by wi2 the mean time the
process spends in state 2 starting in state i
prior to the absorption. We have, w02 0 w12
0 0.5w02 0.2w12 0.1w22 0.2w32 w22 1
0.2w02 0.1w12 0.6w22 0.1w32 w32 0 The
solution is w02 0, w12 0.3230, w22 2.5808,
w32 0.
9
Question 2
Denote by vi the mean time to the absorption
starting in state i prior to the absorption. We
have, v0 0 v1 1 0.5v0 0.2v1 0.1v2
0.2v3 v2 1 0.2v0 0.1v1 0.6v2 0.1v3 v3
0 The solution is v0 0, v1 1.613, v2
2.9033, v3 0. We have verified that, v1 w11
w12
10
Question 3
Consider the MC in question 1. Starting in state
1, determine the probability that the process is
absorbed into state 0. Compare this with the
(1,0)th entry in the matrix powers P2, P4, P8
and P16.
11
Question 3
Denote by ui the probability that the MC is
absorbed by 0 starting in state i. We have, u0
1 u1 0.1u0 0.2u1 0.5u2 0.2u3 u2 0.1u0
0.2u1 0.6u2 0.1u3 u3 0 The solution
is, u0 1, u1 0.4091, u2 0.4545, u3 0.
12
Compare u1 0.4091
13
Question 3
By definition,
Consider a (4 x 4) transition probability matrix,
14
Question 3
But for our case, p00 1, p03 0.
15
Question 3-Optional

• Let
• F(t) be the set of t-step first passage paths
  from state 1 to state 0
• G(n-t) be the set of (n-t)-step paths from state
  0 to state 0
• H(t) be the set of paths that is formed jointly
  by F(t) followed by G(n-t). Note paths are
  n-step.

16
Question 3-Optional
17
Question 3-Optional
Let L(n) be the set of n-step paths from state 1
to state 0. s.t.
18
Question 3-Optional
f1,0(t) is the t-step first passage probability
from state 1 to state 0. If state 0 is an
absorbing state, Also, trivially,
19
Question 3-Optional
20
Question 4
Which of the following MC is regular a) b)
21
Question 4
a) YES, because (all entries are greater
than 0)
b) NO, because it has absorbing states