ST3236: Stochastic Process Tutorial 4

1
ST3236 Stochastic ProcessTutorial 4

• TA Mar Choong Hock
• Email g0301492_at_nus.edu.sg
• Exercises 5

2
Question 1
An urn initially contains a single red and single
green ball. A ball is drawn at random, removed
and replaced by a ball of the opposite color and
this procedure repeats so that there are always
two balls in the urn. Let Xn be the number of
red balls in the urn after n draws, with X0 1.
Specify the transitions probabilities for MC
X.
3
Question 1
Case (Xn0) Both balls are green. One ball will
certainly be replaced with red after a ball is
drawn. P(Xn1 1 Xn 0) 1, P(Xn1 0 Xn
0) 0, P(Xn1 2 Xn 0) 0
Case (Xn2) Both balls are red. One ball will
certainly be replaced with green after a ball is
drawn. P(Xn1 1 Xn 2) 1, P(Xn1 0
Xn 2) 0, P(Xn1 2 Xn 2) 0
4
Question 1
Case (Xn 1) One ball is red. Outcome dependent
on the colour of the ball drawn. Note P(Green
is drawnXn1) P (Red is drawnXn1) 0.5
P(Xn1 0 Xn 1) P(Red is drawnXn 1)
0.5 P(Xn1 2 Xn 1) P(Green is drawnXn
1) 0.5 P(Xn1 1 Xn 1) 0
5
Question 1
The transition matrix is given as follows
6
Question 2
Find the mean time to reach state 3 starting from
state 0 for the MC whose transition probability
matrix is
7
Question 2
Let T minn Xn 3 and vi E(T X0 i).
The mean time to reach state 3 starting from
state 0 is v0. We apply first step analysis.
8
Question 2
Therefore,
v0 1 0.4v0 0.3v1 0.2v2 0.1v3 v1 1
0v0 0.7v1 0.2v2 0.1v3 v2 1 0v0 0v1
0.9v2 0.1v3 v3 0 Solving the equations, we
have v0 10.
9
Question 3

• Consider the MC with transition probability
  matrix
• Starting in state 1, determine the probability
  that
• the MC ends in state 0
• (b) Determine the mean time to absorption.

10
Question 3a
Let T minn Xn 0 and Xn 2, ui P(XT
0X0 i) and vi E(TX0 i). u0 1 u1
0.1u0 0.6u1 0.3u2 u2 0 we have u1 0.25.
11
Question 3b
Let T minn Xn 0 and Xn 2, ui P(XT
0X0 i) and vi E(TX0 i). v0 0 v1 1
0.1v0 0.6v1 0.3v2 v2 0 we have v1 2.5.
12
Question 4

• Consider the MC with transition probability
  matrix
• Starting in state 1, determine the probability
  that
• the MC ends in state 0
• (b) Determine the mean time to absorption.

13
Question 4a
Let T minn Xn 0 and Xn 2, ui P(XT
0X0 i) and vi E(TX0 i). u0 1 u1
0.1u0 0.6u1 0.1u2 0.2u3 u2 0.2u0 0.3u1
0.4u2 0.1u3 u3 0 we have, u1 0.3810, u2
0.5238 Starting in state 1, the probability
that the MC ends in state 0 is u1 0.3810
14
Question 4b
Let T min n Xn 0 and Xn 2, ui P(XT
0X0 i) and vi E(TX0 i). v0 0 v1 1
0.1v0 0.6v1 0.1v2 0.2v3 v2 1 0.2v0
0.3v1 0.4v2 0.1v3 v3 0 we have v1 v2
3.33.
15
Question 5
A coin is tossed repeatedly until two successive
heads appear. Find the mean number of tosses
required. Hint Let Xn be the cumulative
number of successive heads. Then the state space
is 0,1,2 before stop
16
Question 5 Method 1
Let Yn be the outcome H, T of each toss
and (Yn-1, Yn) denotes the sample point for the
sucessive tosses. There are 4 possible sample
points.
17
(No Transcript)
18
Question 5 Method 1
The transition probability matrix is
19
Question 5 Method 1
Let vi denote the mean time to each state 3
starting from state i. By the first step
analysis, we have the following equations
v0 1 0.5v0 0.5v1 v1 1 0.5v2 0.5v3 v2
1 0.5v0 0.5v1 v3 0   Therefore, v0 6
(v1 4, v2 6)
20
Question 5 Method 2
Let Xn be the cumulative number of successive
heads. The 3-state state space now is
0,1,2. Example
n 0 1 2 3 4
Yn T H T H H
Xn 0 1 0 1 2
21
Question 5 Method 2
Case (Xn 0) Previous two tosses are tails Or a
head followed by tail. Note P(Head) P(Tail)
0.5 P(Xn1 1 Xn 0) P(Head) 0.5,
P(Xn1 0 Xn 0) P(Tail) 0.5, P(Xn1
2 Xn 0) 0
Case (Xn 1) A tail is followed by a
head P(Xn1 2 Xn 1) P(Head) 0.5 P(Xn1
0 Xn 1) P(Tail) 0.5 P(Xn1 1 Xn
1) 0
22
Question 5 Method 2
Case (Xn 2) Previous two tosses are heads. We
make this state absorbing. P(Xn1 1 Xn 2)
0 P(Xn1 2 Xn 2) 1, P(Xn1 0 Xn
2) 0
23
Question 5 Method 2
The transition probability matrix is
24
Question 5 Method 2
Let vi denote the mean time to each state 2
starting from state i. By the first step
analysis, we have the following equations
v0 1 0.5v0 0.5v1 v1 1 0.5v0 0.5v2 v2
0 Thus, we have, v0 6, v1 4