Lecture 12.5

1
Lecture 12.5 Additional Issues Concerning
Discrete-Time Markov Chains

• Topics
• Review of DTMC
• Classification of states
• Economic analysis
• First-time passage
• Absorbing states

2
Discrete-Time Markov Chain
A stochastic process Xn where n ? N 0, 1,
2, . . . is called a discrete-time Markov chain
if Pr Xn1 j X0 k0, . . . , Xn-1 kn-1,
Xn i Pr Xn1 j Xn i ?
transition probabilities for every i, j, k0,
. . . , kn-1 and for every n.
The future behavior of the system depends only on
the current state i and not on any of the
previous states.
3
Stationary Transition Probabilities
Pr Xn1 j Xn i Pr X1 j X0 i
for all n (They dont change over time) We will
only consider stationary Markov chains.
4
Classification of States
Accessible Possible to go from state i to state
j (path exists in the network from i to j).
Two states communicate if both are accessible
from each other. A system is irreducible if all
states communicate. State i is recurrent if the
system will return to it after leaving some time
in the future. If a state is not recurrent, it is
transient.
5
Classification of States (continued)
A state is periodic if it can only return to
itself after a fixed number of transitions
greater than 1 (or multiple of a fixed number). A
state that is not periodic is aperiodic.
a. Each state visited every 3 iterations
b. Each state visited in multiples of 3 iterations
6
Classification of States (continued)
An absorbing state is one that locks in the
system once it enters.
This diagram might represent the wealth of a
gambler who begins with 2 and makes a series of
wagers for 1 each. Let ai be the event of
winning in state i and di the event of losing in
state i. There are two absorbing states 0 and 4.
7
Classification of States (continued)
Class set of states that communicate with each
other. A class is either all recurrent or all
transient and may be either all periodic or
aperiodic. States in a transient class
communicate only with each other so no arcs enter
any of the corresponding nodes in the network
diagram from outside the class. Arcs may leave,
though, passing from a node in the class to one
outside.
8
Illustration of Concepts
Example 1
Every pair of states communicates forming a
single recurrent class moreover, the states are
not periodic. Thus the stochastic process is
aperiodic and irreducible.
9
Illustration of Concepts
Example 2
States 0 and 1 communicate and form a recurrent
class. States 3 and 4 form separate transient
classes. State 2 is an absorbing state and forms
a recurrent class.
10
Illustration of Concepts
Example 3
Every state communicates with every other state,
so we have an irreducible stochastic process.
Periodic?
Yes, so Markov chain is irreducible and periodic.
11
Classification of States
Example
.6
.7
1
2
.4
4
.5
.4
.5
.5
.3
.8
.1
5
.2
12
A state j is accessible from state i if pij(n) gt
0 for some n gt 0. In example, state 2 is
accessible from state 1 state 3 is
accessible from state 5 but state 3 is not
accessible from state 2.
States i and j communicate if i is accessible
from j and j is accessible from i. States 1 2
communicate also states
3, 4 5 communicate. States 2 4 do not
communicate
States 1 2 form one communicating class.
States 3, 4 5 form a 2nd communicating class.
13
If all states in a Markov chain communicate
(i.e., all states are members of the same
communicating class) then the chain is
irreducible.
The current example is not an irreducible Markov
chain. Neither is the Gamblers Ruin
example which has 3 classes 0, 1, 2,
3 and 4.
First Passage Times Let fii probability that
the process will return to state i
(eventually) given that it starts in state i.
If fii 1, then state i is called recurrent.
If fii lt 1, then state i is called transient.
14
If pii 1, then state i is called an absorbing
state. Above example has no
absorbing states States 0 4 are absorbing in
Gamblers Ruin problem.
The period of a state i is the smallest k gt 1
such that all paths leading back to i have a
length that is a multiple of k i.e., pii(n)
0 unless n k, 2k, 3k, . . . If a process can
be in state i at time n or time n 1 having
started at state i then state i is aperiodic.
Each of the states in the current example are
aperiodic.
15
Example of Periodicity - Gamblers Ruin
States 1, 2 and 3 each have period 2.
0 1 2 3 4 0 1 0 0 0 0 1 1-p 0 p 0 0 2 0
1-p 0 p 0 3 0 0 1-p 0 p 4 0 0 0 0 1
If all states in a Markov chain are recurrent,
aperiodic, the chain is irreducible then it
is ergodic.
16
Existence of Steady-State Probabilities
A Markov chain is ergodic if it is aperiodic and
allows the attainment of any future state from
any initial state after one or more transitions.
If these conditions hold, then
Conclusion chain is ergodic.
17
Economic Analysis

• Two kinds of economic effects
• those incurred when the system is in a specified
  state, and
• those incurred when the system makes a transition
  from one state to another.

18
Expected Cost for Markov Chain
Expected cost of being in state i
Let C (c1, . . . cm)T ei (0, 0, 1, 0, 0) be
the ith row of the m ? m identity matrix, and fn
random variable representing the economic
return associated with the stochastic process at
time n.
Property 3 Let Xn n 0, 1, . . . be a
Markov chain with finite state space S,
state-transition matrix P, and expected state
cost (profit) vector C. Assuming that the
process starts in state i, the expected cost
(profit) at the nth step is given by  Efn(Xn)
X0 i eiP(n)C.
19
Additional Cost Results
What if the initial state is not known? Property
5 Let Xn n 0, 1, . . . be a Markov chain
with finite state space S, state-transition
matrix P, initial probability vector q(0), and
expected state cost (profit) vector C. The
expected economic return at the nth step is given
by   Efn(Xn) q(0) q(0)P(n)C.
Property 6 Let Xn n 0, 1, . . . be a
Markov chain with finite state space S,
state-transition matrix P, steady-state vector p,
and expected state cost (profit) vector C. Then
the long-run average return per unit time is
given by   Si?S pici pC.
20
Insurance Company Example
An insurance company charges customers annual
premiums based on their accident history in
the following fashion

• No accident in last 2 years 250
  annual premium
• ? Accidents in each of last 2 years 800 annual
  premium
• ? Accident in only 1 of last 2 years 400 annual
  premium

• Historical statistics
• If a customer had an accident last year then they
  have a 10 chance of having one this year
• If they had no accident last year then they have
  a 3 chance of having one this year.

21
Problem Find the steady-state probability and
the long-run average annual premium paid by the
customer.
Solution approach Construct a Markov chain with
four states (N, N), (N, Y), (Y, N), (Y,Y) where
these indicate (accident last year, accident
this year).
22
State-Transition Network for Insurance Company
.90
.03
.90
.03
.97
.10
.97
.10

• This is an ergodic Markov chain.
• All states communicate (irreducible)
• Each state is recurrent (you will return,
  eventually)
• Each state is aperiodic

23
Solving the SteadyState Equations
m i1
?(N,N) 0.97 ?(N,N) 0.97 ?(Y,N) ?(N,Y)
0.03 ?(N,N) 0.03 ?(Y,N) ?(Y,N) 0.9
?(N,Y) 0.9 ?(Y,Y) ?(N,N) ?(N,Y)?(Y,N)
?(Y,Y) 1
pj å pipij, j 0,,m å pj 1, pj ? 0, ?
j
m j 1
Solution ?(N,N) 0.939, ?(N,Y) 0.029, ?(Y,N)
0.029, ?(Y,Y) 0.003
the long-run average annual premium
is 0.939250 0.029400 0.029400 0.003800
260.5
24
Markov Chain Add-in Matrix
25
Economic Data and Solution
26
Transient Analysis for Insurance Company
27
First Passage Times
Let ?ij expected number of steps to
transition from state i to state j If the
probability that we will eventually visit state j
given that we start in i is less than 1,
then we will have ?ij ?.
For example, in the Gamblers Ruin problem,
?20 ? because there is a positive probability
that we will be absorbed in state 4 given
that we start in state 2.
28
Computations when All States are Recurrent
If the probability of eventually visiting state
j given that we start in i is 1 then the
expected number of steps until we first visit
j is given by
We go from i to r in the first step with
probability pir and it takes mrj steps from r to
j.
It will always take at least one step.
For j fixed, we have linear system in m equations
and m unknowns mij , i 0,1, . . . , m1.
29
First-Passage Analysis for Insurance Company
Suppose that we start in state (N,N) and want to
find the expected number of years until we have
accidents in two consecutive years (Y,Y). This
transition will occur with probability 1,
eventually.
For convenience number the states 0 1
2 3 (N,N) (N,Y) (Y,N) (Y,Y) Then,
?03 1 p00 ?03 p01 ?13 p02?23 ?13
1 p10 ?03 p11 ?13 p12?23 ?23 1
p20 ?03 p21 ?13 p22?23
30
First-Passage Computations
(N, N) (N, Y) (Y, N) (Y, Y)
(N, N) 0.97 0.03 0 0 (N, Y) 0
0 0.90 0.10 (Y, N) 0.97 0.03 0 0 (Y, Y)
0 0 0.90 0.10
0 1 2 3
states
Using P
?03 1 0.97?03 0.03?13 ?13 1
0.9?23 ?23 1 0.97?03 0.03?13
31
First Passage Probabilities
32
Game of Craps
Probability of win Pr 7 or 11 0.167
0.056 0.223 Probability of loss Pr 2, 3, 12
0.028 0.56 0.028 0.112
33
First Passage Probabilities for Craps
34
Absorbing States
An absorbing state is a state j with pjj 1.
Given that we start in state i, we can
calculate the probability of being absorbed in
state j. We essentially performed this
calculation for the Gamblers Ruin problem by
finding
P(n) (pij(n) ) for large n.
But we can use a more efficient analysis like
that used for calculating first passage times.
35
Let 0, 1, . . . , k be transient states and k
1, . . . , m 1 be absorbing states.
Let qij probability of being absorbed in state
j given that we start in
transient state i. Then for each j we have the
following relationship
qij pij ? pirqrj , i 0, 1, . . . , k

k r 0
Go directly to j Go to r and then to j
For fixed j (absorbing state) we have k 1
linear equations in k 1 unknowns, qrj , i 0,
1, . . . , k.
36
Absorbing States Gamblers Ruin
Suppose that we start with 2 and want to
calculate the probability of going broke, i.e.,
of being absorbed in state 0. We know p00 1
and p40 0, thus q20 p20 p21 q10 p22
q20 p23 q30 ( p24 q40) q10 p10
p11 q10 p12 q20 p13 q30 0 q30
p30 p31 q10 p32 q20 p33 q30
0 where P
0 1 2 3 4 0 1 0 0 0 0 1 1-p 0 p 0 0
2 0 1-p 0 p 0 3 0 0 1-p 0 p 4 0 0 0 0 1
37
Solution to Gamblers Ruin Example
Now we have three equations with three
unknowns. Using p 0.75 (probability of winning
a single bet) we have q20 0 0.25 q10
0.75 q30 q10 0.25 0.75 q20 q30 0
0.25 q20 Solving yields q10 0.325, q20
0.1, q30 0.025 (This is consistent with the
values found earlier.)
38
What You Should Know About The Mathematics of
DTMCs

• How to classify states.
• What an ergodic process is.
• How to perform economic analysis.
• How to compute first-time passages.
• How to compute absorbing probabilities.