INFM 718A / LBSC 705 Information For Decision Making

1
INFM 718A / LBSC 705 Information For Decision
Making

• Lecture 9

2
Outline

• Markov chains
• Gamblers ruin
• Expected time of an event
• Steady-state probabilities

3
Markov Chains

• A Markov chain is a process that evolves from
  state to state at random.
• The probabilities of moving to a state are
  determined solely by the current state.
• Hence, Markov chains are memoryless processes.

4
Example Gamblers Ruin

• p. 444 in the textbook
• You have 1. You play a game of luck, which can
  end in one of two states you have 6, or you
  have 0.
• You bet 1 in each round, and earn 1 if you win,
  and lose your dollar if you lose the round.
• P (win a round) .6 P (lose a round) .4

5
Probability of Reaching a State

• What is the probability of reaching 6, if you
  start with i dollars?
• First lets analyze the situation.
• Then we will use Excel/Solver to calculate the
  w(i)s. (Please refer to the solution on pp.
  444-446 as necessary.)

6
Gamblers Ruin
1
P 0.6
0.6
0.6
0.6
0.6
0
1
2
3
4
5
6
P 0.4
0.4
0.4
0.4
0.4
1
7
Gamblers Ruin

• w(i) P (state 6 will be rached state I is
  occupied now)
• w(i) 0.6 w(i1) 0.4 w(i-1)
• w(0) 0
• w(6) 1
• w(1) 0.6 w(2) 0.4 w(0)
• w(2) 0.6 w(3) 0.4 w(1)
• w(3) 0.6 w(4) 0.4 w(2)
• w(4) 0.6 w(5) 0.4 w(3)
• w(5) 0.6 w(6) 0.4 w(4)

8
Gamblers Ruin

• w(0) 0
• w(6) 1
• - 0.6 w(2) w(1) - 0.4 w(0) 0
• - 0.6 w(3) w(2) - 0.4 w(1) 0
• - 0.6 w(4) w(3) - 0.4 w(2) 0
• - 0.6 w(5) w(4) - 0.4 w(3) 0
• - 0.6 w(6) w(5) - 0.4 w(4) 0

9
Expected Time of an Event

• Please refer to Problem B on pp. 447-449.

4 T(B)
2
3
1/3
1/3
1/2
1/3
1/3
4 T(A)
1 T(A)
1/3
1/3
1/2
1 T(C)
5 T(C)
5 T(B)
10
Expected Time of an Event

• m(A) ET(A), m(B) ET(B), m(C) ET(C)
• m(A) (1/2)4 ET(B) (1/2)1 ET(C)
• (1/2)4 m(B) (1/2)1 m(C)
• m(B) (1/3)(2) (1/3)4 m(A) (1/3)5
  m(C)
• m(B) (1/3)(3) (1/3)1 m(A) (1/3)5
  m(B)

11
Expected Time of an Event

• m(A) - (1/2)m(B) - (1/2)m(C) 5/2
• - (1/3)m(A) m(B) - (1/3)m(C) 11/3
• - (1/3)m(A) - (1/3)m(B) m(C) 9/3
• Solve in Excel for m(A), m(B), and m(C)

12
Steady-state Probabilities

• What are the probabilities of being in each state
  over the long run (i.e., when those probabilities
  become steady.)
• Using multiple transitions (refer to pp.
  450-453).
• Using direct calculation (refer to p. 454).
• Using flux (refer to pp. 455-456).