An Introduction to Markov Chains

1
An Introduction to Markov Chains

2
Homer and Marge repeatedly play a gambling game.
Each time they play,the probability that Homer
wins is 0.4, and the probability that Homer loses
is 0.6
A Drunkards Walk
3
0 1 2 3 4
0 1 2 3 4
P(Homer wins) .4 P(Homer loses) .6
Homer and Marge both start with 2
4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
A Markov Chain is a mathematical model for a
process which moves step by step through various
states.
In a Markov chain, the probability that the
process moves from any given state to any other
particular state is always the same, regardless
of the history of the process.
13
A Markov chain consists of states and transition
probabilities.
Each transition probability is the probability of
moving from one state to another in one step.
The transition probabilities are independent of
the past, and depend only on the two states
involved. The matrix of transition probabilities
is called the transition matrix.
14
0 1 2 3 4
P(Homer wins) .4 P(Homer loses) .6
Homer and Marge both start with 2
15
If P is the transition matrix for a Markov
Chain, then the nth power of P gives the
probabilities of going from state to state in
exactly n steps.
16
If the vector v represents the initial state,
then the probabilities of winding up in the
various states in exactly n steps are exactly v
times the nth power of P .
17
When they both start with 2, the probability
that Homer is ruined is 9/13.
If Homer starts with x and Marge starts with
N-x, and P(Homer wins) p, P(Homer loses) q,
then the probability Homer is ruined is
18
Suppose you be on red in roulette. P(win)
18/38 9/19 P(lose) 10/19.
Suppose you and the house each have 10
Now suppose you have 10 and the house has 20
19
Now suppose you and the house each have 100.
20
Andrei Markov (1856-1922)
Paul Eherenfest Diffusion model, early
1900sStatistical interpretation of the second
law of thermodynamics The entropy of a closed
system can only increase.Proposed the Urn
Model to explain diffusion.
Albert Einstein, 1905Realized Brownian motion
would provide a magnifying glass into the world
of the atom. Brownian motion has been
extensively modeled by Markov Chains
21
(No Transcript)
22
Particles are separated by a semi-permeable
membrane, which they can pass through in either
direction.
Suppose that there are N black particles inside
the membrane, and N white particles outside the
membrane. Each second, one random molecule goes
from outside the membrane to inside, and vice
versa.
There are N1 states, given by the number of
white molecules inside.
Osmosis
23
(No Transcript)
24
0 1 2 3 4 5
5 molecules
25
N molecules
26
N molecules
27
If this process runs for a while, an interesting
question is How much time, on average, is the
process in each state?
A Markov chain with transition matrix P is said
to be regular if some power of P has all positive
entries for some n. In a regular Markov chain,
it is possible to get from any state to any other
state in n steps.
28
The Markov chain for our osmosis process is
regular. Even starting with all black particles
inside, if a white particle entered at every
step, then the process would pass from zero white
inside through all possible states.
29
For a regular Markov chain, the amount of time
the process spends in each state is given by the
fixed probability vector, which is the vector a
such that Pa a. Moreover, for any probability
vector w,
No matter what the starting state, if the process
runs for a long time, the probability of being in
a given state is given by a.
30
In the long run, the fraction of time the process
spends in each state is given by the fixed
probability vector.
31
For N particles, the fixed vector is
1
1 1
1 2 1
1 3 3 1 1 4
6 4 1 1 5 10 10
5 1 1 6 15 20 15 6
1
32
Fixed vectors
N 4 (1/70, 16/70, 36/70, 16/70, 1/70)
33
(No Transcript)
34
(No Transcript)
35
Now suppose 500 molecules
The percent of the time that there are between
225 and 275 black molecules inside is 0.999.
The percent of the time that there are either
fewer than 100 black or more than 400 black
molecules inside is
36
If the universe is 15 billion years old, the
average amount of time that a system with 500
molecules will have fewer than 100 black or more
than 400 black molecules inside the membrane is