EE255/CPS226 Continuous Time Markov Chain (CTMC)

1
EE255/CPS226Continuous Time Markov Chain (CTMC)

• Dept. of Electrical Computer engineering
• Duke University
• Email bbm_at_ee.duke.edu, kst_at_ee.duke.edu

2
Discrete State-Continuous Time Stochastic Process

• A CTMC is characterized by state changes that can
  occur at any arbitrary time (in contrast to a
  DTMC, where state changes can occur only at well
  defined times).
• Index space is un-countable.
• The state space continues to a discrete valued.
• I0,1,2,.. denotes the process state space
• T0, ) is the index space.
• This forms discrete space, cont. time stochastic
  process

3
Continuous Time Markov Chain (CTMC)

• The process qualifies to be
  Markov chain, if for t0 lt t1 lt t2 lt . lt tn lt t ,
  the conditional pmf satisfies
• Process can then be completely described by
• Initial state probability vector for X(t0)
• Transition probabilities.
• Also,

4
Homogenous CTMCs

• is a time-homogenous CTMC iff ,
  
• Or, the conditional pmf satisfies
• Marginal pmf (state probability) is given by
• State prob. May be written as,

5
CTMC Chapman-Kolmogorov Equation

• Proof using total prob. Law
• Since v lt u and invoking Markov property proves
  it.
• The final goal of obtaining pj(t) using
  Chapman-Kolmogorov eq. is cumbersome.
• Instead we are forced to prob. transition rates

6
Prob. Transition Rates

• Lot of math/calculus follows. Define,
• Relating transition probs. rates

7
Calculus steps

• In general,
• Therefore, (1) can be rewritten as,

8
Prob. Transition Matrix

9
Transition Prob. (Homogenous case)

• Transition rates qij(t) and qj(t) are independent
  of t. The Kolmogorov forward equation reduces
  to,
• In the matrix form, (Matrix Q is called the
  infinitesimal generator matrix (or simply
  Generator Matrix)
• Defining,

10
Classification

• Classification of states of a CTMC is identical
  to DTMC
• i is an absorbing state if qij 0 for j I.
  Once the process enters this state, it then never
  leaves this state.
• Steady-state behavior when there are absorbing
  states
• j is a Reachable state from i, if for some tgt0,
  pij(t) gt 0.
• Irreducible CTMC iff every state is reachable
  from every other state. For an irreducible CTMC,
  the following is true
• always exists and are independent of the
  initial state i. In case the limiting
  probabilities pj exist, then,

11
CTMC Steady-state Solution

• Irreducible CTMCs having ve steady-state pj
  values are called recurrent non-null.
• Performance measures may not be computed by
  assigning rewards to all states and computing
  Ereward
• Accumulated reward (over an interval of time)
• Markov chain exhibits memory-less property
• Hj(t) follows EXP( ) distribution.

12
Continuous Time Birth-Death Process

• The CTMC and i0,1,2,
  forms a B-D process, if ?i, i0,1,2,.. and µi,
  i1,2,.. exists, and,
• ?i, Birth rate (gt 0) and µi, Death rate
  (gt 0)

13
Continuous Time Birth-Death Process (contd.)

14
Steady State Equations
These are called balance eqs. Re-arranging
above,
0
15
M/M/1 Queue

• Arrivals follow Poisson distribution, i.e.,
  inter-arrival times are all i.i.d, EXP(?).
• Departures are also Poissonian i.e.,
  inter-departure times are all i.i.d, EXP(µ).
• Models systems such as,
• Packet arrivals at a router port (router switch
  is the server)
• Arrival of tasks at a computer system or a
  scheduler (server computer)
• Customer queues (clerk/cashier is the server)
• Repair workshops, etc. (Repair station or the
  technician is the server)

Poisson arrival Process with rate ?
16
M/M/1 queue (contd.)

• N(t) birth-death proc., ?k? µkµ
• Define, ??/µ (traffic intensity, in Erlangs)
• ? lt 1 (for reasons of stability).

17
M/M/1 queue performance measures

• Server utilization 1- p0
• Expected of customers,
• Above measures can be viewed as expected reward,
• Resulting model is known as the MRM.
• Other measures
• Average queue length (En)
• Average (expected) response time
• Average (expected) wait time et.

18
M/M/1 queue Littles formula

• Response time (R) wait time (W) service time
  (S)
• ER EN/? (Littles formula)
• N (1(t2-t1)2(t3-t2)1(t4-t3)1(t6-t5)2(t7-t6)
  3(t8-t7)2(t9-t8)1(T- t9 ))/T
• (area under the curve)/T (Tt9 t8 t7
  t6 t5 t4 t3 t2 t1)/T
• R ((t3-t1) (t4-t2) (t8-t5) (t9-t6)(T-
  t7))/5
• (Tt9 t8 t7 t6 t5 t4 t3 t2
  t1)/5 (area under the curve)/K
• R.K N.T . Note that, ? K/T . This yields the
  Littles formula.

19
Expected response time

• Analysis has tacitly assumed the FCFS
  scheduling.
• Other scheduling policies may be pre-emptive,
  e.g., RR.
• Round Robin (RR) ? time-slice, then back to the
  end of queue.
• Scope for more than one RR queue.
• The ER formula holds for any scheduling policy
  provided, some conditions are met.

20
Response time distribution

• Assuming FCFS and steady-state conditions
• If there are already n jobs in the system, the
  next job (N1)st will experience a response time
  R SS1S2..SN
• S service time for the (N1)st job S1
  residual service time for job currently
  undergoing service (1).
• Because of the memory-less property, these times
  are EXP( ).
• Hence, for some Nn, the LST of R is,
• Therefore,

21
M/M/k queue

• m-servers together service the queue.

• µ

Poisson arrivals (?)
µ
22
M/M/m Queue Solution
23
M/M/m Queue performance measures

• Average queue length EN rk k

24
M/M/m Queue performance measures

• Server utilization rv M - number of busy
  servers. M may be defined in terms of the number
  of items N in the queue.
• A customer may have to join the queue.

25
Poisson stream behavior

• M/M/m input/output both form Poisson streams.
• m2 case
• Case 1 Two independent queues
• Case 2 M/M/2 case

Two separate Poisson streams
? 2 separate M/M/1 queues
Two separate Poisson streams
Combined Poisson steams
26
Comparative performance

• Case 1 For each M/M/1 queue,
• Case 2 Common queue M/M/2

27
M/M/1/n Queue

• Finite queue size, finite buffer space ? finite
  state space.
• Transient soln?

• Steady State Solution

28
M/M/1/n Queue Performance Measures

• Mean queue length (expected of jobs in the
  system).
• rk k,
• Loss probability
• rn 1, rk 0, k0,1,..,n-1
• Throughput
• rk m , k1,2, ..,n r0 0 (or, rk l ,
  k0,1,2, ..,n-1 rn 0)

29
M/M/1/n Response time distribution

• Response time distribution Job may be rejected
  (or accepted)
• Unconditional (rejected)
• Conditional (accepted)
• Reward assignment for the kth state, response
  time experienced by the tagged task is sum of
  k-service times, each of which is EXP(µ), i.e.,
  k-stage Erlang.
• Unconditional
• Conditional

30
M/M/1/n Example

• Machine failure/repair model
• Machine fails with MTTF 1/? and MTTR 1/µ
• Transient solution
• pUU(t) can also be computed.
• A(t) pUU(t)
• Interval Availability
• Read examples

?
U
D
µ
31
M/M/1/n Example varying birth rate

• Birth rate in state j is ?j (M-j) ? and µj
  µ.
• Multiple clients generating service requests and
  single server. After requesting, client waits
  for response.
• M- parallel component - single repair station

32
M/M/1/n M-components Example

• Various repair possibilities exist
• Each component may have its own repair facility
  (U/D model)
• Single repair station (to reduce cost)
• Single repair station solution may be slow,
  increase the repair rate from µ ? M µ

33
Special cases of Birth-Death Process

• Pure birth processes
• Poisson process
• Software Reliability Growth Model NHPP
• Number of software failures occurring in (0, t
  is N(t), and N(t) is Poisson with, ?(t) abe-bt
  and m(t) EN(t) a(1- e-bt)
• Instantaneous failure intensity, ?(t)
  ba-m(t)
• Transient solution may be found using Laplace
  transforms
• Pure death processes
• No-repairs

34
Non Birth-Death processes

• Not all CTMC may exhibit nearest neighbor
  transittions only.
• Markov chains with arbitrary transition pattern
• Availability modeling
• Performance modeling
• Performability modeling
• Example based

35
Availability model using CTMCs

• Failure/repair model
• repair failure detection/location actual
  repair
• Life time EXP(? ) Detection/location EXP(µ1)
  Repair EXP(µ2)
• The flow equations are

µ1
?
0
1
2
µ2
36
2-component Availability model

• 2-component availability model
• Ass 1-p0
• Failures detection stage takes random time,
  EXP(d)
• Down states are 0 and 1D ? Ass 1- p0- p1D

37
2-components finite coverage

• Coverage factor c
• 1C state is a re-boot (down) state.

38
2-components delayfinite coverage

• Model has detection delaycoverage factor
• Down states are 0, 1C and 1D.

39
Preventive Maintenance example

• Prolonged usage of a component may lead to
  increased failure rate (i.e. IFR situation)
• Hence, life time may be modeled as HypoEXP()
  distribution, say 2-stage Hypo.
• Component is inspected randomly. Time between
  inspections is a random, following EXP(?i).
  Inspection completion time is EXP(µi).
• What does inspection do?
• First stage of life no action
• Second stage of life repair
• That is, preventive maintenance
• State ltstage, faultygt

40
Performance Models

• Example 2-servers with different service times.
• State ltn1, n2gt
• Performance Average no. of jobs in the system,
  En1n2
• Reward rn1, n2 n1n2
• Except for the lt0,0gt, in all other states, viz.,
  ltk,0gt and ltk,1gt, there are k jobs in the system.

41
Markov Modulated Poisson Process (MMPP)

• MMPP is a doubly stochastic Poisson process, such
  that the arrival rate is dependent on the state
  of another CTMC.
• The second CTMC (call it MM) is the modulating
  process having m states in general, Q qij.
• If MM is in state I, then the (first) Poisson
  process has ?i as its arrival rate.
• The Poisson process is a counting process as the
  overall modulated process.
• 2-d state ltPoisson (counting) proc, Modulating
  procgt .

42
MMPP Counting process

?1
?3
?2
?1
?2
?3
?3
?2
43
CTMCs with Absorbing states

• Example 2-component parallel system (perfect
  package)
• Steady-state solution- not meaningful. We need to
  find transient solution.

Initial starting state i.e., p2(0)1
44
2-components, finite coverage

• Faults are covered with coverage factor c.

Initial starting state i.e., p2(0)1
45
Petri Net

• Is a modeling tool used for modeling issues, such
  as, concurrency, synchronization, mutual
  exclusion etc.

Conflict/concurrency PN
Producer/Consumer PN
PN Markings
p1
p2
p1
p1 p2 p3 p4 p5
t1
C
11000
t1
p3
t2
t1
p2
01100
p5
B
t2
t3
t3
p3
11001
p4
t2
10010
p5
p4
t4
t3
t5
t4
t4
n
Denotes a place with finite capacity for tokens
(n)
n
46
PN Definitions

• A PN is a 5-tuple (P, T, A, X, M) . PN is a
  bi-partite graph.
• Arcs T?P or P?T
• An arc may have multiplicity X.
• Rules for enabling a transition t when all
  places p (e P) such p?t exists have one or more
  tokens.
• Firing a transitions an enabled transition may
  fire, if all the places p that have non-null
  t?p arcs have no tokens.
• Markings, aka reachability Graph
• Inhibitor arcs

t1
47
Stochastic PNs

• Random delay (firing time) between enabling and
  firing a transition. Such a transition
  stochastic transition.
• Frequently, firing time EXP( ) distributed.
• Resulting reachability graph of an SPN Markov
  chain.
• Example Poisson process

48
Stochastic PN M/M/1 Queue

• Arrival and service transitions
• M/M/1/n queue
• Reachability graph

49
Generalized SPN (GSPN)

• GSPN is an an enhancement to the SPN.
• GSPN has both immediate and timed transitions
• Firing probabilities GSPN also admits the
  possibility of firing more than one immediate
  transitions.
• Transition priorities by assigning an integer
  priority level to each transition.
• GSPN vanishing and tangible markings
• A vanishing marking involves atleast one
  immediate transition.
• Tangible markings involve only timed transitions.

50
GSPN examples

• Example M/Em/1/n1 queue
• Analysis of a GSPN has 4-steps
• Generate reachability graphs
• Eliminate vanishing markings ? CTMC of tangible
  markings
• Analyze steady state or transient behavior of the
  CTMC
• Evaluate any measures

51
GSPN Examples M/M/i/n queue

• GPSN also allows for place dependent firing rate
• Example M/M/i/n queue
• M/M/n/n queue

52
Stochastic Reward Net (SRN)

• GSPN may be extended further
• associate a reward value with each tangible
  marking,
• Marking dependent guard function
• if (guard T) ? transition may be fired)
• Marking dependent arc multiplicities (typically
  used to fire a flush transition that empties a
  place of all tokens)
• Marking dependent firing rates.

53
SRN fault coverage example

• A work station fault may have imperfect coverage.