ESI 6912: Dynamic Programming

1
ESI 6912Dynamic Programming

• Stochastic Dynamic Programming
• Equipment Inspection and Replacement problems

2
Uncertainty

• So far we have assumed that all problem data are
  known in advance.
• In particular, this means that
• All costs and profits are known
• The consequences of any action are known
• Stochastic Dynamic Programming allows for
  uncertainties.

3
Stochastic Dynamic Programming

• To illustrate, we will look at three different
  types of problems
• Resource allocation problem
• Equipment replacement problems
• Equipment inspection and replacement problem

4
Resource allocation problem

• You are planning the allocation of a budget of X
  dollars to N winner-take-all presidential
  primaries.
• If xi dollars are allocated to primary n, the
  candidate wins with probability p(xi) and gets
  all vi delegates.
• You would like to maximize the probability that
  the candidates delegate total is at least V.

5
Resource allocationoptimal value function

• Optimal value function
• f(n,x,v) the maximum probability of attaining a
  total number of delegates of at least v from
  primaries n,,N, when a budget of x dollars is
  available.
• Boundary condition
• f(N1,x,v)1 if v?0, 0 otherwise
• f(n,x,v)1 for all v?0
• We wish to determine f(1,X,V).

6
Resource allocationrecurrence relation

• Recurrence relation

7
Stochastic Equipment replacement

• Recall the deterministic statement of a basic
  equipment replacement problem.
• Consider a type of machine that deteriorates with
  age, and the decision to replace it.
• We have a need for such a machine during each of
  the next N time periods.
• The problem is to decide when (or if) to replace
  an existing machine by a new one so as to
  minimize the total costs.

8
Stochastic Equipment replacement

• In the time-invariant deterministic case, we had
  the following costs
• ci cost of operating a machine that is i
  periods old at the start of a period for 1 period
  
• p price of a new machine
• ti trade-in value received when a machine that
  is of age i at the beginning of a period is
  traded for a new machine
• si salvage value received for a machine that
  has just turned age i at the end of period N
• a age of machine at start of 1st year

9
Stochastic Equipment replacement

• Now instead assume that the following
  uncertainties exist.
• The cost of operating an i periods old machine
  for 1 year is a random variable, say Ci.
• A machine that is i periods old at the beginning
  of a period suffers, with probability qi, a
  catastrophic failure which we assume happens at
  the end of the period.

10
Stochastic Equipment replacement

• This has the following implications for the
  trade-in and salvage value definitions
• ti trade-in value received for a machine that
  has just turned age i and is in working order
• si salvage value received for a machine that
  has just turned age i and is in working order at
  the end of period N

11
Stochastic Equipment replacement

• In addition, we define the trade-in and salvage
  values of a machine in failed condition
• ui trade-in value received for a machine that
  has just turned age i and is in failed condition
• vi salvage value received for a machine that
  has just turned age i and is in failed condition
  at the end of period N

12
Stochastic Equipment replacement

• The uncertainty in the problem has the
  consequence that the total cost that is incurred
  for a given replacement policy is stochastic.
• We are interested in finding the policy that
  minimizes the expected costs.

13
Stochastic Equipment replacement states and
decisions

• State
• (k,i) (current period, age of machine)
• k1,,N1 i0,1,,k-1,ak-1
• k is also the stage variable
• Initial state (1,a)
• Ending states (N,i), i0,1,,N,aN-1
• Decisions
• buy or keep

14
Stochastic Equipment replacement optimal value
function

• Optimal value function
• f(k,i) the minimum expected cost of owning a
  machine in periods k,,N, starting year k with a
  machine that is i periods old and in working
  order
• We wish to find f(1,a)

15
Stochastic Equipment replacement recurrence
relation

• Recurrence relation

16
Stochastic Equipment replacement recurrence
relation

• Boundary condition

17
Stochastic Equipment replacement running time

• Recursive fixing applied to this problem has the
  same running time as for the original problem!
• Note that the expected operating costs can be
  pre-computed!
• The only fundamental change in the formulation is
  due to the probability of failure!

18
Stochastic Equipment replacement extension 1

• Now suppose that, in case of catastrophic
  failure, we can
• either replace the machine (as before), or
• pay an amount wi, if the machine has just turned
  age i, to restore the machine to working order

19
Stochastic Equipment replacement recurrence
relation

• Recurrence relation

20
Stochastic Equipment replacement recurrence
relation

• The boundary condition remains the same
• Note the qualitative difference between the
  buy/keep and the replace/restore decisions
• the former is made before the random failure
  event takes place
• The latter is made after the random failure
  event takes place

21
Stochastic Equipment replacement extension 2a

• Now suppose that, in case of catastrophic
  failure, we can
• either replace the machine (as before), or
• pay a random amount Wi, if the machine has just
  turned age i, to restore the machine to working
  order
• What is the effect of this on the DP formulation?

22
Stochastic Equipment replacement extension 2b

• In addition, we can have a failed machine
  inspected before the replace/restore decision is
  made.
• Inspection costs are b
• Inspection reveals the actual cost of restoration
• Modify the DP formulation to incorporate this
  possibility.

23
An equipment inspection model

• Consider a system containing multiple unreliable
  parts.
• If any of the parts fails, the system does not
  function.
• Only by inspecting a part can we determine
  whether its condition is good or failed.
• We can also decide to replace a part.

24
An equipment inspection model

• A part just inspected and found good is as good
  as new (and has age 0).
• The age of a part is the number of periods since
  the part was found to be good.
• If part n is good at the start of a period, it
  will still be good at the start of the next
  period with probability pn.
• We are interested in finding an inspection and
  replacement policy that maximizes the expected
  number of periods during which the system is
  working.

25
An equipment inspection model

• Inspection and Replacement
• It takes multiple periods to inspect or replace a
  part
• Time can be saved by inspecting or replacing
  multiple parts simultaneously
• Inspection and replacement cannot take place at
  the same time
• While inspection or replacement is taking place,
  the system is not working
• During inspection or replacement, no part ages

26
An equipment inspection model

• We will consider the case of 2 parts.
• Inspection
• It takes cn periods to inspect part n
• It takes c12ltc1c2 periods to inspect both parts
• Replacement
• It takes rn periods to inspect part n
• It takes r12ltr1r2 periods to inspect both parts

27
Equipment inspectionoptimal value function

• Optimal value function
• f(i,j,k) the maximum expected number of periods
  during i,,N that the system is working, given
  that we start in period i with part 1 being age j
  and part 2 being age k
• We wish to find f(1,j0,k0)

28
Equipment inspectionoptimal value function

• In addition, it will be convenient to define the
  following functions
• Fn(i,k) the maximum expected number of
  remaining periods during which the system is
  working given that we start period i with part n
  just found failed, and the other part of age k
  (n1,2)

29
Equipment inspectionrecurrence relation

• Given that we are in any state, say (i,j,k), we
  have 7 options
• Continue as is
• Inspect part 1, part 2, or both
• Replace part 1, part 2, or both
• Continue as is

30
Equipment inspectionrecurrence relation

• Inspect part 1
• Inspect part 2
• Inspect both parts

31
Equipment inspectionrecurrence relation

• Replace part 1
• Replace part 2
• Replace both parts
• Clearly, we let S(i,j,k) be the maximum of the 7
  values.

32
Equipment inspectionrecurrence relation

• For the auxiliary functions

33
Equipment inspectionrecurrence relation

• Note that the expressions for S(i,0,k), S(i,j,0),
  and S(i,0,0), as well as Fn(i,0) (n1,2) can be
  simplified
• not all options are of interest in these cases

34
Equipment inspectionboundary conditions

• The boundary conditions are
• S(i,j,k) 0 whenever igtN
• Similarly for the auxiliary functions
• F1(i,k) F2(i,j) 0 whenever igtN