ESI 4313 Operations Research 2

1
ESI 4313Operations Research 2

• Dynamic Programming

2
Dynamic Programming

• Dynamic programming is a technique for solving
  certain types of optimization problems
• The idea is to break up a large, complex problem
  into many smaller, much easier ones
• Usually, this technique can be applied to
  problems in which a sequence of decisions over
  time needs to be made to optimize some criterion

3
Dynamic Programming

• In many cases, solving a problem by dynamic
  programming means
• formulating this problem as a shortest path
  problem in an acyclic network
• The art of dynamic programming lies in how to
  construct this network!

4
ExampleTravel from coast to coast

• You currently live in NYC (1), but plan to move
  to LA (10)
• You will drive
• To save money, you will spend each night of your
  trip at a friends house
• You structure your potential stopovers as
  follows
• In 1 day you can reach Columbus (2), Nashville
  (3), or Louisville (4)
• On the 2nd day, you can reach Kansas City (5),
  Omaha (6), or Dallas (7)
• On the 3rd day, you can reach San Antonio (8) or
  Denver (9)
• On the 4th day, you can reach LA
• To minimize your gas expenses, you are looking
  for the route of minimum length

5
ExampleTravel from coast to coast

• We can classify the cities as follows
• Call all cities that you can be in at the
  beginning of your nth day the Stage n cities
• The idea of solving this problem by dynamic
  programming is to start by solving easy problems
  that will eventually help you solve the entire
  problem
• In particular, we will work backward

6
ExampleTravel from coast to coast

• Denote the distance between city i and city j by
  ci,j
• If city i is a stage t city, we denote the length
  of the shortest path from city i to LA by ft(i)
• Clearly, we would like to find f0(1)

7
ExampleTravel from coast to coast

• First, find the shortest path to LA from each of
  the cities from which you can reach LA in 1 day
  the stage 4 cities
• Note that these problems are trivial, since in
  each case theres only 1 way to go to LA
• More formally,
• f4(8) c8,10
• f4(9) c9,10

8
ExampleTravel from coast to coast

• Then, find the shortest path to LA from each of
  the stage 3 cities
• Note that this means that you should first go to
  a stage 4 city, and then use the shortest path
  from this stage 4 city to LA
• These problems are not as trivial as the first
  ones, but by simply looking at all possible city
  4 problems and the solutions to the first set of
  problems this remains relatively easy

9
ExampleTravel from coast to coast

• From each stage 3 city
• go to a stage 4 city, and then use the shortest
  path from this stage 4 city to LA
• So, for example, f3(5) is equal to
• c5,8 f4(8), or
• c5,9 f4(9)
• Since were interested in the shortest path, we
  have
• f3(5) minc5,8 f4(8) , c5,9 f4(9)

10
ExampleTravel from coast to coast

• Perform the same procedure for the stage 2 cities
• Perform the same procedure for the stage 1 city,
  NYC
• From NYC you should first go to a stage 2 city,
  and then use the shortest path from this stage 2
  city to LA
• We can find the best route from NYC to LA by
  considering all possible stage 2 cities

11
ExampleTravel from coast to coast

• In general, in stage t we are interested in
  finding ft(i) for all stage t cities i
• Using the earlier approach, we can write
• ft(i) minj j is a stage t1 city ci,j
  ft1(j)
• for all stage t cities i

12
Computational efficiency of dynamic programming

• In the example, we could simply enumerate all
  possible paths from NYC to LA
• It is easy to see that there are 3x3x218 paths
• However, suppose that we have more options
• Starting city is again stage 1
• 5 cities in each of 5 stages (stages 2,,6)
• Destination city is stage 7
• Then there are 553,125 paths
• Determining the length of each of these paths
  takes a total of 5x55 15,625 additions and
  3,124 comparisons

13
Computational efficiency of dynamic programming

• How much work is the dynamic programming
  algorithm?
• The stage 6 problems are trivial
• Each of the other problems require
• 5 additions (potential choices for next city to
  visit) and 4 comparisons
• For a total of 4x5x5 5 105 additions and
  4x5x4 4 84 comparisons

14
Characteristics of dynamic programming

• The problem should have stages
• Each stage corresponds to a point at which a
  decision needs to be made
• Each stage should have a number of associated
  states
• The state contains all information that is needed
  to make an optimal decision for the remaining
  problem

15
Characteristics of dynamic programming

• The decision chosen at each stage describes how
  the state at the current stage is transformed in
  the state at the next stage
• The optimal decision at the current state should
  not depend on previously visited states or
  previous decisions
• This is called the principle of optimality

16
Characteristics of dynamic programming

• There must be a recursion that relates the cost
  or reward for stages t, t1, , T to the cost or
  reward for stages t1, t2, , T
• This recursion formalizes the procedure of
  working backwards from the last stage to the
  first stage

17
Dynamic programming formulation

• Stages t 1,,5
• States city
• Decision in each stage
• Choose the stage t1 city to go to
• Dynamic programming recursion
• f4(i) ci,10 for all stage 4 cities i
• ft(i) minj j is a stage t1 city ci,j
  ft1(j)
• for all stage t cities i

18
Dynamic programming without stages

• You must drive from Bloomington to Cleveland
• You are interested in the route that takes the
  least amount of time

19
Dynamic programming without stages
Gary
Toledo
Cleveland
3 hours
3 hours
1 hour
2 hours
3 hours
Dayton
Indianapolis
Columbus
2 hours
3 hours
1 hour
2 hours
2.5 hours
Bloomington
Cincinnati
3 hours
20
Production inventory planning

• Consider the following production inventory
  planning problem for a single item
• Consider a planning period of T periods, and
  assume that
• the demand for the item in each of the periods is
  known
• the initial inventory level is known
• At the start of each period, you must decide how
  many units to produce production capacity is
  limited
• Each periods demand must be met on time
• There is a limited amount of storage space
  available
• The goal is to minimize the total production
  inventory costs over the planning horizon

21
Production inventory planning

• This is a periodic review model
• Denote the demand in period t by dt (t
  1,,T )
• Denote the cost of producing x units in period t
  by ct(x) (often, this function is independent of
  t, i.e., ct(x)c(x) )
• If at the end of period t the inventory level is
  I, a cost of ht(I) is charged (often, these costs
  are independent of t, i.e., ht(I)h(I) )

22
Production inventory planning

• If the production and inventory holding cost
  functions are linear, we can formulate this
  problem as an LP problem (how?)
• Often, the production costs are assumed to have a
  fixed-charge structure
• c(x) 0 if x 0, c(x) a bx if x gt 0
• In that case, we can formulate this problem as a
  mixed-integer LP problem (how?)

23
Production inventory planning

• More generally, we can formulate this problem as
  an NLP problem (how?)
• Production (and inventory) costs are often
  assumed to be concave reflecting economies of
  scale
• What does that mean for the ease of solvability
  of the NLP problem?

24
Production inventory planning

• NLP formulation

25
Production inventory planning

• Dynamic programming provides a solution
  methodology that can be applied for general cost
  functions
• We only need to assume that the units of demand,
  production, and inventory are integers which is
  not unrealistic in many practical situations
• This methodology will be efficient if the
  magnitude of the numbers involved is not too large

26
Production inventory planning

• We must identify
• Stages
• time
• States
• (starting) inventory level
• Decisions
• production quantity
• Recursion
• minimal cost from start of stage t
• Clearly, we are looking for f1(I0)

27
Production inventory planning

• Recursion
• Cost at the beginning of stage T
• Note that you will always want to end up with 0
  inventory so the final periods production will
  be
• So
• (what if dT-I lt 0 or dT-I gt C ???)

28
Production inventory planning

• Recursion
• Cost at the beginning of stage t
• We have to make sure that we have sufficient
  storage capacity, i.e., we need
• We have to make sure that we deliver on time

29
Production inventory planning example

• Example
• T 4 periods
• Demands 1, 4, 2, 3
• Inventory holding costs 0.50 per unit
• Production costs
• fixed setup cost 3
• variable cost 1 per unit
• Production capacity C 5 units
• Inventory capacity B 4 units

30
Production inventory planning

• Initialization T 4
• d4 3
• Cost at the beginning of stage T 4
• I 0 f4(0) c4(3-0) 3 3?1 6
• I 1 f4(1) c4(3-1) 3 2?1 5
• I 2 f4(2) c4(3-2) 3 1?1 4
• I 3 f4(3) c4(3-3) 0
• I 4 f4(4) c4(3-4) ???

31
Production inventory planning

• Next stage t 3
• d3 2
• Cost at the beginning of stage t 3

32
Production inventory planning

• I 0

33
Production inventory planning

• I 0

34
Production inventory planning

• I 2

35
Production inventory planning

• I 2

36
Production inventory planning

• Network representation
• Nodes stage/state combinations (t,I)
• Arcs decisions x
• Arc from node (t,I) corresponding to decision x
  leads to node (t1,Ix-dt)
• Cost of this arc is ct(x) ht(Ix-dt)

37
Resource allocationthe knapsack problem (1)

• Stockco is considering 4 investments
• Investment 1 will yield a NPV of 16K, but
  requires a cash outflow of 5K
• Investment 2 will yield a NPV of 22K, but
  requires a cash outflow of 7K
• Investment 3 will yield a NPV of 12K, but
  requires a cash outflow of 4K
• Investment 4 will yield a NPV of 8K, but
  requires a cash outflow of 3K
• You have a budget of 14K

38
Resource allocationthe knapsack problem (1)

• IP formulation

39
Resource allocationthe knapsack problem (2)

• You are planning an overnight hike, and are
  considering taking 4 items along on your trip
• Item 1 yields a benefit of 16, but weighs 5 lbs
• Item 2 yields a benefit of 22, but weighs 7 lbs
• Item 3 yields a benefit of 12, but weighs 4 lbs
• Item 4 yields a benefit of 8, but weighs 3 lbs
• You do not want to carry more than 14 lbs
• You want to maximize your benefit
• Mathematically, this is the same problem as the
  investment problem!

40
Resource allocationmore general

• Stockco is considering n investments
• Investment n will yield a NPV of rn(dn) when
  dn?1,000 is invested
• You only want to (or can) invest in integer
  multiples of 1,000
• You have a budget of B ? 1,000
• Example
• n 3, B 6
• r1(d1) 7d12 (d1gt0), r1(0) 0
• r2(d2) 3d27 (d2gt0), r2(0) 0
• r3(d3) 4d35 (d3gt0), r3(0) 0

41
Resource allocationmore general

• NLP formulation

42
Resource allocation

• To formulate this problem as a DP problem, we
  must identify
• Stages
• investment categories
• States
• budget available
• Decisions
• investment amount
• Recursion
• maximal return from inv. categories i,,3
• Clearly, we are looking for f1(6)

43
Resource allocation

• Recursion
• Return from investment in category 3 only
• Note that you will always invest all remaining
  budget in category 3 at this stage, i.e., dy

44
Resource allocation

• Recursion
• Return from investment in categories 2 and 3
• These subproblems are a little harder
• y0 f2(0) 0
• y1 f2(1) max(r2(0)f3(1),r2(1)f3(0))
• max(09,100) 10
• y2 f2(2)
• max(r2(0)f3(2),r2(1)f3(1),r2(2)f3(0))
• max(013,109,130) 19

45
Resource allocation

• Network representation
• Nodes stage/state combinations (i,y)
• Arcs decisions d
• Arc from node (i,y) corresponding to decision x
  leads to node (i1,y-d)
• Return of this arc is ri(d)

46
Resource allocationeven more general

• NLP formulation
• Find the DP formulation for this general case

47
Equipment replacement problem

• A company faces the problem of how long a machine
  should be utilized before it should be traded in
  for a new one
• Example
• A new machine costs p1,000, and has a useful
  lifetime of 3 years
• Maintaining a machine during its first 3 years
  costs m160, m280, m3120, respectively
• If a machine is traded in, a salvage value is
  obtained s1800, s2600, and s3500,
  respectively, after the first 3 years

48
Equipment replacement problem

• We currently have a y year old machine
• Find a policy that minimizes total net costs over
  the next 5 years

49
Equipment replacement problem

• To formulate this problem as a DP problem, we
  must identify
• Stages
• time
• States
• age of machine
• Decisions
• keep or trade-in
• Recursion
• minimal net cost after period t
• Clearly, we are looking for f0(y)

50
Equipment replacement problem

• Recursion
• Note that you will always salvage the machine at
  the end of year 5
• Net cost after period 5

51
Equipment replacement problem

• Recursion
• At the end of period t lt 5, you must decide
  whether to keep or trade-in the machine
• If y3, you must trade it in
• If ylt3, you have a real choice

52
Equipment replacement problem

• Network representation
• Nodes stage/state combinations (t,y)
• Arcs decisions x
• Arc from node (t,y) corresponding to decision
• x0 leads to node (t1,y1)
• x1 leads to node (t1,1)
• Return of the arc is
• my when x0
• sy p when x1