Chapter 4. Duality Theory

1
Chapter 4. Duality Theory

• Given min cx, s.t. Ax b, x ? 0,
  (called primal problem)
• there exists another LP derived from the primal
  problem using the same data, but with different
  structure (called the dual problem, ????).
• The relation between the primal and the dual
  problem constitutes very important basis for
  understanding the deeper structure of LP
  (compared to systems of linear equations).
• It provides numerous insights in theory and
  algorithms and important ingredients in the
  theory of LP.
• Objective value of the dual problem of a
  feasible dual solution provides lower bound on
  the optimal primal objective value and the dual
  problem can be derived for this purpose.
  However, the text derives it as a special case of
  the Lagrangian dual problem.

2

• Given min cx, s.t. Ax b, x ? 0,
• consider a relaxed problem in which the
  constraints Ax b is eliminated and instead
  included in the objective with penalty p(b-Ax),
  where p is a price vector of the same dimension
  as b.
• Lagrangian function L(x, p) cx p(b
  Ax)
• The problem becomes
• min L(x, p) cx p(b Ax)
• s.t. x ? 0
• Optimal value of this problem for fixed p ? Rm
  is denoted as g(p).
• Suppose x is optimal solution to the LP, then
• g(p) minx ? 0 cx p(b Ax) ? cx
  p(b Ax) cx
• since x is a feasible solution to LP.
• g(p) gives a lower bound on the optimal value of
  the LP.
• Want close lower bound.

3

• Lagrangian dual problem
• max g(p)
• s.t. no constraints on p
• ,where g(p) minx ? 0 cx p(b Ax)
• g(p) minx ? 0 cx p(b Ax)
• pb minx ? 0 ( c pA )x
• minx ? 0 ( c pA )x 0, if c pA ?
  0
• - ?, otherwise
• Hence the dual problem is
• max pb
• s.t. pA ? c

4

• Remark
• (1) If LP has inequality constraints Ax ? b ?
  Ax s b, s ? 0
• ? A -I x s b, x, s ? 0
• ? dual constraints are p A -I ? c
  0
• ? pA ? c, p ? 0
• Or the dual can be derived directly
• min cx, s.t. Ax ? b, x ? 0
• L( x, p) cx p(b Ax) ( Let p ?
  0 )
• g(p) minx ? 0 cx p(b Ax) ? cx
  p(b Ax) ? cx
• max g(p) pb minx ? 0 ( c pA
  )x, s.t. p ? 0
• min x ? 0 ( c pA )x 0, if c pA ? 0
• - ?, otherwise
• Hence the dual problem is
• max pb, s.t. pA ? c, p ? 0

5

• (2) If x are free variables, then
• minx ( c pA )x 0, if c pA 0
• -?, otherwise
• ? dual constraints are pA c

6
4.2 The dual problem
7
PRIMAL minimize maximize DUAL
constraints ? bi ? bi bi ? 0 ? 0 free variables
variables ? 0 ? 0 free ? cj ? cj cj constraints

• Table 4.1 Relation between primal and dual
  variables and constraints
• Dual of a maximization problem can be obtained by
  converting it into an equivalent min problem, and
  then take its dual.

8

• Vector notation
• min cx max pb
• s.t. Ax b ? s.t. pA ? c
• x ? 0
• min cx max pb
• s.t. Ax ? b ? s.t. pA c
• p ? 0
• Thm 4.1 If we transform the dual into an
  equivalent minimization problem and then form its
  dual, we obtain a problem equivalent to the
  original problem.
• ( the dual of the dual is primal, involution
  property)
• For simplicity, we call the min form as primal
  and max form as dual. But any form can be
  considered as primal and the corresponding dual
  can be defined.

9
(Ex 4.1)

• Dual
• ?

Dual ?
?
10

• Ex 4.2 Duals of equivalent LPs are equivalent.
• min cx max pb
• s.t. Ax ? b ? s.t. p ? 0
• x free pA c
• min cx 0s max pb
• s.t. Ax s b ? s.t. p free
• x free, s ? 0 pA c
• -p ? 0
• min cx - cx- max pb
• s.t. Ax - Ax- ? b ? s.t. p ? 0
• x ? 0 pA ? c
• x- ? 0 -pA ? -c

11

• Ex 4.3 Redundant equation can be ignored.
• Consider
• min cx max pb
• s.t. Ax b (feasible) s.t. pA ? c
• x ? 0

12

• Thm 4.2 If we use the following
  transformations, the corresponding duals are
  equivalent, i. e., they are either both
  infeasible or they have the same optimal cost.
• (a) free variable ? difference of two
  nonnegative variables
• (b) inequality ? equality by using
  nonnegative slack var.
• (c) If LP in standard form (feasible) has
  redundant equality constraints, eliminate them.

13
4.3 The duality theorem

• Thm 4.3 ( Weak duality )
• If x is feasible to primal, p is feasible to
  dual, then pb ? cx.
• pf) Let ui pi ( aix bi ), vj ( cj
  pAj ) xj ,
• If x, p feasible to P and D respectively, then
  ui , vj ? 0 ? i, j
• ?i ui p(Ax b) pAx pb
• ?j vj cx pAx
• ? 0 ? ?i ui ?j vj cx pb ? pb ?
  cx ?
• Cor 4.1 If any one of primal, dual is
  unbounded
• ? the other infeasible
• Cor 4.2 If x, p feasible and cx pb, then
  x, p optimal
• pf) cx pb ? cy for all primal feasible
  y. Hence x is optimal to the primal problem.
  Similarly for p. ?

14

• Thm 4.4 (Strong duality)
• If a LP has an optimal solution, so does its
  dual, and the respective optimal costs are equal.
• pf) Get optimal dual solution from the optimal
  basis.
• Suppose have min cx, Ax b, x ? 0, A full
  row rank, and this LP has optimal solution.
• Use simplex to find optimal basis B with B-1b ?
  0, c cBB-1A ? 0
• Let p cBB-1, then pA ? c ? p dual
  feasible
• Also pb cBB-1b cBxB cx, hence p
  optimal dual solution and cx pb
• For general LP, convert it to standard LP with
  full row rank and apply the result, then convert
  the dual to the dual for the original LP. ?

15
?1

• D1

Duals of equivalent problems are equivalent
Equivalent
?2
D2
Duality for standard form problems
Fig 4.1 Proof of the duality theorem for
general LP
16
(D)
Finite optimum Unbounded Infeasible
Finite optimum Possible Impossible Impossible
Unbounded Impossible Impossible Possible
Infeasible Impossible Possible Possible
(P)

• Table 4.2 different possibilities for the
  primal and the dual

17

• Note
• (1) Later, we will show strong duality without
  using simplex method. (see example 4.4 later)
• (2) Optimal dual solution provides certificate
  of optimality
• certificate of optimality is (short) information
  that can be used to check the optimality of a
  given solution in polynomial time.
• ( Two view points 1. Finding the optimal
  solution.
• 2. Proving that a given solution is
  optimal.
• For computational complexity of a problem, the
  two view points usually give the same complexity
  (though not proven yet, P NP ?). Hence,
  researchers were almost sure that LP can be
  solved in polynomial time even before the
  discovery of a polynomial time algorithm for LP.)
• (3) The nine possibilities in Table 4.2 can be
  used importantly in determining the status of
  primal or dual problem.

18

• Thm 4.5 (Complementary slackness)
• Let x and p be feasible solutions to primal and
  dual, then x and p are optimal solutions to the
  respective problems iff
• pi ( aix bi ) 0 for all i,
• ( cj pAj ) xj 0 for all j.
• pf) Define ui pi ( aix bi ), vj (
  cj pAj ) xj
• Then ui , vj ? 0 for feasible x, p
• ?ui ?vj cx pb
• From strong duality, if x, p optimal, then
  cx pb
• Hence ?ui ?vj 0 ? ui 0, vj 0
  ? i, j
• Conversely, if ui vj 0 ? cx pb,
  hence optimal. ?

19

• Note
• (1) CS theorem provides a tool to prove the
  optimality of a given primal solution. Given a
  primal solution, we may identify a dual solution
  that satisfies the CS condition. If the dual
  solution is feasible, both x and y are feasible
  solutions that satisfy CS conditions, hence
  optimal (If nondegenerate primal solution, then
  the system of equations has a unique solution.
  See ex. 4.6).
• (2) CS theorem does not need that x and p be
  basic solutions.
• (3) CS theorem can be used to design algorithms
  for special types of LP (e.g. network problems).
  Also interior point alg tries to find a system
  of nonlinear equations similar to the CS
  conditions.
• (4) See the strict complementary slackness in
  exercise 4.20.

20
Geometric view of optimal dual solution

• (P) min cx
• s.t. aix ? bi , i 1, , m
• x ? Rn , ( assume ais span Rn )
• (D) max pb
• s.t. ?i pi ai c
• p ? 0
• Let I ? 1, , m , I n, such that ai,
  i ? I linearly independent.
• aix bi , i ? I has unique solution xI which
  is a basic solution. Assume that xI is
  nondegenerate, i.e. aix ? bi , for i ? I .
• Let p ? Rm be dual vector.

21

• The conditions that xI and p are optimal are
• (a) aixI ? bi, ? i ( primal
  feasibility )
• (b) pi 0, ? i ? I ( complementary slackness
  )
• (c) ?i pi ai c ( dual feasibility )
• ? ?i?I pi ai c ( unique solution pI )
• (d) p ? 0, ( dual feasibility )

22
c
c
a5
a3
a1
a4
A
c
a1
a2
B
a3
c
a4
a1
a2
c
C
a1
a5
D
a1

• Figure 4.3

23
a2
a3
c
a3
a1
a2
a1
x

• Figure 4.4 degenerate basic feasible solution

24
4.4 Optimal dual var as marginal costs

• Suppose standard LP has nondegenerate optimal
  b.f.s. x and B is the optimal basis, then for
  small d ? Rm,
• xB B-1(bd) gt 0.
• c cBB-1A ? 0 not affected, hence B remains
  optimal basis.
• Objective value is cBB-1(b d) cx pd
  ( p cBB-1 )
• So pi is marginal cost (shadow price) of i-th
  requirement bi.

25
4.5 Dual simplex method

• For standard problem, a basis B gives primal
  solution
• xB B-1b, xN 0 and dual solution p
  cBB-1
• At optimal basis, have xB B-1b ? 0, c pA
  ? 0 ( pA ? c)
• and cBB-1b cBxB pb, hence optimal
• ( have primal feasible, dual feasible solutions
  and objective values are the same).
• Sometimes, it is easy to find dual feasible
  basis. Then, starting from a dual feasible
  basis, try to find the basis which satisfies the
  primal feasibility.
• (Text gives algorithm for tableau form, but
  revised dual simplex algorithm also possible.)

26

• Given tableau, cj ? 0 for all j, xB(i) lt 0
  for some i ? B.
• (1) Find row l with xB(l) lt 0. vi is l-th
  component of B-1Ai
• (2) For i with vi lt 0, find j such that
• cj / vj min i vi lt 0 ci / vi
• (3) Perform pivot ( Aj enters, AB(l) leaves
  basis )
• ( dual feasibility maintained)

27

• Ex 4.7

28

• Note
• (1) row 0 ? row 0 ( l-th row ) ? cj /
  vj
• ? ci ? ci vi ? ( cj / vj ) ( ci ?
  0 from the choice of j )
• For vi gt 0, add vi ? (nonnegative number) to
  row 0 ? ci ? 0
• For vi lt 0, have cj / vj ? ci / vi .
• Hence dual feasibility maintained.
• - cBB-1b ? - cBB-1b (xB(l) ? cj ) / vj
  
• - (cBB-1b (xB(l)? cj ) / vj )
• Objective value increases by (xB(l)? cj ) /
  vj ? 0.
• ( note that xB(l) lt 0, cj ? 0 )
• If cj gt 0, objective value strictly increases.

29

• (2) If cj gt 0, ? j ? N in all iterations,
  objective value strictly increases, hence finite
  termination. (need lexicographic pivoting rule
  for general case)
• At termination
• case a) B-1b ? 0, optimal solution.
• case b) entries v1, , vn of row l are all
  ? 0, then dual is unbounded. Hence primal is
  infeasible.
• ( reasoning
• (1) Find unbounded dual solution.
• Let p cBB-1 be the current dual feasible
  solution ( pA ? c).
• Suppose xB(l) lt 0. Let q - elB-1 (negative
  of l-th row of B-1), then qb - elB-1b gt 0
  and qA - elB-1A ? 0.
• Hence ( p ?q)b ? ? as ? ? ? and p ?q is
  dual feas.
• (2) Current row l xB(l) ?i vixi
• Since vi ? 0, and xi ? 0 for feasibility, but
  xB(l) lt 0
• ? no feasible solution to primal exists
• ? hence dual unbounded. )

30
The geometry of the dual simplex method

• For standard LP, a basis B gives a basic solution
  (not necessarily feasible) xB B-1b, xN 0.
• The same basis provides a dual solution by
  pAB(i) cB(i) , i 1, , m, i. e. pB cB.
• The dual solution p is dual feasible if c
  pA ? 0.
• So, in the dual simplex, we search for dual
  b.f.s. while corresponding primal basic solutions
  are infeasible until primal feasibility (hence
  optimality) is attained.
• ( see Figure 4.5 )
• See example 4.9 for the cases when degeneracy
  exists

31
4.6 Farkas lemma and linear inequalities

• Thm 4.6 Exactly one of the following holds
• (a) There exists some x ? 0 such that Ax b
• (b) There exists some p such that pA ? 0 and
  pb lt 0
• (Note that we used (I) yA c, y ? 0 (II) Ax
  ? 0, cx gt 0 earlier. Rows of A are considered
  as generators of a cone. But, here, columns of A
  are considered as generators of a cone.)
• pf) not both pAx pb ? 0 ( i.e. one
  holds ? the other not hold)
• Text shows (a) ? (b) which is equivalent to (b)
  ? (a)
• (a) ? (b) Consider
• (P) max 0x (D) min pb
• Ax b pA ? 0
• x ? 0
• (a) ? primal infeasible ? dual infeasible or
  unbounded
• but p 0 is dual feasible ? dual unbounded
• ? ? p with pA ? 0 and pb lt 0 ?

32

• Other expression for Farkas lemma.
• Cor 4.3 ) Suppose that any vector p satisfying
  pAi ? 0, i 1, , n also satisfy pb ? 0.
• Then ? x ? 0 such that Ax b.
• See Applications of Farkas lemma to asset
  pricing and separating hyperplane theorem used
  for proving Farkas lemma.

33
The duality theorem revisited

• Proving strong duality using Farkas lemma (not
  using simplex method as earlier)
• (P) min cx (D) max pb
• s.t. Ax ? b pA c
• p ? 0
• Suppose x optimal to (P). Let I i
  aix bi
• Then ? d that satisfies aid ? 0, i ? I, must
  satisfy cd ? 0
• ( i. e. aid ? 0, i ? I, cd lt 0 infeasible
  )
• ( Note that above statement is nothing but
  saying, if x optimal, then there does not exists
  a descent and feasible direction at x .
  Necessary condition for optimality of x )

34

• (continued)
• Otherwise, let y x ?d
• Then ai ( x ?d ) aix ?aid ? aix
  bi , i ? I
• ai ( x ?d ) aix ?aid gt bi , for
  small ? gt 0, i ? I
• Hence y feasible for small ? and cy cx
  ?cd lt cx
• Contradiction to optimality of x.
• By Farkas, ? pi ? 0, i ? I such that c
  ?i?I piai.
• Let pi 0 for i ? I
• ? ? p ? 0 such that pA c and pb
  ?i?I pibi ?i?I piaix cx
• By weak duality, p is optimal dual solution. ?

35
a3
a2
a1
c
p2a2
p1a1
aid?0, i ?I
x
cd lt 0

• Figure 4.2 strong duality theorem