Chapter 4 Geometry of Linear Programming

1
Chapter 4Geometry of Linear Programming

• There are strong relationships between the
  geometrical and algebraic features of LP problems
• Convenient to examine this aspect in two
  dimensions (n2) and try to extrapolate to higher
  dimensions (be careful!)

2
4.1 Example
z max z 4x1 3x2
3
Feasible Region

• First constraint

• Corresponding hyperplane

4
(No Transcript)
5
x
2
40
30
20

10
x
1
10
20
30
40
6
(No Transcript)
7
(No Transcript)
8
x115
9

• Objective function
• f(x) z 4x1 3x2
• Hence
• x2 (z -4x1)/3
• so that for a given value of z the level curve is
  a straight line with slope -4/3.
• We can plot it for various values of z.
• We can identify the (x1,x2) pair yielding the
  largest feasible value for z.

10
x
2
z 4x1 3x2
x2 (z - 4x1)/3
40
30
20
10
x
1
10
20
30
40
11
z 4x1 3x2
x2 (z - 4x1)/3
12
z 4x1 3x2
x2 (z - 4x1)/3
13
z 4x1 3x2
x2 (z - 4x1)/3
14
(No Transcript)
15
Important Observations

• The graphical method is used to identify the
  hyperplanes specifying the optimal solution.
• The optimal solution itself is determined by
  solving the respective equations.
• Dont be tempted to read the optimal solution
  directly from the graph!
• The optimal solution in this example is an
  extreme point of the feasible region.

16
Questions????

• What guarantee is there that an optimal solution
  exists?
• In fact, is there any a priori guarantee that a
  feasible solution exists?
• Could there be more that one optimal solution?

17
4.2 Multiple Optimal Solutions
18
No Feasible Solutions
19
Unbounded Feasible Region
x
2
Direction of
40
increasing z
30
20
10
x
1
10
20
30
40
z200
z120
z160
20
(No Transcript)
21
Feasible region not closed
22
Feasible Region Not Closed
x
2
40
30
Corner point at (10,20) is no longer feasible
20
10
x
1
10
20
30
40
23
4.4 Geometry in Higher Dimensions

• Time Out!!!
• We need the first section of Appendix C

24
Appendix CConvex Sets and Functions

• C.1 Convex Sets
• C.1.1 Definition
• Given a collection of points x(1),...,x(k) in Rn,
  a convex combination of these points is a point w
  such that
• w a1x(1) a2x(2) ... akx(k)
• where 0ai1 and Siai1.

25

• c.1.2 Definition
• The line segment joining two points p,q in Rn is
  the collection of all points x such that
• x lp (1-l)q
• for some 0l1.

26
NILN
p
w lp (1-l)q
q
27

• c.1.3. definition
• A subset C of Rn is convex if for every pair of
  points (p,q) in C and any 0l1, the point
• w lp (1-l)q
• is also in C.
• Namely, for any pair of points (p,q) in C, the
  line segment connecting these points is in C.

28
(No Transcript)
29

• C.1.5 Theorem
• The intersection of any finite number of convex
  sets in a convex set.

Intersection
30

• C.1.6 Definition
• A set of points H ? Rn satisfying a linear
  equation of the form
• a1x1 a2x2 ... anxn b
• for a ? (0,0,0,...,0), is a hyperplane.
• Observation
• Such hyperplanes are of dimension n-1. (why?)

31
Example (not in the notes)
x2
3
x1 x2 3
2
1
x1
1
2
3
32

• C.1.7 Definition
• The two closed half-spaces of the hyperplane
  defined by
• a1x1 a2x2 ... anxn b
• are the set defined by
• a1x1 a2x2 ... anxn b (positive half
  space)
• and
• a1x1 a2x2 ... anxn b (negative half
  space)

33
Example (not in the notes)
x2
3
positive half space
2
negative half-space
1
x1 x2 3
x1
1
2
3
34

• C.1.8 Theorem
• Hyperplanes and their half-spaces are convex
  sets.
• C.1.9 Definition
• A convex polytope is a set that can be expressed
  as the intersection of a finite number of closed
  half-spaces.

35

• C.1.9 Definition
• A polyhedron is a non-empty bounded polytope.

36

• C.1.10 Definition
• A point x of a convex set C is said to be an
  extreme point if it cannot be expressed as a
  convex combination of other points in C.
• More specifically, there are no points y and z in
  C (different from x) such that x lies on the line
  segment connecting these points.

37
examples(not in notes)

• Corner points are extreme points
• Boundary points
  are extreme points

38
examples(not in notes)

• Corner points are extreme points
• Boundary points
  are extreme points

39
Linear Combination(Not in Lecture Notes)

• A linear combination is similar to a convex
  combination, except that the coefficients are not
  restricted to the interval 0,1. Thus, formally

40

• Definition
• A vector x in Rn is said to be a linear
  combination of vectors x(1),...,x(s) in Rn if
  and only if there are scalars a1,...,as - not
  all zeros - such that
• x S t1,...,s atx(t)

41
Example

• x(3,2,1) is a linear combination of
• x(1) (1,0,0)
• x(2) (0,1,0)
• x(3) (0,0,1)
• using the coefficients a13, a22 and a31.
• y(9,4,1) is not a linear combination of
• x(1) (3,1,0)
• x(2) (2,4,0)
• x(3) (4,3,0)
• Why?

42
Geometrically
Linear Combination
a
la (1-l)b l unrestricted
a
b
Convex Combination
b
la (1-l)b 0 lt l lt 1
43
Set of all convex combinations of a and b.
a
b
Linear combinations of these two vectors span the
entire plane.
44
Linear Independence

• A collection of vectors x(1),...,x(s) in Rn are
  said to be linearly independent if no vector in
  this collection can be expressed as a linear
  combination of the other vectors in this
  collection.
• This means that if
• S t1,...,s atx(t) (0,...,0)
• then at0 for t1,2,...,s.
• Try to show this equivalence on your own!

45
Example

• The vectors (1,0,0) , (0,1,0), (0,0,1) are
  linearly independent.
• The vectors (2,4,3) , (1,2,3), (1,2,0) are not
  linearly independent.

46
a
b
o
Linearly independent
b
o
a
Linearly dependent
47
Back to Chapter 4 .....4.4 Geometry in Higher
Dimensions
48

• The region of contact between the optimal
  hyperplane of the objective function and the
  polytope of the feasible region is either an
  extreme point or a face of the polytope.

(NILN)
Objective function
Objective function
Feasible region
Feasible region
49

• 4.4.1 Theorem
• The set of feasible solutions of the standard LP
  problem is a convex polytope

50

• Proof
• Follows directly from the definition of a convex
  polytope, i.e. a convex polytope is the
  intersection of finitely many half-spaces .

51

• 4.4.2 Theorem
• If a linear programming problem has exactly one
  optimal solution, then this solution must be an
  extreme point of the feasible region.
• Proof
• We shall prove this theorem by contradiction!!!

52

• So contrary to the theorem assume that the
  problem has exactly one optimal solution, call it
  x, and that x is not an extreme point of the
  feasible region.

53

• This means that there are two distinct feasible
  solutions, say x and x, and a scalar l, 0ltllt1,
  such that
• x lx (1-l)x

(NILN)
x
x
x
54

• If we rewrite the objective function in terms of
  x and x rather than x, we obtain
• f(x) f(lx (1-l)x)
• hence

lf(x) (1-l)f(x)
(4.13)
55

• Now, because 0ltllt1, there are only three cases to
  consider with regard to the relationship between
  f(x), f(x) and f(x)
• 1. f(x) lt f(x) lt f(x)
• 2. f(x) lt f(x) lt f(x)
• 3. f(x) f(x) f(x)
• But since x is an optimal solution, the first two
  cases are impossible (why?).
• Thus, the third case must be true.
• But this contradicts the assertion that x is the
  only optimal solution to the problem.

f(x) lf(x) (1-l)f(x)
56

• On your own, prove the following
• 4.4.3 Lemma
• If the LP has more than one optimal solution, it
  must have infinitely many optimal solutions.
  Furthermore, the set of optimal solutions is
  convex.

57

• 4.4.5 Proposition
• If a linear programming problem has an
  optimal solution, then at least one optimal
  solution is an extreme point of the feasible
  region.
• Observation
• This result does not say that all the optimal
  solutions are extreme points.

58

• This result is so important that we discuss it
  under the header
• 4.5 The Fundamental Theorem of Linear Programming

59

• 4.5.1 Canonical Form

As in the standard format, bi0 for all i.
60

• 4.5.2 Corollary
• The canonical form has at least one feasible
  solution, namely
• x (0,0,0,...,0,b1,b2,...,bm)
• Note
• This solution is obtained by
• Setting all the original variables to zero
• setting the new variables to the respective
  right-hand side values.
• The new variables are called slack variables

(n zeros)
61
4.5.3 DefinitionBasic Feasible Solutions

• Bla, bla , bla ....................
• Given a system of m linear equations with k
  variables such that k gt m
• Select m columns whose coefficients are linearly
  independent.
• Solve the system comprising these columns and the
  right hand side.

62

• Set the other k-m variables to zero.
• Any solution of this nature is called a basic
  solution.

63
(NILN)
k
m
m
m
64

• A basic feasible solution is a basic solution
  satisfying the non-negativity constraints xj 0,
  for all i.

65
4.5.4. Example
66
canonical form

• Trivial basic feasible solution x(0,0,4,3)

67
Other basic feasible solutions ?

• Suppose we select x2 and x3 to be basic Then,
  the reduced system is

This yields the basic feasible solution x
(0,3/2,5/2,0)
68

• If we select x1 and x2 to be the basic variables,
  the reduced system is

• This yields the basic feasible solution
• x(5/3,2/3,0,0).

69

• If we select x1 and x3 as basic variable, the
  reduced system is

• This yields the basic solution x(3,0,-2,0).
  This solution is not feasible.

70
Next Result

• Relation between the geometric and algebraic
  representations of LP problems

Geometry
Algebra
(NILN)
-
-
-
-
-
Basic feasible solutions
Extreme Points
71
4.5.5 Theorem

• Consider the LP problem

(4.17)
72

• Where
• kgtm
• bio, for all i
• and the coefficient matrix has m linearly
  independent columns.
• Then,
• A vector x in Rn is an extreme point of the
  feasible region of this problem if, and only if,
  x is a basic feasible solution of this problem.
• Proof In the Lecture Notes (NE).

73
4.5.6 The Fundamental Theorm of Linear
Programming

• Consider the LP problem featured in Theorem
  4.5.5.
• If this problem has a feasible solution then it
  must have a basic feasible solution.
• If this problem has an optimal solution then it
  must have an optimal basic feasible solution.
• Proof In the Lecture Notes (NE).

74

• 4.5.7 Corollary
• If the set determined by (4.17) is not empty then
  it must have at least one extreme point.
• 4.5.8 Corollary
• The convex set determined by (4.17) possesses at
  most a finite number of extreme points (Can you
  suggest an upper bound?)

75

• 4.5 9 Corollary
• If the linear programming problem determined by
  (4.16)-(4.17) possesses a finite optimal
  solution, then there is a finite optimal solution
  which is an extreme point of the feasible
  solution.

76

• 4.5.10 Corollary
• If the feasible region determined by (4.17) is
  not empty and bounded, then the feasible region
  is a polyhedron.
• 4.5.11 Corollary
• At least one of the points that optimizes a
  linear objective function over a polyhedron is an
  extreme point of the polyhedron.

77

• Direct Proof (utilising the fact that the
  feasible region is a polyhedron).
• Let x(1),...,x(s) be the set of extreme points
  of the feasible region (note x(q) is a
  k-vector).
• Thus, any point in the feasible region can be
  expressed as a convex combination of these
  points, namely
• x S t1,...,s atx(t)
• where
• S t1,...,s at 1 , at 0, t1,2,...,s.

78

• Thus, the objective function can be rewritten as
  follows
• z(x) S j1,..,kcjxj S j1,..,k cjS
  t1,..,s atxj(t)j
• S t1,..,satS j1,..,k cjxj(t)
• S t1,..,satz(t) (4.41)
• where
• z(t) S j1,..,k cjxj(t) , t1,...,s.

79

• Because at 0, and S j1,...,kaj 1, it follows
  from (4.41) that
• max z(t) t1,...,s z min z(t)
  t1,...,s (4.43)
• where z S j1,...,kcjxj.
• Since x is an arbitrary feasible solution, (4.43)
  entails that at least one extreme point is an
  optimal solution (regardless of what opt is).

80
A Subtlety (NILN)

• Given a list of numbers (y1,...,yp) and a list of
  coefficients (a1,...,ap) each in the unit
  interval 0,1 and their sum is equal to 1, we
  have
• max y1,...,yp Sj1,...,paj yj min
  y1,...,yp
• In words, any convex combination of a collection
  of numbers is in the interval specified by the
  smallest and largest elements of the collection

81
b max y1,...,yp
Any convex combination of y1,...,yp must lie in
the interval a,b
a min y1,...,yp

• Try to prove it on your own!

82
4.6 Solution Strategies

• Bottom Line
• Given a LP with an optimal solution, at least one
  of the optimal solutions is an extreme point of
  the feasible region.
• So how about solving the problem by enumerating
  all the extreme points of the feasible region?
  

83

• Since each extreme point of the feasible region
  of the standard problem is a basic solution of
  the system linear constraints having nm
  variables and m (functional) constraints, it
  follows that there are at most

84

• Since each extreme point of the feasible region
  of the standard problem is a basic solution of
  the system linear constraints having nm
  variables and m (functional) constraints, it
  follows that there are at most

extreme points.
85

• For large n and m this yields a very large number
  (Curse of Dimensionality!), eg
• for nm50, this yields 1029.

86
Most popular Methods

• Simplex, Dantzig, 1940s
• Visits only extreme points
• Interior Point Karmarkar, 1980s
• Moves from the (relative) interior of the region
  or faces, towards the optimal solution.
• In this years version of 620-261 we shall focus
  on the Simplex Method.