Optimization

1
Optimization

• Assoc. Prof. Dr. Pelin Gündes
• gundesbakir_at_yahoo.com

2
Optimization

• Basic Information
• Instructor Assoc. Professor Pelin Gundes
  (http//atlas.cc.itu.edu.tr/gundes/)
• E-mail gundesbakir_at_yahoo.com
• Office Hours TBD by email appointment
• Website http//atlas.cc.itu.edu.tr/gundes/teachi
  ng/Optimization.htm
• Lecture Time Wednesday 1300 - 1600
• Lecture Venue M 2180

3
Optimization literature

• Textbooks
• Nocedal J. and Wright S.J., Numerical
  Optimization, Springer Series in Operations
  Research, Springer, 636 pp, 1999.
• Spall J.C., Introduction to Stochastic Search and
  Optimization, Estimation, Simulation and Control,
  Wiley, 595 pp, 2003.
• Chong E.K.P. and Zak S.H., An Introduction to
  Optimization, Second Edition, John Wiley Sons,
  New York, 476 pp, 2001.
• Rao S.S., Engineering Optimization - Theory and
  Practice, John Wiley Sons, New York, 903 pp,
  1996.
• Gill P.E., Murray W. and Wright M.H., Practical
  Optimization, Elsevier, 401 pp., 2004.
• Goldberg D.E., Genetic Algorithms in Search,
  Optimization and Machine Learning, Addison
  Wesley, Reading, Mass., 1989.
• S. Boyd and L. Vandenberghe, Convex Optimization,
  Cambridge University Press, 2004.(available at
  http//www.stanford.edu/boyd/cvxbook/)

4
Optimization literature

• Journals
• Engineering Optimization
• ASME Journal of Mechnical Design
• AIAA Journal
• ASCE Journal of Structural Engineering
• Computers and Structures
• International Journal for Numerical Methods in
  Engineering
• Structural Optimization
• Journal of Optimization Theory and Applications
• Computers and Operations Research
• Operations Research and Management Science

5
Optimization

• Course Schedule
• Introduction to Optimization
• Classical Optimization Techniques
• Linear programming and the Simplex method
• Nonlinear programming-One Dimensional
  Minimization Methods
• Nonlinear programming-Unconstrained Optimization
  Techniques
• Nonlinear programming-Constrained Optimization
  Techniques
• Global Optimization Methods-Genetic algorithms
• Global Optimization Methods-Simulated Annealing
• Global Optimization Methods- Coupled Local
  Minimizers

6
Optimization

• Course Prerequisite
• Familiarity with MATLAB, if you are not familiar
  with MATLAB, please visit
• http//www.ece.ust.hk/palomar/cours
  es/ELEC692Q/lecture200620-20cvx/matlab_crashcou
  rse.pdf
• http//www.ece.ust.hk/palomar/cours
  es/ELEC692Q/lecture200620-20cvx/official_gettin
  g_started.pdf

7
Optimization

• 70 attendance is required!
• Grading
• Homeworks 15
• Mid-term projects 40
• Final Project 45

8
Optimization

• There will also be lab sessions for MATLAB
  exercises!

9
1. Introduction

• Optimization is the act of obtaining the best
  result under given circumstances.
• Optimization can be defined as the process of
  finding the conditions that give the maximum or
  minimum of a function.
• The optimum seeking methods are also known as
  mathematical programming techniques and are
  generally studied as a part of operations
  research.
• Operations research is a branch of mathematics
  concerned with the application of scientific
  methods and techniques to decision making
  problems and with establishing the best or
  optimal solutions.

10
1. Introduction

• Operations research (in the UK) or operational
  research (OR) (in the US) or yöneylem arastirmasi
  (in Turkish) is an interdisciplinary branch of
  mathematics which uses methods like
• mathematical modeling
• statistics
• algorithms to arrive at optimal or good decisions
  in complex problems which are concerned with
  optimizing the maxima (profit, faster assembly
  line, greater crop yield, higher bandwidth, etc)
  or minima (cost loss, lowering of risk, etc) of
  some objective function.
• The eventual intention behind using operations
  research is to elicit a best possible solution to
  a problem mathematically, which improves or
  optimizes the performance of the system.

11
1. Introduction
12
1. Introduction

• Historical development
• Isaac Newton (1642-1727)
• (The development of differential calculus
• methods of optimization)
• Joseph-Louis Lagrange (1736-1813)
• (Calculus of variations, minimization of
  functionals,
• method of optimization for constrained
  problems)
• Augustin-Louis Cauchy (1789-1857)
• (Solution by direct substitution, steepest
• descent method for unconstrained
  optimization)

13
1. Introduction

• Historical development
• Leonhard Euler (1707-1783)
• (Calculus of variations, minimization of
• functionals)
• Gottfried Leibnitz (1646-1716)
• (Differential calculus methods
• of optimization)

14
1. Introduction

• Historical development
• George Bernard Dantzig (1914-2005)
• (Linear programming and Simplex method
  (1947))
• Richard Bellman (1920-1984)
• (Principle of optimality in dynamic
• programming problems)
• Harold William Kuhn (1925-)
• (Necessary and sufficient conditions for the
  optimal solution of programming problems, game
  theory)

15
1. Introduction

• Historical development
• Albert William Tucker (1905-1995)
• (Necessary and sufficient conditions
• for the optimal solution of programming
• problems, nonlinear programming, game
• theory his PhD student
• was John Nash)
• Von Neumann (1903-1957)
• (game theory)

16
1. Introduction

• Mathematical optimization problem
• f0 Rn R objective function
• x(x1,..,xn) design variables (unknowns of the
  problem, they must be linearly independent)
• gi Rn R (i1,,m) inequality constraints
• The problem is a constrained optimization problem

17
1. Introduction

• If a point x corresponds to the minimum value of
  the function f (x), the same point also
  corresponds to the maximum value of the negative
  of the function, -f (x). Thus optimization can be
  taken to mean minimization since the maximum of a
  function can be found by seeking the minimum of
  the negative of the same function.

18
1. Introduction

• Constraints
• Behaviour constraints Constraints that represent
  limitations on the behaviour or performance of
  the system are termed behaviour or functional
  constraints.
• Side constraints Constraints that represent
  physical limitations on design variables such as
  manufacturing limitations.

19
1. Introduction

• Constraint Surface
• For illustration purposes, consider an
  optimization problem with only inequality
  constraints gj (X) ? 0. The set of values of X
  that satisfy the equation gj (X) 0 forms a
  hypersurface in the design space and is called a
  constraint surface.

20
1. Introduction

• Constraint Surface
• Note that this is a (n-1) dimensional subspace,
  where n is the number of design variables. The
  constraint surface divides the design space into
  two regions one in which gj (X) ? 0and the other
  in which gj (X) ?0.

21
1. Introduction

• Constraint Surface
• Thus the points lying on the hypersurface will
  satisfy the constraint
• gj (X) critically whereas the points lying
  in the region where gj (X) gt0 are infeasible or
  unacceptable, and the points lying in the region
  where gj (X) lt 0 are feasible or acceptable.

22
1. Introduction

• Constraint Surface
• In the below figure, a hypothetical two
  dimensional design space is depicted where the
  infeasible region is indicated by hatched lines.
  A design point that lies on one or more than one
  constraint surface is called a bound point, and
  the associated constraint is called an active
  constraint.

23
1. Introduction

• Constraint Surface
• Design points that do not lie on any constraint
  surface are known as free points.

24
1. Introduction

• Constraint Surface
• Depending on whether a particular design
  point belongs to the acceptable or unacceptable
  regions, it can be identified as one of the
  following four types
• Free and acceptable point
• Free and unacceptable point
• Bound and acceptable point
• Bound and unacceptable point

25
1. Introduction

• The conventional design procedures aim at finding
  an acceptable or adequate design which merely
  satisfies the functional and other requirements
  of the problem.
• In general, there will be more than one
  acceptable design, and the purpose of
  optimization is to choose the best one of the
  many acceptable designs available.
• Thus a criterion has to be chosen for comparing
  the different alternative acceptable designs and
  for selecting the best one.
• The criterion with respect to which the design is
  optimized, when expressed as a function of the
  design variables, is known as the objective
  function.

26
1. Introduction

• In civil engineering, the objective is usually
  taken as the minimization of the cost.
• In mechanical engineering, the maximization of
  the mechanical efficiency is the obvious choice
  of an objective function.
• In aerospace structural design problems, the
  objective function for minimization is generally
  taken as weight.
• In some situations, there may be more than one
  criterion to be satisfied simultaneously. An
  optimization problem involving multiple objective
  functions is known as a multiobjective
  programming problem.

27
1. Introduction

• With multiple objectives there arises a
  possibility of conflict, and one simple way to
  handle the problem is to construct an overall
  objective function as a linear combination of the
  conflicting multiple objective functions.
• Thus, if f1 (X) and f2 (X) denote two objective
  functions, construct a new (overall) objective
  function for optimization as
• where ?1 and ?2 are constants whose values
  indicate the relative importance of one objective
  function to the other.

28
1. Introduction

• The locus of all points satisfying f (X) c
  constant forms a hypersurface in the design
  space, and for each value of c there corresponds
  a different member of a family of surfaces. These
  surfaces, called objective function surfaces, are
  shown in a hypothetical two-dimensional design
  space in the figure below.

29
1. Introduction

• Once the objective function surfaces are drawn
  along with the constraint surfaces, the optimum
  point can be determined without much difficulty.
• But the main problem is that as the number of
  design variables exceeds two or three, the
  constraint and objective function surfaces become
  complex even for visualization and the problem
  has to be solved purely as a mathematical problem.

30
Example

• Example
• Design a uniform column of tubular section
  to carry a compressive load P2500 kgf for
  minimum cost. The column is made up of a material
  that has a yield stress of 500 kgf/cm2, modulus
  of elasticity (E) of 0.85e6 kgf/cm2, and density
  (?) of 0.0025 kgf/cm3. The length of the column
  is 250 cm. The stress induced in this column
  should be less than the buckling stress as well
  as the yield stress. The mean diameter of the
  column is restricted to lie between 2 and 14 cm,
  and columns with thicknesses outside the range
  0.2 to 0.8 cm are not available in the market.
  The cost of the column includes material and
  construction costs and can be taken as 5W 2d,
  where W is the weight in kilograms force and d is
  the mean diameter of the column in centimeters.

31
Example

• Example
• The design variables are the mean diameter
  (d) and tube thickness (t)
• The objective function to be minimized is
  given by

32
Example

• The behaviour constraints can be expressed as
• stress induced yield stress
• stress induced buckling stress
• The induced stress is given by

33
Example

• The buckling stress for a pin connected column
  is given by
• where I is the second moment of area of
  the cross section of the
  column given by

34
Example

• Thus, the behaviour constraints can be
  restated as
• The side constraints are given by

35
Example

• The side constraints can be expressed in
  standard form as

36
Example

• For a graphical solution, the constraint surfaces
  are to be plotted in a two dimensional design
  space where the two axes represent the two design
  variables x1 and x2. To plot the first constraint
  surface, we have
• Thus the curve x1x21.593 represents the
  constraint surface g1(X)0. This curve can be
  plotted by finding several points on the curve.
  The points on the curve can be found by giving a
  series of values to x1 and finding the
  corresponding values of x2 that satisfy the
  relation x1x21.593 as shown in the Table below

x1 2 4 6 8 10 12 14
x2 0.7965 0.3983 0.2655 0.199 0.1593 0.1328 0.114
37
Example

• The infeasible region represented by g1(X)gt0 or
  x1x2lt 1.593 is shown by hatched lines. These
  points are plotted and a curve P1Q1 passing
  through all these points is drawn as shown

38
Example

• Similarly the second constraint g2(X) lt 0 can be
  expressed as
• The points lying on the constraint surface g2
  (X)0 can be obtained as follows (These points
  are plotted as Curve P2Q2

x1 2 4 6 8 10 12 14
x2 2.41 0.716 0.219 0.0926 0.0473 0.0274 0.0172
39
Example

• The plotting of side constraints is simple since
  they represent straight lines.
• After plotting all the six constraints, the
  feasible region is determined as the bounded area
  ABCDEA

40
Example

• Next, the contours of the objective function are
  to be plotted before finding the optimum point.
  For this, we plot the curves given by
• for a series of values of c. By giving
  different values to c, the contours of f can be
  plotted with the help of the following points.

41
Example

• For
• For
• For
• For

x2 0.1 0.2 0.3 0.4 0.5 0.6 0.7
x1 16.77 12.62 10.10 8.44 7.24 6.33 5.64
x2 0.1 0.2 0.3 0.4 0.5 0.6 0.7
x1 13.40 10.10 8.08 6.75 5.79 5.06 4.51
x2 0.1 0.2 0.3 0.4 0.5 0.6 0.7
x1 10.57 7.96 6.38 5.33 4.57 4 3.56
x2 0.1 0.2 0.3 0.4 0.5 0.6 0.7
x1 8.88 6.69 5.36 4.48 3.84 3.36 2.99
42
Example

• These contours are shown in the figure below and
  it can be seen that the objective function can
  not be reduced below a value of 26.53
  (corresponding to point B) without violating some
  of the constraints. Thus, the optimum solution is
  given by point B with dx15.44 cm and
  tx20.293 cm with fmin26.53.

43
Examples

• Design of civil engineering structures
• variables width and height of member
  cross-sections
• constraints limit stresses, maximum and minimum
  dimensions
• objective minimum cost or minimum weight
• Analysis of statistical data and building
  empirical models from measurements
• variables model parameters
• Constraints physical upper and lower bounds for
  model parameters
• Objective prediction error

44
Classification of optimization problems

• Classification based on
• Constraints
• Constrained optimization problem
• Unconstrained optimization problem
• Nature of the design variables
• Static optimization problems
• Dynamic optimization problems

45
Classification of optimization problems

• Classification based on
• Physical structure of the problem
• Optimal control problems
• Non-optimal control problems
• Nature of the equations involved
• Nonlinear programming problem
• Geometric programming problem
• Quadratic programming problem
• Linear programming problem

46
Classification of optimization problems

• Classification based on
• Permissable values of the design variables
• Integer programming problems
• Real valued programming problems
• Deterministic nature of the variables
• Stochastic programming problem
• Deterministic programming problem

47
Classification of optimization problems

• Classification based on
• Separability of the functions
• Separable programming problems
• Non-separable programming problems
• Number of the objective functions
• Single objective programming problem
• Multiobjective programming problem

48
Geometric Programming

• A geometric programming problem (GMP) is one in
  which the objective function and constraints are
  expressed as posynomials in X.

49
(No Transcript)
50
Quadratic Programming Problem

• A quadratic programming problem is a nonlinear
  programming problem with a quadratic objective
  function and linear constraints. It is usually
  formulated as follows
• subject to
• where c, qi,Qij, aij, and bj are constants.

51
Optimal Control Problem

• An optimal control (OC) problem is a mathematical
  programming problem involving a number of stages,
  where each stage evolves from the preceding stage
  in a prescribed manner.
• It is usually described by two types of
  variables the control (design) and the state
  variables. The control variables define the
  system and govern the evolution of the system
  from one stage to the next, and the state
  variables describe the behaviour or status of the
  system in any stage.

52
Optimal Control Problem

• The problem is to find a set of control or design
  variables such that the total objective function
  (also known as the performance index) over all
  stages is minimized subject to a set of
  constraints on the control and state variables.
• An OC problem can be stated as follows
• Find X which minimizes
• subject to the constraints
• where xi is the ith control variable, yi is
  the ith control variable, and fi is the
  contribution of the ith stage to the total
  objective function gj, hk and qi are functions
  of xj, yk and xi and yi, respectively, and l is
  the total number of stages.

53
Integer Programming Problem

• If some or all of the design variables
  x1,x2,..,xn of an optimization problem are
  restricted to take on only integer (or discrete)
  values, the problem is called an integer
  programming problem.
• If all the design variables are permitted to take
  any real value, the optimization problem is
  called a real-valued programming problem.

54
Stochastic Programming Problem

• A stochastic programming problem is an
  optimization problem in which some or all of the
  parameters (design variables and/or preassigned
  parameters) are probabilistic (nondeterministic
  or stochastic).
• In other words, stochastic programming deals with
  the solution of the optimization problems in
  which some of the variables are described by
  probability distributions.

55
Separable Programming Problem

• A function f (x) is said to be separable if it
  can be expressed as the sum of n single variable
  functions, f1(x1), f2(x2),.,fn(xn), that is,
• A separable programming problem is one in which
  the objective function and the constraints are
  separable and can be expressed in standard form
  as
• Find X which minimizes
• subject to
• where bj is constant.

56
Multiobjective Programming Problem

• A multiobjective programming problem can be
  stated as follows
• Find X which minimizes f1 (X), f2 (X),., fk
  (X)
• subject to
• where f1 , f2,., fk denote the objective
  functions to be minimized simultaneously.

57
Review of mathematics

• Concepts from linear algebra Positive
  definiteness
• Test 1 A matrix A will be positive definite if
  all its eigenvalues are positive that is, all
  the values of ? that satisfy the determinental
  equation
• should be positive. Similarly, the matrix A
  will be negative definite if its eigenvalues are
  negative.

58
Review of mathematics

• Positive definiteness
• Test 2 Another test that can be used to find the
  positive definiteness of a matrix A of order n
  involves evaluation of the determinants
• The matrix A will be positive definite if and
  only if all the values A1, A2, A3,?An are
  positive
• The matrix A will be negative definite if and
  only if the sign of Aj is (-1)j for j1,2,?,n
• If some of the Aj are positive and the remaining
  Aj are zero, the matrix A will be positive
  semidefinite

59
Review of mathematics

• Negative definiteness
• Equivalently, a matrix is negative-definite if
  all its eigenvalues are negative
• It is positive-semidefinite if all its
  eigenvalues are all greater than or equal to zero
• It is negative-semidefinite if all its
  eigenvalues are all less than or equal to zero

60
Review of mathematics

• Concepts from linear algebra
• Nonsingular matrix The determinant of the matrix
  is not zero.
• Rank The rank of a matrix A is the order of the
  largest nonsingular square submatrix of A, that
  is, the largest submatrix with a determinant
  other than zero.

61
Review of mathematics

• Solutions of a linear problem
• Minimize f(x)cTx
• Subject to g(x) Axb
• Side constraints x 0
• The existence of a solution to this problem
  depends on the rows of A.
• If the rows of A are linearly independent, then
  there is a unique solution to the system of
  equations.
• If det(A) is zero, that is, matrix A is singular,
  there are either no solutions or infinite
  solutions.

62
Review of mathematics

• Suppose
• The new matrix A is called the augmented
  matrix- the columns of b are added to A.
  According to the theorems of linear algebra
• If the augmented matrix A and the matrix of
  coefficients A have the same rank r which is
  less than the number of design variables n (r lt
  n), then there are many solutions.
• If the augmented matrix A and the matrix of
  coefficients A do not have the same rank, a
  solution does not exist.
• If the augmented matrix A and the matrix of
  coefficients A have the same rank rn, where the
  number of constraints is equal to the number of
  design variables, then there is a unique
  solution.

63
Review of mathematics

• In the example
• The largest square submatrix is a 2 x 2
  matrix (since m 2 and m lt n). Taking the
  submatrix which includes the first two columns of
  A, the determinant has a value of 2 and therefore
  is nonsingular. Thus the rank of A is 2 (r 2).
  The same columns appear in A making its rank
  also 2. Since
• r lt n, infinitely many solutions exist.

64
Review of mathematics

• In the example
• One way to determine the solutions is to
  assign ( n-r) variables arbitrary values and use
  them to determine values for the remaining r
  variables. The value n-r is often identified as
  the degree of freedom for the system of
  equations.
• In this example, the degree of freedom is 1
  (i.e., 3-2). For instance x3 can be assigned a
  value of 1 in which case x10.5 and x21.5

65
Homework

• What is the solution of the system given
  below?
• Hint Determine the rank of the matrix of the
  coefficients and the augmented matrix.

66
2. Classical optimization techniques Single
variable optimization

• Useful in finding the optimum solutions of
  continuous and differentiable functions
• These methods are analytical and make use of the
  techniques of differential calculus in locating
  the optimum points.
• Since some of the practical problems involve
  objective functions that are not continuous
  and/or differentiable, the classical optimization
  techniques have limited scope in practical
  applications.

67
2. Classicial optimization techniques Single
variable optimization

• A function of one variable f (x) has a relative
  or local minimum at x x if f (x) f
  (xh) for all sufficiently small positive and
  negative values of h
• A point x is called a relative or local maximum
  if f (x) f (xh) for all values of h
  sufficiently close to zero.

68
2. Classicial optimization techniques Single
variable optimization

• A function f (x) is said to have a global or
  absolute minimum at x if f (x) f (x) for
  all x, and not just for all x close to x, in the
  domain over which f (x) is defined.
• Similarly, a point x will be a global maximum
  of f (x) if f (x) f (x) for all x in the
  domain.

69
Necessary condition

• If a function f (x) is defined in the interval a
  x b and has a relative minimum at x x,
  where a lt x lt b, and if the derivative df (x) /
  dx f(x) exists as a finite number at x x,
  then f (x)0
• The theorem does not say that the function
  necessarily will have a minimum or maximum at
  every point where the derivative is zero. e.g. f
  (x)0 at x 0 for the function shown in figure.
  However, this point is neither a minimum nor a
  maximum. In general, a point x at which f(x)0
  is called a stationary point.

70
Necessary condition
FIGURE 2.2 SAYFA 67

• The theorem does not say what happens if a
  minimum or a maximum occurs at a point x where
  the derivative fails to exist. For example, in
  the figure
• depending on whether h approaches zero
  through positive or negative values,
  respectively. Unless the numbers or are
• equal, the derivative f (x) does not
  exist. If f (x) does not exist, the theorem is
  not applicable.

71
Sufficient condition

• Let f(x)f(x)f (n-1)(x)0, but f(n)(x)
  ? 0. Then f(x) is
• A minimum value of f (x) if f (n)(x) gt 0 and n
  is even
• A maximum value of f (x) if f (n)(x) lt 0 and n
  is even
• Neither a minimum nor a maximum if n is odd

72
Example

• Determine the maximum and minimum values of the
  function
• Solution Since f(x)60(x4-3x32x2)60x2(x-1)(x-2
  ),
• f(x)0 at x0,x1, and
  x2.
• The second derivative is
• At x1, f(x)-60 and hence x1 is a relative
  maximum. Therefore,
• fmax f (x1)
  12
• At x2, f(x)240 and hence x2 is a relative
  minimum. Therefore,
• fmin f (x2)
  -11

73
Example

• Solution contd
• At x0, f(x)0 and hence we must investigate
  the next derivative.
• Since at x0, x0 is neither a
  maximum nor a minimum, and it is an inflection
  point.

74
Multivariable optimization with no constraints

• Definition rth Differential of f
• If all partial derivatives of the function f
  through order r 1 exist and are continuous at a
  point X, the polynomial
• is called the rth differential of f at X.

r summations
75
Multivariable optimization with no constraints

• Example rth Differential of f
• when r 2 and n 3, we have

r summations
76
Multivariable optimization with no constraints

• Definition rth Differential of f
• The Taylor series expansion of a function f
  (X) about a point X is given by
• where the last term, called the remainder is
  given by

77
Example

• Find the second order Taylors series
  approximation of the function
• about the point
• Solution The second order Taylors series
  approximation of the function f about point
  X is given by

78
Example contd

• where

79
Example contd

• where

80
Example contd

• Thus, the Taylors series approximation is given
  by
• Where h1x1-1, h2x2, and h3x32

81
Multivariable optimization with no constraints

• Necessary condition
• If f(X) has an extreme point (maximum or
  minimum) at XX and if the first partial
  derivatives of f (X) exist at X, then
• Sufficient condition
• A sufficient condition for a stationary point
  X to be an extreme point is that the matrix of
  second partial derivatives (Hessian matrix) of f
  (X) evaluated at X is
• Positive definite when X is a relative minimum
  point
• Negative definite when X is a relative maximum
  point

82
Example

• Figure shows two frictionless rigid bodies
  (carts) A and B connected by three linear elastic
  springs having spring constants k1, k2, and k3.
  The springs are at their natural positions when
  the applied force P is zero. Find the
  displacements x1 and x2 under the force P by
  using the principle of minimum potential energy.

83
Example

• Solution According to the principle of
  minimum potential energy, the system will be in
  equilibrium under the load P if the potential
  energy is a minimum. The potential energy of the
  system is given by
• Potential energy (U)
• Strain energy of springs-work done by
  external forces
• The necessary condition for the minimum of U are

84
Example

• Solution contd The sufficiency conditions
  for the minimum at (x1,x2) can also be verified
  by testing the positive definiteness of the
  Hessian matrix of U. The Hessian matrix of U
  evaluated at (x1,x2) is
• The determinants of the square submatrices
  of J are
• Since the spring constants are always
  positive. Thus the matrix J is positive definite
  and hence (x1,x2) corresponds to the minimum of
  potential energy.

85
Semi-definite case

• The sufficient conditions for the case when
  the Hessian matrix of the given function is
  semidefinite
• In case of a function of a single variable, the
  higher order derivatives in the Taylors series
  expansion are investigated

86
Semi-definite case

• The sufficient conditions for a function of
  several variables for the case when the Hessian
  matrix of the given function is semidefinite
• Let the partial derivatives of f of all orders
  up to the order k 2 be continuous in the
  neighborhood of a stationary point X, and
• so that dk f XX is the first
  nonvanishing higher-order differential of f at
  X.
• If k is even
• X is a relative minimum if dk f XX is
  positive definite
• X is a relative maximum if dk f XX is
  negative definite
• If dk f XX is semidefinite, no general
  conclusions can be drawn
• If k is odd, X is not an extreme point of f(X)

87
Saddle point

• In the case of a function of two variables f
  (x,y), the Hessian matrix may be neither positive
  nor negative definite at a point (x,y) at which
• In such a case, the point (x,y) is called
  a saddle point.
• The characteristic of a saddle point is that it
  corresponds to a relative minimum or maximum of f
  (x,y) wrt one variable, say, x (the other
  variable being fixed at yy ) and a relative
  maximum or minimum of f (x,y) wrt the second
  variable y (the other variable being fixed at
  x).

88
Saddle point

• Example Consider the function
• f (x,y)x2-y2. For this function
• These first derivatives are zero at x 0
  and y 0. The Hessian matrix of f at (x,y) is
  given by
• Since this matrix is neither positive
  definite nor negative definite, the point ( x0,
  y0) is a saddle point.

89
Saddle point

• Example contd
• It can be seen from the figure that f (x, y)
  f (x, 0) has a relative minimum and f (x, y) f
  (0, y) has a relative maximum at the saddle point
  (x, y).

90
Example

• Find the extreme points of the function
• Solution The necessary conditions for the
  existence of an extreme point are
• These equations are satisfied at the points
  (0,0), (0,-8/3), (-4/3,0), and (-4/3,-8/3)

91
Example

• Solution contd To find the nature of these
  extreme points, we have to use the sufficiency
  conditions. The second order partial derivatives
  of f are given by
• The Hessian matrix of f is given by

92
Example

• Solution contd
• If J16x14 and
  , the values of J1 and J2 and
• the nature of the extreme point are as
  given in the next slide

93
Example
Point X Value of J1 Value of J2 Nature of J Nature of X f (X)
(0,0) 4 32 Positive definite Relative minimum 6
(0,-8/3) 4 -32 Indefinite Saddle point 418/27
(-4/3,0) -4 -32 Indefinite Saddle point 194/27
(-4/3,-8/3) -4 32 Negative definite Relative maximum 50/3
94
Multivariable optimization with equality
constraints

• Problem statement
• Minimize f f (X) subject to gj(X)0,
  j1,2,..,m where
• Here m is less than or equal to n,
  otherwise the problem becomes overdefined and, in
  general, there will be no solution.
• Solution
• Solution by direct substitution
• Solution by the method of constrained variation
• Solution by the method of Lagrange multipliers

95
Solution by direct substitution

• For a problem with n variables and m equality
  constraints
• Solve the m equality constraints and express any
  set of m variables in terms of the remaining n-m
  variables
• Substitute these expressions into the original
  objective function, the result is a new objective
  function involving only n-m variables
• The new objective function is not subjected to
  any constraint, and hence its optimum can be
  found by using the unconstrained optimization
  techniques.

96
Solution by direct substitution

• Simple in theory
• Not convenient from a practical point of view as
  the constraint equations will be nonlinear for
  most of the problems
• Suitable only for simple problems

97
Example

• Find the dimensions of a box of largest
  volume that can be inscribed in a sphere of unit
  radius
• Solution Let the origin of the Cartesian
  coordinate system x1, x2, x3 be at the center of
  the sphere and the sides of the box be 2x1, 2x2,
  and 2x3. The volume of the box is given by
• Since the corners of the box lie on the
  surface of the sphere of unit radius, x1, x2 and
  x3 have to satisfy the constraint

98
Example

• This problem has three design variables and
  one equality constraint. Hence the equality
  constraint can be used to eliminate any one of
  the design variables from the objective function.
  If we choose to eliminate x3
• Thus, the objective function becomes
• f(x1,x2)8x1x2(1-x12-x22)1
  /2
• which can be maximized as an unconstrained
  function in two variables.

99
Example

• The necessary conditions for the maximum of f
  give
• which can be simplified as
• From which it follows that x1x21/?3 and hence
  x3 1/?3

100
Example

• This solution gives the maximum volume of the
  box as
• To find whether the solution found
  corresponds to a maximum or minimum, we apply the
  sufficiency conditions to f (x1,x2) of the
  equation f (x1,x2)8x1x2(1-x12-x22)1/2. The
  second order partial derivatives of f at
  (x1,x2) are given by

101
Example

• The second order partial derivatives of f at
  (x1,x2) are given by

102
Example

• Since
• the Hessian matrix of f is negative definite
  at (x1,x2). Hence the point (x1,x2)
  corresponds to the maximum of f.

103
Solution by constrained variation

• Minimize f (x1,x2)
• subject to g(x1,x2)0
• A necessary condition for f to have a minimum at
  some point (x1,x2) is that the total derivative
  of f (x1,x2) wrt x1 must be zero at (x1,x2)
• Since g(x1,x2)0 at the minimum point, any
  variations dx1 and dx2 taken about the point
  (x1,x2) are called admissable variations
  provided that the new point lies on the
  constraint

104
Solution by constrained variation

• Taylors series expansion of the function about
  the point (x1,x2)
• Since g(x1, x2)0
• Assuming
• Substituting the above equation into

105
Solution by constrained variation

• The expression on the left hand side is called
  the constrained variation of f
• Since dx1 can be chosen arbitrarily
• This equation represents a necessary condition in
  order to have (x1,x2) as an extreme point
  (minimum or maximum)

106
Example

A beam of uniform rectangular cross section
is to be cut from a log having a circular cross
secion of diameter 2 a. The beam has to be used
as a cantilever beam (the length is fixed) to
carry a concentrated load at the free end. Find
the dimensions of the beam that correspond to the
maximum tensile (bending) stress carrying
capacity.
107
Example

Solution From elementary strength of
materials, we know that the tensile stress
induced in a rectangular beam ? at any fiber
located at a distance y from the neutral axis is
given by where M is the bending
moment acting and I is the moment of inertia of
the cross-section about the x axis. If the width
and the depth of the rectangular beam shown in
the figure are 2x and 2y, respectively, the
maximum tensile stress induced is given by
108
Example
solution contd Thus for any specified
bending moment, the beam is said to have maximum
tensile stress carrying capacity if the maximum
induced stress (?max) is a minimum. Hence we need
to minimize k/xy2 or maximize Kxy2, where k3M/4
and K1/k, subject to the constraint This
problem has two variables and one constraint
hence the equation can be applied for
finding the optimum solution.
109
Example
Solution Since

we have Equation
gives
110
Example
Solution that is Thus the beam of
maximum tensile stress carrying capacity has a
depth of ?2 times its breadth. The optimum values
of x and y can be obtained from the above
equation and as
111
Solution by constrained variation

• Necessary conditions for a general problem
• The procedure described can be generalized to a
  problem with n variables and m constraints.
• In this case, each constraint equation gj(x)0,
  j1,2,..,m gives rise to a linear equation in the
  variations dxi, i1,2,,n.
• Thus, there will be in all m linear equations in
  n variations. Hence any m variations can be
  expressed in terms of the remaining n-m
  variations.
• These expressions can be used to express the
  differential of the objective function, df, in
  terms of the n-m independent variations.
• By letting the coefficients of the independent
  variations vanish in the equation df 0, one
  obtains the necessary conditions for the
  cnstrained optimum of the given function.

112
Solution by constrained variation

• Necessary conditions for a general problem
• These conditions can be expressed as
• It is to be noted that the variations of the
  first m variables (dx1, dx2,.., dxm) have been
  expressed in terms of the variations of the
  remaining n-m variables (dxm1, dxm2,.., dxn) in
  deriving the above equation.

113
Solution by constrained variation

• Necessary conditions for a general problem
• This implies that the following relation is
  satisfied
• The n-m equations given by the below equation
  represent the necessary conditions for the
  extremum of f(X) under the m equality
  constraints, gj(X) 0, j1,2,,m.

114
Example

• Minimize
• subject to
• Solution This problem can be solved by applying
  the necessary conditions given by

115
Example

• Solution contd Since n 4 and m 2, we
  have to select two variables as independent
  variables. First we show that any arbitrary set
  of variables can not be chosen as independent
  variables since the remaining (dependent)
  variables have to satisfy the condition of
• In terms of the notation of our equations,
  let us take the independent variables as x3y3
  and x4y4 so that x1y1 and x2y2. Then the
  Jacobian becomes
• and hence the necessary conditions can not
  be applied.

116
Example

• Solution contd Next, let us take the
  independent variables as x3y2 and x4y4 so
  that x1y1 and x2y3. Then the Jacobian
  becomes
• and hence the necessary conditions of
• can be applied.

117
Example

• Solution contd The equation
• give for k m13

118
Example

• Solution contd
• For k m24
• From the two previous equations, the
  necessary conditions for the minimum or the
  maximum of f is obtained as

119
Example

• Solution contd
• When the equations
• are substituted, the equations
• take the form

120
Example

• Solution contd
• from which the desired optimum solution can
  be obtained as

121
Solution by constrained variation

• Sufficiency conditions for a general problem
• By eliminating the first m variables, using the m
  equality constraints, the objective function can
  be made to depend only on the remaining variables
  xm1, xm2, ,xn. Then the Taylors series
  expansion of f , in terms of these variables,
  about the extreme point X gives
• where is used to denote
  the partial derivative of f wrt xi (holding all
  the other variables xm1, xm2, ,xi-1, xi1,
  xi2,,xn constant) when x1, x2, ,xm are allowed
  to change so that the constraints gj(XdX)0,
  j1,2,,m are satisfied the second derivative
  is used to denote a similar
  meaning.

122
Solution by constrained variation

• Example
• Consider the problem of minimizing
• Subject to the only constraint
• Since n3 and m1 in this problem, one can
  think of any of the m variables, say x1, to be
  dependent and the remaining n-m variables, namely
  x2 and x3, to be independent.
• Here the constrained partial derivative
  means the rate of change of f with
  respect to x2 (holding the other independent
  variable x3 constant) and at the same time
  allowing x1 to change about X so as to satisfy
  the constraint g1(X)0

123
Solution by constrained variation

• Example
• In the present case, this means that dx1 has to
  be chosen to satisfy the relation
• since g1(X)0 at the optimum point and dx3 0
  (x3 is held constant.)

124
Solution by constrained variation

• Example
• Notice that (df/dxi)g has to be zero for im1,
  m2,...,n since the dxi appearing in the equation
• are all independent. Thus, the necessary
  conditions for the existence of constrained
  optimum at X can also be expressed as

125
Solution by constrained variation

• Example
• It can be shown that the equations
• are nothing bu the equation

126
Sufficiency conditions for a general problem

• A sufficient condition for X to be a constrained
  relative minimum (maximum) is that the quadratic
  form Q defined by
• is positive (negative) for all nonvanishing
  variations dxi and the matrix
• has to be positive (negative) definite to
  have Q positive (negative) for all choices of dxi
  

127
Solution by constrained variation

• The computation of the constrained derivatives in
  the sufficiency condition is difficult and may be
  prohibitive for problems with more than three
  constraints
• Simple in theory
• Difficult to apply since the necessary conditions
  involve evaluation of determinants of order m1

128
Solution by Lagrange multipliers

• Problem with two variables and one constraint
• Minimize f (x1,x2)
• Subject to g(x1,x2)0
• For this problem, the necessary condition was
  found to be
• By defining a quantity ?, called the Lagrange
  multiplier as

129
Solution by Lagrange multipliers

• Problem with two variables and one constraint
• Necessary conditions for the point (x1,x2) to
  be an extreme point
• The problem can be rewritten as
• In addition, the constraint equation has to be
  satisfied at the extreme point

130
Solution by Lagrange multipliers

• Problem with two variables and one constraint
• The derivation of the necessary conditions by the
  method of Lagrange multipliers requires that at
  least one of the partial derivatives of g(x1,x2)
  be nonzero at an extreme point.
• The necessary conditions are more commonly
  generated by constructing a function L,known as
  the Lagrange function, as

131
Solution by Lagrange multipliers

• Problem with two variables and one constraint
• By treating L as a function of the three
  variables x1, x2 and ?, the necessary conditions
  for its extremum are given by

132
Example

• Example Find the solution using the Lagrange
  multiplier method.
• Solution
• The Lagrange function is

133
Example

• Solution contd
• which yield

134
Solution by Lagrange multipliers

• Necessary conditions for a general problem
• Minimize f(X)
• subject to
• gj (X) 0, j1, 2,.,m
• The Lagrange function, L, in this case is
  defined by introducing one Lagrange multiplier ?j
  for each constraint gj(X) as

135
Solution by Lagrange multipliers

• By treating L as a function of the nm
  unknowns, x1, x2,,xn,?1, ?2,, ?m, the necessary
  conditions for the extremum of L, which also
  corresponds to the solution of the original
  problem are given by
• The above equations represent nm equations
  in terms of the nm unknowns, xi and ?j

136
Solution by Lagrange multipliers

• The solution
• The vector X corresponds to the relative
  constrained minimum of f(X) (sufficient
  conditions are to be verified) while the vector
  ? provides the sensitivity information.

137
Solution by Lagrange multipliers

• Sufficient Condition
• A sufficient condition for f(X) to have a
  constrained relative minimum at X is that the
  quadratic Q defined by
• evaluated at XX must be positive definite
  for all values of dX for which the constraints
  are satisfied.
• If
• is negative for all choices of the admissable
  variations dxi, X will be a constrained maximum
  of f(X)

138
Solution by Lagrange multipliers

• A necessary condition for the quadratic form
  Q to be positive (negative) definite for all
  admissable variations dX is that each root of the
  polynomial zi, defined by the following
  determinantal equation, be positive (negative)
• The determinantal equation, on expansion, leads
  to an (n-m)th-order polynomial in z. If some of
  the roots of this polynomial are positive while
  the others are negative, the point X is not an
  extreme point.

139
Example 1

• Find the dimensions of a cylindirical tin
  (with top and bottom) made up of sheet metal to
  maximize its volume such that the total surface
  area is equal to A024?.
• Solution
• If x1 and x2 denote the radius of the base
  and length of the tin, respectively, the problem
  can be stated as
• Maximize f
  (x1,x2) ?x12x2
• subject to

140
Example 1

• Solution
• Maximize f (x1,x2) ?x12x2
• subject to
• The Lagrange function is
• and the necessary conditions for the maximum of f
  give

141
Example 1

• Solution
• that is,
• The above equations give the desired solution as

142
Example 1

• Solution
• This gives the maximum value of f as
• If A0 24?, the optimum solution becomes
• To see that this solution really corresponds to
  the maximum of f, we apply the sufficiency
  condition of equation

143
Example 1

• Solution
• In this case

144
Example 1

• Solution
• Thus, equation
• becomes

145
Example 1

• Solution
• that is,
• This gives
• Since the value of z is negative, the point
  (x1,x2) corresponds to the maximum of f.

146
Example 2

• Find the maximum of the function f (X)
  2x1x210 subject to g (X)x122x22 3 using the
  Lagrange multiplier method. Also find the effect
  of changing the right-hand side of the constraint
  on the optimum value of f.
• Solution
• The Lagrange function is given by
• The necessary conditions for the solution
  of the problem are

147
Example 2

• Solution
• The solution of the equation is
• The application of the sufficiency condition
  yields

148
Example 2

• Solution
• Hence X will be a maximum of f with f
  f (X)16.07

149
Multivariable optimization with inequality
constraints

• Minimize f (X)
• subject to
• gj (X) 0, j1, 2,,m
• The inequality constraints can be
  transformed to equality constraints by adding
  nonnegative slack variables, yj2, as
• gj (X) yj2 0, j 1,2,,m
• where the values of the slack variables are
  yet unknown.

150
Multivariable optimization with inequality
constraints

• Minimize f(X) subject to
• Gj(X,Y) gj (X) yj20, j1,
  2,,m
• where is the vector of slack
  variables
• This problem can be solved by the method of
  Lagrange multipliers. For this, the Lagrange
  function L is constructed as

151
Multivariable optimization with inequality
constraints

• The stationary points of the Lagrange
  function can be found by solving the following
  equations (necessary conditions)
• (n2m) equations
• (n2m) unknowns
• The solution gives the optimum solution
  vector X, the Lagrange multiplier vector, ?,
  and the slack variable vector, Y.

152
Multivariable optimization with inequality
constraints

• Equation
• ensure that the constraints
• are satisfied, while the equation
• implies that either ?j0 or yj0

153
Multivariable optimization with inequality
constraints

• If ?j0, it means that the jth constraint is
  inactive and hence can be ignored.
• On the other hand, if yj 0, it means that the
  constraint is active (gj 0) at the optimum
  point.
• Consider the division of the constraints into two
  subsets, J1 and J2, where J1 J2 represent the
  total set of constraints.
• Let the set J1 indicate the indices of those
  constraints that are active at the optimum point
  and J2 include the indices of all the inactive
  constraints.
• Those constraints that are satisfied with an
  equality sign, gj 0, at the optimum point are
  called the active constraints, while those that
  are satisfied with a strict inequality sign, gjlt
  0 are termed inactive constraints.

154
Multivariable optimization with inequality
constraints

• Thus for j? J1, yj 0 (constraints are active),
  for j? J2, ?j0 (constraints are inactive), and
  the equation
• can be simplified as

155
Multivariable optimization with inequality
constraints

• Similarly, the equation
• can be written as
• The equations (1) and (2) represent
  np(m-p)nm equations in the nm unknowns xi
  (i1,2,,n), ?j (j ? J1), and yj (j ? J2), where
  p denotes the number of active constraints.

(1)
(2)
156
Multivariable optimization with inequality
constraints

• Assuming that the first p constraints are
  active, the equation
• can be expressed as
• These equations can be collectively written
  as

157
Multivariable optimization with inequality
constraints

• Equation
• indicates that the negative of the gradient
  of the objective function can be expressed as a
  linear combination of the gradients of the active
  constraints at the optimum point.

158
Multivariable optimization with inequality
constraints-Feasible region

• A vector S is called a feasible direction from a
  point X if at least a small step can be taken
  along S that does not immediately leave the
  feasible region.
• Thus for problems with sufficiently smooth
  constraint surfaces, vector S satisfying the
  relation
• can be called a feasible direction.

159
Multivariable optimization with inequality
constraints-Feasible region

• On the other hand, if the constraint is either
  linear or concave, any vector satisfying the
  relation
• can be called a feasible region.
• The geometric interpretation of a feasible
  direction is that the vector S makes an obtuse
  angle with all the constraint normals.

160
Multivariable optimization with inequality
constraints-Feasible region

161
Multivariable optimization with inequality
constraints

• Further we can show that in the case of a
  minimization problem, the ?j values (j ? J1),
  have to be positive. For simplicity of
  illustration, suppose that only two constraints
  (p2) are active at the optimum point.
• Then the equation
• reduces to

162
Multivariable optimization with inequality
constraints

• Let S be a feasible direction at the optimum
  point. By premultiplying both sides of the
  equation
• by ST, we obtain
• where the superscript T denotes the
  transpose. Since S is a feasible direction, it
  should satisfy the relations

163
Multivariable optimization with inequality
constraints

• Thus if, ?1 gt 0 and ?2 gt 0 the quantity ST?f is
  always positive.
• As ?f indicates the gradient direction, along
  which the value of the function increases at the
  maximum rate, ST?f represents the component of
  the increment of f along the direction S.
• If ST?f gt 0, the function value increases, the
  function value increases as we move along the
  direction S.
• Hence if ?1 and ?2 are positive, we will not be
  able to find any direction in the feasible domain
  along which the function value can be decreased
  further.

164
Multivariable optimization with inequality
constraints

• Since the point at which the equation
• is valid is assumed to be optimum, ?1 and
  ?2 have to be positive.
• This reasoning can be extended to cases where
  there are more than two constraints active. By
  proceeding in a similar manner, one can show that
  the ?j values have to be negative for a
  maximization problem.

165
Kuhn-Tucker Conditions

• The conditions to be satisfied at a constrained
  minimum point, X, of the problem can b