BASIC CONCEPTS IN OPTIMIZATION: PART II: Continuous

1
BASIC CONCEPTS IN OPTIMIZATION PART II
Continuous Unconstrained
Important concepts for the optimization of
systems with continuous variables and non-linear
equations. Since we will limit the topic to
unconstrained problems, we will concentrate on
the OBJECTIVE FUNCTION.

• Opt. Conditions for Single Variable
• Opt. Conditions for Multivariable Variable
• Convexity and Its Importance

2
BASIC CONCEPTS IN OPTIMIZATION PART II
Continuous Unconstrained
Wait a minute. No problem is unconstrained so,
why do we need to know this?

• Unconstrained problems - sometimes the solution
  doesnt involve constraints
• Used in methods for constrained problems

3
BUILDING EXPERIENCE IN OPTIMIZATION
CLASS EXERCISE The reactor is isothermal and the
reaction kinetics are first order. Is this
system linear or non-linear?

• What must we define before defining an optimum?
• - The goal is to maximize CB in the effluent at
  S-S
• - You can adjust only the flow rate of feed

This is an isothermal CFSTR with the reaction A
? B ? C You can only adjust F
4
WHAT DEFINES THE LOCATION OF AN OPTIMUM?
For LP, the optimum is at a corner point.
For NLP the optimum is located .?
5
WHAT DEFINES THE LOCATION OF AN OPTIMUM Single
variable?
We will start with a single-variable system and
then generalize to multiple variable. We will
not yet include constraints.
The general definition of a minimum of f(x) is
x is a minimum if f(x) ? f(x ?x) for small
??x
We want to apply this concept, but we need to
determine specific criteria that test for
conformance to the statement in the box above.
6
WHAT DEFINES THE LOCATION OF AN OPTIMUM Single
variable?
Necessary Condition for a single-variable system
df(x)/dxx 0
Lets look at the definition of a derivative,
which is continuous
Why isnt this sufficient for a minimum?
If this exists and f(x) ? (f(x?x), then
7
WHAT DEFINES THE LOCATION OF AN OPTIMUM Single
variable?
Necessary Condition for a single-variable system
df(x)/dxx 0
(a)
(b)
(c)
(d)
(e)
Where is the derivative zero?
8
WHAT DEFINES THE LOCATION OF AN OPTIMUM Single
variable?
Sufficient condition A function with f(x)0
has f(x) fn-1(x) 0 (the next n-1
derivatives zero) has for n even fn(x) gt 0
(the nth derivative at x gt 0 )
Approximate the function with a Taylor Series.
0
0
Remainder ( 0 ? h ? 1)
9
WHAT DEFINES THE LOCATION OF AN OPTIMUM Single
variable?
Sufficient condition A function with f(x)0
has f(x) fn-1(x) 0 (2nd to n-1
derivatives zero) has for n even fn(x) gt 0
(the nth derivative at x gt 0 )
Rearrange the result.
For n even, (?x)n gt 0 when nth derivative is
positive, the condition for a minimum is
satisfied!
10
WHAT DEFINES THE LOCATION OF AN OPTIMUM Single
variable?
Necessary Sufficient Conditions df(x)/dxx
0 d2f(x)/dx2 gt 0
(a)
(b)
(c)
(d)
(e)
Which satisfy the necessary sufficient?
11
WHAT DEFINES THE LOCATION OF AN OPTIMUM Single
variable?
Lets look at the following examples. f1 3 2x
5x2 df1/dx 2 10x 0 x
-.20 d2f1/dx2 10 gt 0 at x -.20 Therefore,
the function has a local minimum at x x
-.20 f1 3 2x - 5x2 df1/dx 2 - 10x 0
x .20 d2f1/dx2 -10 lt 0 at x
.20 Therefore, the function has a local maximum
at x x .20
12
WHAT DEFINES THE LOCATION OF AN OPTIMUM Single
variable?

• Are these results consistent with the methods you
  have learned previously?
• What do we conclude if n odd?
• What type of extremum occurs for f(x) x4?

13
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
Necessary Lets extend these results to
multivariable systems, with x a vector of
dimension n.
Necessary condition
The proof is similar to the single-variable case.
We call these equations the stationarity
conditions.
14
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
The necessary condition for unconstrained
optimization of a multivariable system is often
stated as the following.
The gradient equaling zero is the stationarity
condition.
15
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
Sufficient Lets extend these results to
multivariable systems, with x a vector of
dimension n. We will restrict sufficient
conditions to second derivatives.
The first and second differential is defined as
16
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
These terms can be used in the expression for a
Taylor series to determine the sufficient
condition.
0
Remainder ( 0 ? h ? 1)
The condition for a minimum is satisfied when the
remainder is positive.
17
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
H the Hessian of second derivatives
It is symmetric.
18
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
For a minimum, the right hand side is positive
for any non-zero values of the vector ?x. How
can we tell? We need to evaluate an infinite
number of values of ?x!
Lets try a little mathematics to improve the
situation
19
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
We will consider a two-dimensional system. We
start by defining a new vector of variables, w.
Can we define the bs to make the test for
optimality easier?
20
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
The optimality test would be easy if the hessian
were diagonal.
How can we determine the bs to give this nice,
diagonal hessian matrix?
Then,
If,
21
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
The answer is determined from the eigenvalues and
eigenvectors of the hessian matrix!!!
When we prove that the function f(w) has a
minimum at w from ?1 gt 0 and ?2 gt 0 we also
prove that the function f(x) has a minimum at x!
22
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
A schematic of what we did. The coordinates are
rotated to express the quadratic as the sum of
variables squared times eigenvalues.
w1
Clearly, the remainder term must only increase if
all ?i are positive.
w2
23
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
Positive Definite A matrix is positive definite
if all values of its eigenvalues (?) are
positive. Eigenvalues are the solution to the
following equation, with H evaluated at x.
H - ?I 0
What is the form of this equation? How many
solutions are there?
24
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
The following two conditions are necessary
sufficient at x
The Hessian is positive definite
The gradient is zero

• Some good news - We do not typically perform
  these calculations to test problems
• But, these concepts are used in many solution
  methods for non-linear optimization.

25
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
This is a nice objective function, which is
convex and symmetric. Local derivative
information will direct us toward the minimum.
All eigenvalues are positive.
26
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
This is an objective function with a ridge. We
will find the valley quickly then, we will
search the ridge with little success.
One eigenvalue is near zero.
27
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
This objective function has a saddle point, which
has a minimum in one direction and maximum in
another direction. Derivative information will
not direct us well.
One eigenvalue is positive, and another is
negative.
28
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Multivariable?
Whats going on here?
29
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Convexity?
Convexity and the objective function. A function
of x (a vector) is convex if the following is
true.
For points x1 and x2 and 0 ? ? ? 1.
Is this function convex (over the region in the
figure)?
f(x)
x
30
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Convexity?
Convexity and the objective function. A function
of x (a vector) is convex if the following is
true.
For points x1 and x2 and 0 ? ? ? 1.
Is this function convex (over the region in the
figure)?
f(x)
x
31
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Convexity?
Convexity and the objective function. A function
of x (a vector) is convex over a region if the
following is true over the region.
For points x1 and x2 and 0 ? ? ? 1.
Class exercise Sketch a continuous function that
is convex of one region and concave over another
region. For concave, the sense of the inequality
is switched.
32
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Convexity?
Convexity and the objective function. A function
of x (a vector) is convex over a region if the
following is true over the region.
Gradient Test
Hessian Test The function is convex if its
Hessian matrix is positive definite
positive
33
WHAT DEFINES THE LOCATION OF AN OPTIMUM
Convexity?

• What is difference between suff. condition and
  convexity?
• Why is convexity important?

Any local minimum of a convex function (over an
unconstrained region) is a global minimum!
34
BASIC CONCEPTS IN OPTIMIZATION PART II
Continuous Unconstrained
For continuous variables and non-linear
equations, properties of the OBJECTIVE FUNCTION

• Opt. Conditions for Single Variable
• Opt. Conditions for Multivariable Variable
• Convexity and Its Importance

Basis of many optimization algorithms and tests
for convergence
We seek to formulate our models to yield a convex
programming problem