Global Optimization

1
Global Optimization
The Problem minimize f(x) subject to
gi(x)gtbi i1,,m hj(x)cj j1,,n When x is
discrete we call this combinatorial
optimization i.e. an optimization problem with a
finite number of feasible solutions. Note that
when the objective function and/or
constraints cannot be expressed analytically,
solution techniques used in combinatorial
problems can be used to solve this problem
2
(No Transcript)
3
Assignment Problem
A set of n people is available to carry out n
tasks. If person i does task j, it costs cij
units. Find an assignment x1,xn that
minimizes ????ni1cixi The solution is
represented by the permutation x1,xn of the
numbers 1,,n. Example task solution
x1 does task 2 1 2 3 x2 does task
3 x1 3 2 5 x3 does task 1 x2 9 7 1
cost x3 4 5 8
person
4
Knapsack Problem
A set of n items is available to be packed into a
knapsack with capacity C units. Item i has value
vi and uses up ci units of capacity. Determine
the subset I of items which should be packed in
order to minimize ?Ivi subject to ?Icilt C Here
the solution is represented by the subset I of
the set 1,,n. Example value capacity soluti
on 1 2.7 C/2 I1,3 2 3.2 C/4 3 1.1 C/2
5
Traveling Salesman Problem (TSP)
A salesperson must visit n cities once before
returning home. The distance from city i to city
j is dij. What ordering of the cities minimizes
the distance the salesperson must travel before
returning home? SETUP minimize ?ni,j1dijxij
subject to ?ni1xij1, ?nj1xij1 where
xij 1 if go from city i to city j 0
otherwise Note that this is an integer
programming problem and there are (n-1)!
possible solutions to this problem.
6
Integer Programming
Integer problems involve large numbers of
variables and constraint and quickly become very
large problems.
Finding a Solution
If the function is piecewise linear, the problem
can be solved exactly with a mixed integer
program method that uses branch and bound
(later). Otherwise, Heuristic methods (finding
methods) can be used to find approximate
solutions. What are heuristic methods? Definition
A heuristic is a technique which seeks good
(i.e. near optimal) solutions at a reasonable
computational cost without being able to
guarantee either feasibility or optimality, or
even in many cases to state how close to
optimality a particular feasible solution is.
7
Branch and Bound (In general)
Branch and bound is a general search method used
to find the minimum of a function, f(x), where x
is restricted to some feasible region.
L lower bound U upper bound
f1L2 f1U7
f1L2 f1U7
f0L2 f0U9
f2L4 f2U9
f3L4 f3U4
f4L5 f4U9
Step 0
Step 1
Step 2
In step 2 the lower bound equals the upper bound
in region 3, so 4 is a optimal solution for
region 3. Region 4 can be removed
from consideration since it has a lower bound of
5, which is greater than 4. Continue branching
until the location of the global optimal is found.
The difficulty with this method is in determining
the lower and upper bounds on each of the regions.
8
(No Transcript)
9
Clustering Methods
Clustering methods are an improvement to
multistart methods. Multistart methods These
are methods of optimization that determine
the global optimal by comparing the local optimal
attained from a large number of different
starting points. These methods are inefficient
because many starting points may lead to the same
local minimum. Clustering methods A form of a
multistart method, with one major difference.
Neighborhoods of starting points that lead to the
same local minimum are estimated. This decreases
redundancies in local minimum values that are
found.
x2
x2
x2




























x1
x1
x1
Step 0 Sample points
Step 1 Create groups
Step 2 Continue sampling
The challenge is how to identify the groups.
10
Simulated Annealing
A method that is useful in solving combinatorial
optimization problems. At the the heart of this
method is the annealing process studied
in thermodynamics.
High Temperature
Low Temperature
Thermal mobility is lost as the temperature
decreases.
Thermodynamic Structure Combinatorial
Optimization System states Feasible
solutions Energy Cost Change of
state Neighboring solution Temperature Control
parameter Frozen state Heuristic solution
11
Simulated Annealing
The Boltzmann probability distribution
Prob(E)exp(-E/kT). k is the Boltzmanns constant
which relates temperature to energy. A system in
thermal equilibrium at temperature T has its
energy probabilistically distributed among all
different energy states E. Even if the
temperature is low, there is a chance that the
energy state will be high. (This is a small
chance, but it is a chance nonetheless.) There is
a chance for the system to get out of local
energy minimums in search of the global minimum
energy state. The general scheme of always
taking downhill steps while sometimes taking an
uphill step has come to be known as
the Metropolis algorithm, named after Metropolis
who first incorporated simulated annealing ideas
in an optimization problem in 1953.
12
Simulated Annealing
Algorithm to minimize the cost function, f(s). 1)
Select an initial solution s0, an initial
temperature t0 and set iter0. 2) Select a
temperature reduction function, ?, and a maximum
number of iterations, nrep. 3) Randomly select a
new point s in the neighborhood of s0 set
iteriter1. 4) If f(s)ltf(s0), then s0s and go
to step 3 until iter gt nrep. 5) Generate random x
between 0 and 1. 6) If x lt exp(f(s0)-f(s)/t) then
s0s and go to step 3 until iter gt nrep. 7) Let
s0 remain the same and go to step 3 until iter gt
nrep. 8) Set t ?(t) until stopping criteria is
met. 9) The approximation to the optimal solution
is s0. All of the following parameters affect
the speed and quality of the solution. t0a high
value for free exchange. N(s0) by swap,
mutation, random, etc. ?cooling should be
gradual t decay. stopping criteriaminimum temp
total nrep related to the dimension number
of iterations exceeded can vary with t (higher
with low t). proportion of acceptable moves.
13
Hybrid Methods
MINLP Mixed Integer Nonlinear
Programming Branch and Bound (Talked about this
earlier.) 1) Relax the integer constraints
forming a nonlinear problem. 2) Fix the integer
values found to be closest to the solution
found in step 1. 3) Solve the new nonlinear
programming problems for the fixed integer values
until all of the integer parameters are
determined. Requires a large number of NLP
problems to be solved.
14
Hybrid Methods
Tree Annealing Simulated annealing applied to
continuous functions Algorithm 1) Randomly
choose an initial point, x, over a search
interval, S0. 2) Randomly travel down the tree to
an arbitrary terminal node i, and generate a
candidate point, y, over the subspace defined by
Si. 3) If f(y) lt f(x), then replace x with y and
go to step 5. 4) Compute Pexp(-(f(y)-f(x))/T).
If P gt R, where R is a random number uniformly
distributed between 0 and 1, then replace x with
y. 5) If y replaced x, then decrease T slightly
and update the tree. 6) Set ii1, and go to 2
until T lt Tmin.
15
Hybrid Methods
Differences between tree and simulated
annealing 1) The points x and y are sampled from
a continuous space. 2) A minimum value is
determined by an increasing density of nodes in a
given region. The subspace over which candidate
points are chosen decreases in size as a minimum
value is approached. 3) The probability of
accepting y is now governed by a
modified criterion Pg(x)p(y)/g(y)p(x) where
p(y)exp(-f(x)/T) and g(x)(1/Vx)qi g is
dependent upon the volume of the node associated
with the subspace defined by x, Vx, as well as
the path from the root node to the current node,
qi. Tree annealing is not guaranteed to converge
and often convergence is very slow. Use tree
annealing as a first step, then use a
gradient method to attain the minimum.
16
Statistical Global Optimization
A statistical model of the objective function is
used to bias the selection of new sample points.
This is a Bayesian argument, where we use
information about the objective function
gathered to make decisions about where to sample
new points. Problems 1) The statistical model
used may not truly represent the objective
function. If the statistical model is
determined prematurely, the optimization
algorithm may be biased and lead to unreliable
solutions. 2) Determining a statistical model can
be mathematically intense.
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
Tabu Search

• Some of the factors that affect the efficiency
  and the quality
• of the tabu search include
• The number of swaps per iteration.
• The number of moves that become tabu.
• The tenure of the tabu.
• Tabu restrictions (can restrict any swap that
  includes one
• member of a tabu pair).
• Can take into account the frequency of swaps and
  penalize
• move values between those pairs that have high
  swap
• frequencies.

21
Nested Partitions Method(Lyeuan Shi, Operations
Research, May 2000)
This method systematically partitions the
feasible region into subregions, and then
concentrates the computational effort in the most
promising region. 1) Assume that we have a most
promising region, ?. 2) Partition this region
into M subregions and aggregate the
entire surrounding region into one region. 3)
Sample each of these M1 regions and determine
the promising index for each. 4) Set ? equal to
the most promising region. Go to step 1. If the
surrounding region is found to be the best, the
algorithm backtracks to a larger region that
contains the old most promising region.
22
Nested Partitions Example
Set M 2 ?????Set ? ?3 ? 1,2,3,4,5,6,7,8
partition partition ?5 1 --gt prom index
2? ?1 1,2,3,4 --gt prom index 5 ?6 2--gt
prom index 3 ?2 5,6,7,8--gt prom index 4
?0\(?5U?6) 3,4,5,6,7,8--gt prom index
4 Set ? ?1 partition Backtrack to
? ?1 ?3 1,2 --gt prom index 5 new
partition? ?4 3,4--gt prom index 3 ?7
1,2,3 --gt prom index 5 ?2 5,6,7,8 --gt prom
index 4 ?8 4 --gt prom index 2 ?2
5,6,7,8 --gt prom index 4 continue
until a minimum is found
23
(No Transcript)
24
(No Transcript)