Engineering Optimization

1
Engineering Optimization

• Concepts and Applications

Fred van Keulen Matthijs Langelaar CLA
H21.1 A.vanKeulen_at_tudelft.nl
2
Summary single variable methods

• Bracketing
• Dichotomous sectioning
• Fibonacci sectioning
• Golden ratio sectioning
• Quadratic interpolation
• Cubic interpolation
• Bisection method
• Secant method
• Newton method

0th order
1st order
2nd order

• And many, many more!

3
Unconstrained optimization algorithms

• Single-variable methods
• Multiple variable methods
• 0th order
• 1st order
• 2nd order

Direct search methods
4
Test functions

• Comparison of performance of algorithms
• Mathematical convergence proofs
• Performance on benchmark problems (test functions)

5
Test functions (2)

• Quadratic function

Optimum (1, 3)
6
Random methods

• Random jumping method(random search)
• Generate random points, keep the best

7
Simulated annealing (Metropolis algorithm)

• Random method inspired by natural process
  annealing
• Heating of metal/glass to relieve stresses
• Controlled cooling to a state of stable
  equilibrium with minimal internal stresses

• Probability of internal energy change
  (Boltzmanns probability distribution function)
• Note, some chance on energy increase exists!
• S.A. based on this probability concept

8
Simulated annealing algorithm

1. Set a starting temperature T, pick a starting
   design x, and obtain f(x)
2. Randomly generate a new design y close to x

9
Simulated annealing (3)

• As temperature reduces, probability of accepting
  a bad step reduces as well

Negative
Reducing

• Accepting bad steps (energy increase) likely in
  initial phase, but less likely at the end
• Temperature zero basic random jumping method
• Variants several steps before test, cooling
  schemes,

10
Random methods properties

• Very robust work also for discontinuous /
  nondifferentiable functions
• Can find global minimum
• Last resort when all else fails
• S.A. known to perform well on several hard
  problems (traveling salesman)
• Quite inefficient, but can be used in initial
  stage to determine promising starting point
• Drawback results not repeatable

11
Cyclic coordinate search

• Search alternatingly in each coordinate direction
• Perform single-variable optimization along each
  direction (line search)

• Directions fixed can lead to slow convergence

12
Powells Conjugate Directions method

• Adjusting search directions improves convergence
• Idea replace first direction with combined
  direction of a cycle

• Guaranteed to converge in n cycles for quadratic
  functions! (theoretically)

13
(No Transcript)
14
Nelder and Mead Simplex method

• Simplex figure of n 1 points in Rn

• For better performance expansion/contraction and
  other tricks

15
Biologically inspired methods

• Popular inspiration for algorithms from
  biological processes
• Genetic algorithms / evolutionary optimization
• Particle swarms / flocks
• Ant colony methods

• Typically make use of population (collection of
  designs)
• Computationally intensive
• Stochastic nature, global optimization properties

16
Genetic algorithms

• Based on evolution theory of Darwin Survival
  of the fittest
• Objective fitness function
• Designs are encoded in chromosomal strings,
  genes e.g. binary strings

x1
x2
17
GA flowchart
Create initial population
Evaluate fitness of all individuals
18
GA population operators

• Reproduction
• Exact copy/copies of individual
• Crossover
• Randomly exchange genes of different parents
• Many possibilities how many genes, parents,
  children
• Mutation
• Randomly flip some bits of a gene string
• Used sparingly, but important to explore new
  designs

19
Population operators

• Crossover

Parent 2
Parent 1
0
1
1
0
1
0
0
1
0
1
1
0
0
0
1
1
1
0
1
0
0
1
0
1
1
0
0
1
0
1
0
1
1
0
0
0
1
0
1
1
0
0
1
0
1
1
1
0
1
1
0
0
1
0
1
1
0
0
0
1
Child 1
Child 2
20
Particle swarms / flocks

• No genes and reproduction, but a population that
  travels through the design space
• Derived from simulations of flocks/schools in
  nature
• Individuals tend to follow the individual with
  the best fitness value, but also determine their
  own path
• Some randomness added to give exploration
  properties(craziness parameter)

21
PSO algorithm

1. Initialize location and speed of individuals
   (random)
2. Evaluate fitness
3. Update best scores individual (y) and overall
   (Y)
4. Update velocity and position

22
Summary 0th order methods

• Nelder-Mead beats Powell in most cases
• Robust most can deal with discontinuity etc.
• Less attractive for many design variables (gt10)
• Stochastic techniques
• Computationally expensive, but
• Global optimization properties
• Versatile
• Population-based algorithms benefit from
  parallel computing

23
Unconstrained optimization algorithms

• Single-variable methods
• Multiple variable methods
• 0th order
• 1st order
• 2nd order

24
Steepest descent method

• Move in direction of largest decrease in f

Divergence occurs! Remedy line search
25
Steepest descent convergence

• Zig-zag convergence behavior

26
Effect of scaling

• Scaling variables helps a lot!

x2
x1
27
Fletcher-Reeves conjugate gradient method

• Based on building set of conjugate directions,
  combined with line searches
• Conjugate directions
• Conjugate directions guaranteed convergence in N
  steps for quadratic problems(recall Powell N
  cycles of N line searches)

28
Fletcher-Reeves Conjugate gradient method

• Set of N conjugate directions

(Special case orthogonal directions,
eigenvectors)
29
Conjugate directions

• Find conjugate coordinates bi

30
Conjugate directions (2)

• Optimization by line searches along conjugate
  directions will converge in N steps (or less)

31
But how to obtain conjugate directions?

• How to generate conjugate directions with only
  gradient information?
• Start with steepest descent direction

32
Conjugate directions (3)

• Condition for conjugate direction

33
Eliminating A (cont.)

• Result

34
Why that last step?

• By Fletcher-Reeves starting from Polak-Rebiere
  version

35
Three CG variants

• For general non-quadratic problems, three
  variants exist that are equivalent in the
  quadratic case
• Hestenes-Stiefel

Generally bestin most cases
36
CG practical

1. Start with abritrary x1
2. Set first search direction
3. Line search to find next point
4. Next search direction
5. Repeat 3
6. Restart every (n1) steps, using step 2

37
CG properties

• Theoretically converges in N steps or less for
  quadratic functions
• In practice
• Non-quadratic functions
• Finite line search accuracy
• Round-off errors

38
Application to mechanics (FE)

• Structural mechanicsQuadratic function!

• Simple operations on element level. Attractive
  for large N!