Evolution strategies

1
Evolution strategies

• Chapter 4

2
Evolution Strategies Quick overview

• Developed Germany in the 1970s
• Early names I. Rechenberg, H. P. Schwefel
• Typically applied to
• numerical optimisation
• Attributed features
• fast
• good optimizer for real-valued optimisation
• Based upon substantial mathematical theory
• Special
• self-adaptation of (mutation) parameters standard

3
ES technical summary table
4
Introductory example

• Task minimize f Rn ? R
• Algorithm typical ES using
• Direct mapping of vectors within Rn from
  genotypes to phenotypes
• Population size 1
• Only mutation, creating one child
• Greedy selection

5
Introductory example Pseudo-code

• Set t 0
• Create initial point xt ? x1t,,xnt ?
• REPEAT UNTIL (TERMIN.COND satisfied) DO
• Draw zi from a normal distr. for all i 1,,n
• yit xit zi
• IF f(xt) lt f(yt) THEN xt1 xt
• ELSE xt1 yt
• END IF
• Set t t1
• END DO

6
Illustration of normal distribution
7
Example illustrated

• Evolutionary strategies typically used for
  continuous parameter optimization
• Strong emphasis on mutation for creating
  off-spring
• Mutation implemented via drawing values from a
  normal distribution

8
Mutation mechanisms

• z values drawn from normal distribution N(?,?)
• mean ? is set to 0
• standard deviation ? is called mutation step
  size
• mutation step size often apart of the genotype
  (self-adaptation)

• Heuristic adaptation
• ? is varied on the fly by the 1/5 success rule
• This rule resets ? after every k iterations by
• ? ? / c if ps gt 1/5
• ? ? c if ps lt 1/5
• ? ? if ps 1/5
• where ps is the of successful mutations, 0.8 ?
  c ? 1

9
Evolutionary strategies for industrial design
Jet nozzle
Task to optimize the shape of a jet
nozzle Approach random mutations to shape
selection
10
Jet nozzle experiment Genotype
Genotype has the form z1, z2, zn Dz1, Dz2 ,
Dn Where z1 is the number of segments in the
convergent part of the nozzle z2 is the number
of segments in the divergent part Dk are the
diameters of the segments
11
Jet nozzle experiment Genotype to Phenotype
For example, the genotype sequence 32, 3, 26,
20, 8, 12, 16, 20 Represents a nozzle whose
in-coming diameter is 32mm, reducing smoothly in
four segments to 8mm, then expanding smoothly in
three segments to a final diameter of 20mm.
12
Another historical examplethe jet nozzle
experiment
Jet nozzle the movie
13
The jet nozzle experiment
(evolved shape)
14
Representation

• Chromosomes consist of three parts
• Object variables x1,,xn
• Strategy parameters
• Mutation step sizes ?1,,?n?
• Rotation angles ?1,, ?n?
• Vector ?x1,,xn? forms only part of an ES
  genotype
• Full size ? x1,,xn, ?1,,?n ,?1,, ?k ?
• where k n(n-1)/2 (no. of i,j pairs)

15
Mutation

• Main mechanism changing value by adding random
  noise drawn from normal distribution
• xi xi N(0,?)
• Key idea
• ? is part of the chromosome ? x1,,xn, ? ?
• ? is also mutated into ? (see later how)
• Thus mutation step size ? is co-evolving with
  the solution x

16
Mutate ? first

• Net mutation effect ? x, ? ? ? ? x, ? ?
• Order is important
• first ? ? ? (see later how)
• then x ? x x N(0,?)
• Rationale new ? x ,? ? is evaluated twice
• Primary x is good if f(x) is good
• Secondary ? is good if the x it created is
  good
• Reversing mutation order, this would not work

17
Mutation case 1Uncorrelated mutation with one ?

• Chromosomes ? x1,,xn, ? ?
• ? ? exp(? N(0,1))
• xi xi ? N(0,1)
• Typically the learning rate ? ? 1/ n½
• And we have a boundary rule ? lt ?0 ? ? ?0

18
Mutants with equal likelihood

• Circle mutants having the same chance to be
  created

19
Mutation case 2Uncorrelated mutation with n ?s

• Chromosomes ? x1,,xn, ?1,, ?n ?
• ?i ?i exp(? N(0,1) ? Ni (0,1))
• xi xi ?i Ni (0,1)
• Two learning rate parameters
• ? overall learning rate
• ? coordinate-wise learning rate
• Boundary rule again ?i lt ?0 ? ?i ?0

20
Mutants with equal likelihood

• Ellipse mutants having the same chance to be
  created

21
Mutation case 3Correlated mutations

• Chromosomes ? x1,,xn, ?1,, ?n ,?1,, ?k ?
• where k n (n-1)/2
• and the covariance matrix C is defined as
• cii ?i2
• cij 0 if i and j are not correlated
• cij ½ ( ?i2 - ?j2 ) tan(2 ?ij) if i and
  j are correlated
• Note the numbering / indices of the ?s

22
Mutants with equal likelihood

• Ellipse mutants having the same chance to be
  created

23
Recombination

• Creates one child
• Acts per variable / position by either
• Averaging parental values, or
• Selecting one of the parental values
• From two or more parents by either
• Using two selected parents to make a child
• Selecting two parents for each position anew

24
Recombination Terminology
25
Parent selection

• Parents are selected by uniform random
  distribution whenever an operator needs one/some
• Thus ES parent selection is unbiased - every
  individual has the same probability to be
  selected
• Note that in ES parent means whole population
  (in GAs a population member selected to undergo
  variation)

26
Survivor selection

• Applied after creating ? children from the ?
  parents by mutation and recombination
• Deterministically chops off the bad stuff
• Basis of selection is either
• The set of children only (?,?)-selection
• The set of parents and children (??)-selection

27
Survivor selection

• Often (?,?)-selection is preferred over
  (??)-selection
• Better in leaving local optima
• Better in following moving optima
• Using the (??)-selection strategy, bad ? values
  can survive in ?x,?? too long if their host x is
  very fit
• (?,?)-selection can forget

28
Self-Adaptation illustrated

• Given a dynamically changing fitness landscape
  (optimum location shifted every 200 generations)
• Self-adaptive ES is able to
• follow the optimum and
• adjust the mutation step size after every shift !

29
Self-Adaptation illustrated
Changes in the fitness values (left) and the
mutation step sizes (right)
30
Prerequisites for self-adaptation

• ? gt 1 to carry different strategies
• ? gt ? to generate offspring surplus
• (?,?)-selection to get rid of mis-adapted ?s
• Mixing strategy parameters by (intermediary)
  recombination on them

31
Robot Design Iterative Product Engineering

• Attempts to design robot morphologies based on
  principle of evolutionary design
• Extremely difficult to pre-design, especially
  for unknown environments where most environmental
  pressures are unanticipated (e.g. Mars)
• Evolution strategies can be applied to physical
  (morphology) design as well as to controller
  design sometimes both are evolved
  simultaneously

32
Monkey Robot

• Example Monkey robot design a phenotype
  (controller and body) that works for its
  tight-rope environment
• Evolution of robots morphology in a physical
  simulation together with a controller for
  traversing a rope
• Achieved momentum by changing the centre of
  gravity to the opposite arm via exploiting the
  momentum of its pendulum
• Fitness function was time taken to traverse a
  rope

33
Monkey design In simulation Iteration x
34
Monkey design In simulation Iteration x i
35
Monkey design In simulation Iteration x i j
36
Physical Version
37
Problems?

• Difficult to transfer a solution evolved in
  simulation to the real world
• E.g. Significantly higher friction between the
  claws and rope in the real world demanding a
  more rigorous swinging behavior, than in
  simulation.