Title: Controlled Observations of the Genetic Algorithm in a Changing Environment
1Controlled Observations of the Genetic Algorithm
in a Changing Environment
Case Studies Using the Shaky Ladder Hyperplane
Defined Functions
- William Rand
- Computer Science and Engineering
- Center for the Study of Complex Systems
- wrand_at_umich.edu
2Overview
- Introduction
- Motivation, GA, Dynamic Environments, Framework,
Measurements
- Shaky Ladder Hyperplane-Defined Functions
- Description and Analysis
- Varying the Time between Changes
- Performance, Satisficing and Diversity Results
- The Effect of Crossover
- Experiments with Self-Adaptation
- Future Work and Conclusions
3Motivation
- Despite years of great research examining the
GA, more work still needs to be done, especially
within the realm of dynamic environments
- Approach
- Applications GA works in many different
environments, particular results
- Theory Places many limitations on results
- Middle Ground Examine a realistic GA on a set of
constructed test functions, Systematic Controlled
Observation
- Benefits
- Make recommendations to application
practitioners
- Provide guidance for theoretical work
4What is a GA?
Evaluative Mechanism
Population of Solutions
Inheritance with Variation
John Holland, Adaptation in Natural and
Artificial Systems, 1975
5Dynamic Environments
- GAs are a restricted model of evolution
- Evolution is an inherently dynamic system, yet
researchers traditionally apply GAs to static
problems
- If building blocks exist within a problem
framework, the GA can recombine those solutions
to solve problems that change in time
- Application examples include Job scheduling,
dynamic routing, and autonomous agent control
- What if we want to understand how the GA works in
these environments?
- Applications are too complicated to comprehend
all of the interactions, we need a test suite for
systematic, controlled observation
6Measures of Interest
- Robustness
- How indicative future performance is the current
performance?
- Best Robustness Fitness of current best divided
by fitness of previous generations best
individual
- Average Robustness Current population fitness
avg. divided by the avg. for the previous
generation
- Diversity
- Measure of the variation of the genomes in the
population
- Best Diversity Avg. Hamming distance between
genomes of best individuals across runs
- Average Diversity Avg. across runs of avg.
Hamming distance of whole population
- Performance
- How well the system solves an objective function
- Best Performance - Avg. across runs of the
fitness of the best ind.
- Avg. Performance Avg. across runs of the
fitness of the avg. ind.
- Satisficability
- Ability of the system to achieve a predetermined
criteria
- Best Satisficability - Fraction of runs where the
best solution exceeds a threshold
- Average Satisficability Avg. across runs of
fraction of population to exceed threshold
7Hyperplane Defined Functions
- HDFs were designed by John Holland, to model
the way the individuals in a GA
search
- In HDFs building blocks are described formally by
schemata
- If search space is binary strings then schemata
are trinary strings (0, 1, wildcard)
- Building blocks are schemata with a positive
fitness contribution
- Combine building blocks to create higher level
building blocks and reward the individual more
for finding them
- Potholes are schemata with a negative fitness
contribution
John Holland, Cohort Genetic Algorithms,
Building Blocks, and Hyperplane-Defined
Functions, 2000
8Shaky Ladder HDFs
- Shaky ladder HDFs place 3 restrictions on HDFs
- Elementary schemata do not conflict with each
other
- Potholes have limited costs
- Final schema is union of elementary schemata
- Gaurantees any string which matches the final
schema is an optimally valued string
- Shaking the ladder involves changing intermediate
schemata
9Three Variants
- Cliffs Variant - All three groups of the fixed
schemata are used to create the intermediate
schemata
- Smooth Variant - Only elementary schemata are
used to create intermediate schemata
- Weight Variant - The weights of the ladder are
shaken instead of the form
10Analysis of the Sl-hdfs
- The Sl-hdfs were devised to resemble an
environment where there are regularities and a
fixed optima
- There are two ways to show that they have these
properties
- Match the micro-level componentry
- Carry out macro-analysis
- Standard technique is the autocorrelation of a
random walk
11Mutation Landscape
12Crossover Landscape
13Time Between Shakes Experiment
- Vary t? which is the time between shakes
- See what effect this has on the performance of
the best individual in the current generation
14Cliffs Variant
15Smooth Variant
16Weight Variant
17Results of Experiment
- t? 1801, 900 Improves performance,?Premature
Convergence prevents finding optimal strings
- t? 1 Outperforms static environment,
intermediate schemata provide little guidance, 19
runs find an optimal string
- t? 25, 100 Best performance, tracks and
adapts to changes in environment, intermediate
schemata provide guidance but not a dead end, 30
runs find optimal strings - Smoother variants perform better early on, but
then the lack of selection pressure prevents them
from finding the global optimum
18Comparison of Threshold Satisficability
19Cliffs Variant Diversity Results
20Smooth Variant Diversity Results
21Weight Variant Diversity Results
22Discussion of Diversity Results
- Initially thought diversity would decrease as
time went on and the system converged, and that
shakes would increase diversity as the GA
explored the new solution space - Neutral mutations allow founders to diverge
- A ladder shake decreases diversity because it
eliminates competitors to the new best
subpopulations
- In the Smooth and Weight variants Diversity
increases even more due to lack of selection
pressure
- In the Weight variant you can see Diversity
leveling off as the population stabilizes around
fit, but not optimal individuals
23Crossover Experiment
- The Sl-hdf are supposed to explore the use of the
GA in dynamic environments
- The GAs most important operator is crossover
- Therefore, if we turn crossover off the GA should
not be able to work as well in the sl-hdf
environments
- That is exactly what happens
- Moreover crossover has a greater effect on the GA
operating in the Weight variant due to the short
schemata
24Cliffs Variant Crossover Results
25Smooth Variant Crossover Results
26Weight Variant Crossover Results
27Self-Adaptation
- By controlling mutation we can control the
balance between exploration and exploitation
which is especially useful in a dynamic
environment - Many techniques have been suggested
hypermutation, variable local search, and random
immigrants
- Bäck proposed the use of self-adaptation in
evolutionary strategies and later in GAs (92)
- Self-adaptation encodes a mutation rate within
the genome
- The mutation rate becomes an additional search
problem for the GA to solve
28The Experiments
29Performance Results (1/2)Generation 1800 Best
Individual Over 30 Runs
30Performance Results (2/2)Generation 1800 Best
Individual Over 30 Runs
31Mutation Rates
32Discussion of Self-Adaptation Results
- Self-adaptation fails to improve the balance
between exploration and exploitation
- Tragedy of the Commons - it is to the benefit
of each individual to have a low mutation rate,
even though a higher average mutation rate is
beneficial to the whole population - Seeding with known good material does not
always increase performance
- Some level of mutation is always good
33Future Work
- Further Exploration of sl-hdf parameter space
- Schema Analysis
- Analysis of local optima
- What levels are attained when?
- Analysis of the sl-hdf landscape
- Population based landscape analysis
- Other dynamic analysis
- Examination of
- Other mechanisms to augment the GA
- Meta-GAs, hypermutation, multiploidy
- Co-evolution of sl-hdf's with solutions
- Combining GAs with ABMs to model ecosystems
34Applications
- Other Areas of Evolutionary Computation
- Coevolution
- Other Evolutionary Algorithms
- Computational Economics
- Market Behavior and Failures
- Generalists vs. Specialists
- Autonomous Agents
- Software agents, Robotics
- Evolutionary Biology
- Phenotypic Plasticity, Baldwinian Learning
35Conclusions
- Systematic, Controlled Observation allows us to
gather regularities about an artificial system
that is useful to both practitioners and
theoreticians - The sl-hdf's provide and the three variants
presented are a useful platform for exploring the
GA in dynamic environments
- Standard autocorrelation fails to completely
describe some landscapes and some dynamism
- Intermediate rates of change provide a better
environment at times by preventing premature
convergence
- Self-adaptation is not always successful,
sometimes it is better to explicitly control GA
parameters
36Acknowledgements
Jason Atkins Dmitri Dolgov Anna Osepayshvili
Jeff Cox Dave Orton Bil Lusa Dan Reeves Jane C
oggshall Brian Magerko Bryan Pardo Stefan Nikle
s Eric Schlegel TJ Harpster Kristin Chiboucas
Cibele Newman Julia Clay Bill Merrill Eric Lars
on Josh Estelle
- John Holland
- Rick Riolo
- Scott Page, John Laird, and Martha Pollack
- Jürgen Branke
- CSCS
- Carl Simon
- Howard Oishi
- Mita Gibson
- Cosma Shalizi
- Mike Charter the admins
- Lori Coleman
- Sluce Research Group
- Dan Brown, Moira Zellner, Derek Robinson
- My Parents and Family
- RR-Group
- Robert Lindsay
- Ted Belding
- Chien-Feng Huang
- Lee Ann Fu
- Boris Mitavskiy
- Tom Bersano
- Lee Newman
- SFI, CSSS, GWE
- Floortje Alkemade
- Lazlo Guylas
- Andreas Pape
- Kirsten Copren
- Nic Geard
- Igor Nikolic
- Ed Venit
- Santiago Jaramillo
- Toby Elmhirst
- Amy Perfors
Friends Brooke Haueisen Kat Riddle Tami Ursem
Kevin Fan Jay Blome Chad Brick Ben
Larson Mike Curtis Beckie Curtis Mike Geiger
Brenda Geiger William Murphy Katy Luchini Dirk
Colbry
- CWI
- Han La Poutré
- Tomas Klos
- EECS
- William Birmingham
- Greg Wakefield
- CSEG
And Many, Many More
37Any Questions?
38My Contributions
- Shaky Ladder Hyperplane Defined Functions
- Three Variants
- Description of Parameter Space to be explored
- Work on describing Systematic, Controlled
Observation Framework
- Initial experiments on sl-hdf
- Crossover Results on variants of the hdf
- Autocorrelation analysis of the hdf and sl-hdf
- Exploration of Self-Adaptation in GAs when it
fails
- Suite of Metrics to better understand GAs in
Dynamic Environments
- Proposals of how to extend the results to
Coevolution
39Motivation
- Despite decades of great research, there is more
work that needs to be done in understanding the
GA
- Performance metrics are not enough to explain the
behavior of the GA, but that is what is reported
in most experiments
- What other measures could be used to describe the
run of a GA in order to gain a fuller
understanding of how the GA behaves?
- The goal is not to understand the landscape or to
classify the performance of particular variations
of the GA
- Rather the goal is to develop a suite of measures
that help to understand the GA via systematic,
controlled observations
40Exploration vs. Exploitation
- A classic problem in optimization is how to
maintain the balance
between exploration
and exploitation - k-armed bandit problem
- If we are allowed a limited number of trials at a
k-armed bandit what is the best way to allocate
those trials in order to maximize our overall
utility? - Given finite computing resources what is the best
way to allocate our computational power to
maximize our results?
- Classical solution Allocate exponential trials
to the best observed distribution based on
historic outcomes
Dubins, L.E. and Savage, L.J. (1965). How to
gamble if you must. McGraw Hill, New York.
Re-published as Inequalities for stochastic
processes. Dover, New York (1976).
41The Genetic Algorithm
- Generate a population of solutions to the search
problem at random
- Evaluate this population
- Sort the population based on performance
- Select a part of the population to make a new
population
- Perform mutation and recombination to fill out
the new population
- Go to step 2 until time runs out or performance
criteria is met
42The Environments
- Static Environment Hyperplane-defined function
(hdf)
- Dynamic Environment New hdf's are selected from
an equivalence set at regular intervals
- Coevolving Environment A separate problem-GA
controls which hdf's the solution-GA faces every
generation
43Dynamics and the Bandit(Like Smoky and the
Bandit only without Jackie Gleason)
- Now what if the distributions underlying the
various arms changes in time?
- The balance between exploration and exploitation
would also have to change in time
- This presentation will attempt to examine one way
to do that and why the mechanism presented fails
44Qualities of Test Suites
- Whitley (96)
- Generated from elementary Building Blocks
- Resistant to hillclimbing
- Scalable in difficulty
- Canonical in form
- Holland (00)
- Generated at random, but not reverse-engineerable
- Landscape-like features
- Include all finite functions in the limit
45Building Blocks and GAs
- GAs combine building blocks to find the solution
to a problem
- Different individuals in a GA have different
building blocks, through crossover they merge
- This can be used to define any possible function
Car
Wheel
Engine
46HDF Example
Building Block Set b1 111 1 b2 00
1 b3 1 1 b4 0 1 b12 11
001 1 b23 001 1 b123 110011 1
b1234 1100101 1
Sample Evaluations f(100111) b3 1 f(1111111)
b1b3-p13 1.5 f(1000100) b2 b3 b23 3
f(1100111) b1 b2 b3 b12 b23 b123 -
p12 p13 p2312 4.5
Potholes p121101 -0.5 p13 1111 -0.5
p2312 1 0011 -0.5
47Hyperplane-defined Functions
- Defined over the set of all binary strings
- Create an elementary level building block set
defined over the set of strings of the alphabet
0, 1,
- Create higher level building blocks by combining
elementary building blocks
- Assign positive weights to all building blocks
- Create a set of potholes that incorporate parts
of multiple elementary building blocks
- Assign the potholes negative weights
- A solution string matches a building block or a
pothole if it matches the character of the
alphabet or if the building block has a '' at
that location
48Problems with the HDFs
- Problems with HDFs for systematic study in
dynamic environments
- No way to determine optimum value of a random
HDF
- No way to create new HDFs based on old ones
- Because of this there is no way to specify a
non-random dynamic HDF
49Creating an sl-hdf
- Generate a set of e non-conflicting elementary
schemata of order o (8), and of string length l
(500), set fitness contribution u(s) (2)
- Combine all elementary schemata to create
highest-level schemata, and set fitness
contribution (3)
- Create a pothole for each elementary schemata,
by copying all defining bits, plus some from
another elementary schemata probabilistically (p
0.5), and set fitness contribution (-1) - Generate intermediate schemata by combining
random pairs of elementary schemata to create e /
2 second level schemata
- Repeat (4) for each level until the number of
schemata to be generated for the next level is 1
- To generate a new sl-hdf from the same
equivalence set delete the previous intermediate
schemata and repeat steps (4) and (5)
50Mutation Blowup
51Crossover Blowup
52Measures of Interest
- Average Fitness average performance of the
system over time
- Robustness ability of the system to maintain
steady state performance
- Satisficability ability of the system to
maintain performance above a certain level
- Diversity difference between solutions that the
system is currently examining
53Cliffs Variant Performance Results
54Smooth Variant Performance Results
55Weight Variant Performance Results
56Cliffs Variant Robustness Results
57Smooth Variant Robustness Results
58Weight Variant Robustness Results
59Cliffs Variant Satisficability Results
60Smooth Variant Satisficability Results
61Weight Variant Satisficability Results
62Crossover Results Cliffs Variant
63Crossover Results Smooth Variant
64Crossover Results Weight Variant
65Why Bäcks Mechanism?
- Does not require external knowledge
- Allows the GA to choose any mutation rate
- Allows control between exploration and
exploitation does not force one or the other
- First order approximation of self-adaptive
mutation mechanisms in haploid organisms
- Bäck showed self-adaptation to be successful
66SA Results Smooth Variant (1/2)Generation 1800
Best Individual Over 30 Runs
67SA Results Smooth Variant (2/2)Generation 1800
Best Individual Over 30 Runs
68Mutation Rates - Smooth Variant
69SA Results Weight Variant (1/2)Generation 1800
Best Individual Over 30 Runs
70SA Results Weight Variant (2/2)Generation 1800
Best Individual Over 30 Runs
71Mutation Rates - Weight Variant
72Discussion of Results
- Local optima drive mutation rates down
- Hard to recover from low mutation rates
- Why was self-adaptation successful in Bäcks
experiments?
- Most of his experiments were unimodal
- Strong selection pressure
- Bäcks problems are amenable to hill-climbing
73Additional ExperimentsMechanisms for increasing
GA performance in dynamic environments
- Adapted Evolution an external function based on
fitness or diversity controls evolutionary
parameters
- Meta-GAs one GA controls the evolutionary
parameters for another GA
- Multiploidy the use of dominant and recessive
genes to maintain a memory of previous solutions
- Niching increasing diversity by decreasing the
fitness of simliar solutions
74Evolutionary Biology
- Coevolution and multiple species evolution
- Exploration versus Exploitation
- Generalists versus Specialists
- Phenotypic Plasticity
- Baldwinian Learning
- Evolution of Evolvability
75Autonomous Agent Control
- How do you create an autonomous agent which can
adapt to changes in its environment?
- What if other agents are coevolving and
interfacing with your agent?
- Is it possible to automatically determine when to
switch strategies?
- Examples robot control, trading agents,
personal assistant agents
76Computational Economics
- Many of the same questions as Evolutionary
Biology
- Exploration vs. Exploitation
- Generalists vs. Specialists
- Many of the same questions as Autonomous Agent
Control
- Coevolution of agents
- When to switch strategies
- Market behavior and failures
77Future and Other Work in Dynamic Environments
- Test suite development and the behavior of a
simple GA in a dynamic environment
(EvoStoc-2005)
- Diversity of solutions in dynamic environments
(EvoDop-2005)
- Explore other ways to balance exploration and
exploitation
- Hypermutation, Multiploidy, and Meta-GAs
- Schematic Analysis
- Analysis of the sl-hdf landscape
- Co-evolution of sl-hdf's with solutions
- Combining GAs with a ABMs to model ecosystems
78Standard Explorations
- Vary td which is the time between shakes
- See what effect this has on the performance of
the best individuals
- In the past we explored a simple GA
- Fixed mutation of one bit out of a thousand
(0.001)
- One-point crossover creates 70 of the new
population
- Cloning creates the other 30
- Population size of 1000
- Selection using a tournament of size 3
79The Environments
- Static Environment Hyperplane-defined function
(hdf)
- Dynamic Environment New hdf's are selected from
an equivalence set at regular intervals
- Coevolving Environment A separate problem-GA
controls which hdf's the solution-GA faces every
generation
80Exploration and Exploitation in Dynamic
Environments
- Ideal system might not have the same behavior as
a static system
- Increase exploration during times of change
- Increase exploitation during times of quiescence
- The mutation rate is one control of this
behavior
- Thus a dynamic mutation rate might allow the
system to better adapt to changes
- Many techniques hypermutation, variable local
search, and random immigrants
81Additional ExperimentsMechanisms for increasing
GA performance in dynamic environments
- Individual Self-Adaptation individuals can
adjust their own mutation rates
- Adapted Evolution an external function based on
fitness or diversity controls evolutionary
parameters
- Meta-GAs one GA controls the evolutionary
parameters for another GA
- Multiploidy the use of dominant and recessive
genes to maintain a memory of previous solutions
- Niching increasing diversity by decreasing the
fitness of simliar solutions
82Evolutionary Biology
- Coevolution and multiple species evolution
- Exploration versus Exploitation
- Generalists versus Specialists
- Phenotypic Plasticity
- Baldwinian Learning
- Evolution of Evolvability
83Autonomous Agent Control
- How do you create an autonomous agent which can
adapt to changes in its environment?
- What if other agents are coevolving and
interfacing with your agent?
- Is it possible to automatically determine when to
switch strategies?
- Examples robot control, trading agents,
personal assistant agents
84Computational Economics
- Many of the same questions as Evolutionary
Biology
- Exploration vs. Exploitation
- Generalists vs. Specialists
- Many of the same questions as Autonomous Agent
Control
- Coevolution of agents
- When to switch strategies
- Market behavior and failures
85Different Ways To Examine Behavior
- Extreme vs. Wholistic behavior the best /
worst a system can do vs. the behavior of the
whole population
- Within vs. Across Runs Are we more interested
in how well the system will do within a
particular run or across many runs?
- Fitness vs. Composition related Fitness is an
indication of how well an individual is doing in
the population, but one could also measure
characteristics of the population that are not
related to fitness
86Discussion of Performance Results
- A GA operating on a regular changing landscape
will initially underperform but will eventually
outperform a GA operating on a static landscape
- Working Hypothesis The static landscape results
in premature convergence, whereas shaking the
landscape forces the GA to explore multiple
solution sub-spaces - The average performance falls farther after a
shake than the best performance, this is because
the best performance loss is mitigated by
individuals that perform well in the new
environment
Rand, W. and R. Riolo, Shaky Ladders,
Hyperplane-Defined Functions and Genetic
Algorithms Systematic Controlled Observation in
Dynamic Environments, EvoStoc-2005
87Discussion of Satisficability Results
- Both the static environment and the regularly
changing environment appear to operate in a
similar fashion despite the better overall
performance of the changing environment - Working Hypothesis Most basic building blocks
are found at roughly the same rate, the dynamic
environment is better at finding intermediate
building blocks - Average Satisficability closely mirrors Best,
despite the fact that Average is within instead
of across runs
88Discussion of Robustness Results
- Static environment constantly maintains
robusteness, except for a few deleterious
mutations
- The robustness measure presented here indicates
that fitness changes are a good indication of
change
- Greatest change in scores in the middle
generations, because the GA is concentrating on
exploring intermediate schemata
89Conclusions
- These measures help to provide a better
understanding of how the GA works in dynamic
environments
- By using these measurements in combination with
each other a great understanding can be gained
than by exploring any one of them individually
- This paper is one step toward understanding the
behavior of GAs through systematic, controlled
experiments
90Future Work
- Further explorations of the parameter space of
the sl-hdfs ( of elementary schemata, string
length)
- Investigations into the difficulty level of the
sl-hdf's
- Examining diversity of schemata present in the
populations in each run