CAP6938 Neuroevolution and Artificial Embryogeny Competitive Coevolution

1
CAP6938Neuroevolution and Artificial
EmbryogenyCompetitive Coevolution

• Dr. Kenneth Stanley
• February 20, 2006

2
ExampleI Want to Evolve a Go Player

• Go is one of the hardest games for computers
• I am terrible at it
• There are no good Go programs either
  (hypothetically)
• I have no idea how to measure the fitness of a Go
  player
• How can I make evolution solve this problem?

3
Generally Fitness May Be Difficult to Formalize

• Optimal policy in competitive domains unknown
• Only winner and loser can be easily determined
• What can be done?

4
Competitive Coevolution

• Coevolution No absolute fitness function
• Fitness depends on direct comparisons with other
  evolving agents
• Hope to discover solutions beyond the ability of
  fitness to describe
• Competition should lead to an escalating arms race

5
The Arms Race
6
The Arms Race is an AI Dream

• Computer plays itself and becomes champion
• No need for human knowledge whatsoever
• In practice, progress eventually stagnates
  (Darwen 1996 Floreano and Nolfi 1997 Rosin and
  Belew 1997)

7
So Who Plays Against Whom?

• If evaluation is expensive, everyone cant play
  everyone
• Even if they could, a lot of candidates might be
  very poor
• If not everyone, who then is chosen as
  competition for each candidate?
• Need some kind of intelligent sampling

8
Challenges with Choosing the Right Opponents

• Red Queen Effect Running in Circles
• A dominates B
• C dominates B
• A dominates B
• Overspecialization
• Optimizing a single skill to the neglect of all
  others
• Likely to happen without diverse opponents in
  sample
• Several other failure dynamics

9
Heuristic in NEATUtilize Species Champions
Each individual plays all the species champions
and keeps a score
10
Hall of Fame (HOF)(Rosin and Belew 1997)

• Keep around a list of past champions
• Add them to the mix of opponents
• If HOF gets too big, sample from it

11
More RecentlyPareto Coevolution

• Separate learners and tests
• The tests are rewarded for distinguishing
  learners from each other
• The learners are ranked in Pareto layers
• Each test is an objective
• If X wins against a superset of tests that Y wins
  again, then X Pareto-dominates Y
• The first layer is a nondominated front
• Think of tests as objectives in a multiobjective
  optimization problem
• Potentially costly All learners play all tests

De Jong, E.D. and J.B. Pollack (2004). Ideal
Evaluation from Coevolution Evolutionary
Computation, Vol. 12, Issue 2, pp. 159-192,
published by The MIT Press.
12
Choosing Opponents Isnt Everything

• How can new solutions be continually created that
  maintain existing capabilities?
• Mutations that lead to innovations could
  simultaneously lead to losses
• What kind of process ensures elaboration over
  alteration?

13
Alteration vs. Elaboration
14
Answer Complexification

• Fixed-length genomes limit progress
• Dominant strategies that utilize the entire
  genome must alter and thereby sacrifice prior
  functionality
• If new genes can be added, dominant strategies
  can be elaborated, maintaining existing
  capabilities

15
Test Domain Robot Duel

• Robot with higher energy wins by colliding with
  opponent
• Moving costs energy
• Collecting food replenishes energy
• Complex task When to forage/save energy,
  avoid/pursue?

16
Robot Neural Networks
17
Experimental Setup

• 13 complexifying runs, 15 fixed-topology runs
• 500 generations per run
• 2-population coevolution with hall of fame (Rosin
  Belew 1997)

18
Performance is Difficult to Evaluate in
Coevolution

• How can you tell if things are improving when
  everything is relative?
• Number of wins is relative to each generation
• No absolute measure is available
• No benchmark is comprehensive

19
Expensive Method Master Tournament(Cliff and
Miller 1995 Floreano and Nolfi 1997)

• Compare all generation champions to each other
• Requires n2 evaluations
• An accurate evaluation may involve e.g. 288 games
• Defeating more champions does not establish
  superiority

20
Strict and Efficient Performance Measure
Dominance Tournament (Stanley Miikkulainen
2002)
21
Result Evolution of Complexity

• As dominance increases so does complexity on
  average
• Networks with strictly superior strategies are
  more complex

22
Comparing Performance
23
Summary of Performance Comparisons
24
The Superchamp
25
Cooperative Coevolution

• Groups attempt to work with each other instead of
  against each other
• But sometimes its not clear whats cooperation
  and whats competition
• Maybe competitive/cooperative is not the best
  distinction?
• Newer idea Compositional vs. test-based

26
Summary

• Picking best opponents
• Maintaining and elaborating on strategies
• Measuring performance
• Different types of coevolution
• Advanced papers on coevolution

Ideal Evaluation from Coevolution by De Jong,
E.D. and J.B. Pollack (2004)Monotonic Solution
Concepts in Coevolution by Ficici, Sevan G.
(2005)
27
Next Topic Real-time NEAT (rtNEAT)

• Simultaneous and asynchronous evaluation
• Non-generational
• Useful in video games and simulations
• NERO Video game with rtNEAT

-Shorter symposium paper Evolving Neural Network
Agents in the NERO Video Game by Kenneth O.
Stanley and Risto Miikkulainen (2005)-Optional
journal (longer, more detailed) paper Real-time
Neuroevolution in the NERO Video Game by Kenneth
O. Stanley and Risto Miikkulainen (2005)
-http//Nerogame.org -Extra coevolution papers
Homework due 2/27/06 Working genotype to
phenotype mapping. Genetic representation
completed. Saving and loading of genome file I/O
functions completed. Turn in summary, code, and
examples demonstrating that it works.
28
Project Milestones (25 of grade)

• 2/6 Initial proposal and project description
• 2/15 Domain and phenotype code and examples
• 2/27 Genes and Genotype to Phenotype mapping
• 3/8 Genetic operators all working
• 3/27 Population level and main loop working
• 4/10 Final project and presentation due (75 of
  grade)