Evolving Multimodal Networks for Multitask Games

1
Evolving Multimodal Networks for Multitask Games

• Jacob Schrum schrum2_at_cs.utexas.edu
• Risto Miikkulainen risto_at_cs.utexas.edu
• University of Texas at Austin
• Department of Computer Science

2

• Evolution in videogames
• Automatically learn interesting behavior
• Complex but controlled environments
• Stepping stone to real world
• Robots
• Training simulators
• Complexity issues
• Multiple contradictory objectives
• Multiple challenging tasks

3
Multitask Games

• NPCs perform two or more separate tasks
• Each task has own performance measures
• Task linkage
• Independent
• Dependent
• Not blended
• Inherently multiobjective

4
Test Domains

• Designed to study multimodal behavior
• Two tasks in similar environments
• Different behavior needed to succeed
• Main challenge perform well in both

5
Front/Back Ramming

• Same goal, opposite embodiments

• Front Ramming
• Attack w/front ram
• Avoid counterattacks

• Back Ramming
• Attack w/back ram
• Avoid counterattacks

6
Predator/Prey

• Same embodiment, opposite goals

• Predator
• Attack prey
• Prevent escape

• Prey
• Avoid attack
• Stay alive

7
Multiobjective Optimization
High health but did not deal much damage

• Game with two objectives
• Damage Dealt
• Remaining Health
• A dominates B iff A is
  strictly better in
  one
  objective and at least
  
  as good in others
• Population of points
  not dominated are best
  
  Pareto Front
• Weighted-sum provably
  incapable of capturing
  non-convex front

Tradeoff between objectives
Dealt lot of damage, but lost lots of health
8
NSGA-II

• Evolution natural approach for finding optimal
  population
• Non-Dominated Sorting Genetic Algorithm II
• Population P with size N Evaluate P
• Use mutation to get P size N Evaluate P
• Calculate non-dominated fronts of P È P size
  2N
• New population size N from highest fronts of P È
  P

K. Deb et al. A Fast and Elitist Multiobjective
Genetic Algorithm NSGA-II. Evol. Comp. 2002
9
Constructive Neuroevolution

• Genetic Algorithms Neural Networks
• Build structure incrementally (complexification)
• Good at generating control policies
• Three basic mutations (no crossover used)

Perturb Weight
Add Connection
Add Node
10
Multimodal Networks (1)

• Multitask Learning
• One mode per task
• Shared hidden layer
• Knows current task
• Previous work
• Supervised learning context
• Multiple tasks learned quicker than individual
• Not tried with evolution yet

R. A. Caruana, "Multitask learning A
knowledge-based source of inductive bias" ICML
1993
11
Multimodal Networks (2)
Starting network with one mode

• Mode Mutation
• Extra modes evolved
• Networks choose mode
• Chosen via preference neurons
• MM Previous
• Links from previous mode
• Weights 1.0
• MM Random
• Links from random sources
• Random weights
• Supports mode deletion

MM(R)
MM(P)
12
Experiment

• Compare 4 conditions
• Control Unimodal networks
• Multitask One mode per task
• MM(P) Mode Mutation Previous
• MM(R) Mode Mutation Random Delete Mutation
• 500 generations
• Population size 52
• Player behavior scripted
• Network controls homogeneous team of 4

13
MO Performance Assessment

• Reduce Pareto front to single number
• Hypervolume of
  dominated region
• Pareto compliant
• Front A dominates
  front B implies
  HV(A) gt HV(B)
• Standard statistical
  comparisons of
  average HV

14
(No Transcript)
15
Front/Back Ramming Behaviors
Multitask
MM(R)
16
(No Transcript)
17
Predator/Prey Behaviors
Multitask
MM(R)
18
Discussion (1)

• Front/Back Ramming
• Control lt MM(P), MM(R) lt Multitask
• Multiple modes help
• Explicit knowledge of task helps

19
Discussion (2)

• Predator/Prey
• MM(P), Control, Multitask lt MM(R)
• Multiple modes not necessarily helpful
• Disparity in relative difficulty of tasks
• Multitask ends up wasting effort
• Mode deletion aids search for one good mode

20
How To Apply

• Multitask good if
• Task division known, and
• Tasks are comparably difficult
• Mode mutation good if
• Task division is unknown, or
• Obvious task division is misleading

21
Future Work

• Games with more tasks
• Does method scale?
• Control mode bloat
• Games with independent tasks
• Ms. Pac-Man
• Collect pills while avoiding ghosts
• Eat ghosts after eating power pill
• Games with blended tasks
• Unreal Tournament 2004
• Fight while avoiding damage
• Fight or run away?
• Collect items or seek opponents?

22
Conclusion

• Domains with multiple tasks are common
• Both in real world and games
• Multimodal networks improve learning in multitask
  games
• Will allow interesting/complex behavior to be
  developed in future

23
Questions?
Jacob Schrum schrum2_at_cs.utexas.edu Risto
Miikkulainen risto_at_cs.utexas.edu University of
Texas at Austin Department of Computer Science