Evolving a SigmaPi Network as a Network Simulator

1
Evolving a Sigma-Pi Network as a Network Simulator

• by Justin Basilico

2
Problem description

• To evolve a neural network that acts as a general
  network which can be programmed by its inputs
  to act as a variety of different networks.
• Input Another network and its input.
• Output The output of the given network on the
  input.
• Use a sigma-pi network and evolve its
  connectivity using a genetic algorithm.

3
Problem motivation

• If a network can be created to simulate other
  networks given as input, then perhaps we can
  build neural networks that act upon other neural
  networks.
• It would be interesting to see if one network
  could apply backpropagation to another network.

4
Previous work

• This problem remains largely unexplored.
• Evolutionary techniques have been applied to
  networks similar to sigma-pi networks
• Janson Frenzel, Training Product Unit Neural
  Networks with Genetic Algorithms (1993)
• Evolved product networks, which are more
  powerful than sigma-pi networks since they allow
  a variable exponent

5
Previous work

• Papers from class that are provide some
  information and inspiration
• Plate, Randomly connected sigma-pi neurons can
  form associator networks (2000)
• Belew, McInerney, Schraudolph, Evolving
  Networks Using the Genetic Algorithm with
  Connectionist Learning (1991)
• Chalmers, The Evolution of Learning An
  Experiment in Genetic Connectionism (1990)

6
Approach (Overview)

• Generate a testing set of 100 random networks to
  simulate.
• Generate initial population of chromosomes for
  sigma-pi network.
• For each generation, decode each chromosome into
  a sigma-pi network and use fitness function to
  evaluate networks fitness as a simulator using
  the testing set.

7
Approach (Overview)

• First try to simulate single-layer networks
• 2 input and 1 output units
• 2 input and 2 output units
• Then try it on a multi-layer network
• 2 input, 2 hidden, and 2 output units

8
Approach

• Input encoding
• The simulation network is given the input to the
  simulated network along with the weight values
  for the network it is simulating.
• Generate a fully-connected, feed-forward network
  with random weights along with a random input,
  then feed the input through the network to get
  the output.

9
Approach

• Input encoding
• Example

Network
bias
w50
w30
w40
w53
hidden 1
w31
output
input 1
w41
w54
hidden 2
w32
input 2
w42
Input encoding
w30
w31
w32
w40
w41
w42
w50
w53
w54
input 1
input 2
inputs
weight layer 1
weight layer 2
10
Approach

• Target output encoding
• The output that the randomly weighted network
  produces on its random input.

11
Approach

• Chromosome encoding
• Each chromosome encodes the connectivity
  (architecture) of the sigma-pi network.
• To simplify things, allow network weights to
  either be 1.0, signifying there is a connection
  there, or 0.0 signifying that there is not.
• Initialize chromosome to random string of bits.

12
Approach

• Chromosome encoding
• To encode the connectivity of a layer with m
  units to a layer with n units, use a binary
  string of length
• (m 1) ? n
• Example 011010 110001

bias
S
P
P
S
13
Approach

• Genetic algorithm
• Selection Chromosomes ranked by fitness,
  probability of selection based on rank.
• Crossover Randomly select bits in chromosome for
  crossover. (I might add in some sort of
  functional unit here.)
• Mutation Each bit in every chromosome has a
  mutation rate of 0.01.

14
Approach

• Fitness function
• Put build a sigma-pi network using the
  chromosome.
• Test the sigma-pi network on a testing set of 100
  networks.
• Better chromosomes have smaller fitness value.

15
Approach

• Fitness function
• Attempt 1 Mean squared error.
• Problem Evolved networks just always guessed 0.5
  because a sigmoid activation function was used.
• Attempt 2 Number of incorrect outputs within a
  threshold of 0.05.
• Problem We want an optimum solution with as few
  weights in the network as possible.
• Attempt 3 Use second function and also factor in
  number of 1s in chromosome.

16
Results

• So far
• Tried to train a backpropagation network to do
  simulation, but it did not work.
• Managed to evolve sigma-pi network architectures
  to simulate simple, one layer networks with
  linear units.
• Still working on simulating networks with two
  layers and with sigmoid units.

17
Results

• Network with 2 input, 1 output units and linear
  activation
• Population 100
• Optimal solution after 24 generations

input 1
input 2
bias
w30
w31
w32
P
P
P
S
18
Results

• Network with 2 input, 2 output units and linear
  activation
• Population 150
• Optimal solution after 121 generations
  (stabilizes after 486)

input 1
input 2
bias
w30
w31
w32
w40
w41
w42
P
P
P
P
P
P
S
S
19
Results (so far)

• Network with 2 input, 2 hidden, and 1 output
  units and linear activation
• Have not gotten it to work yet.
• Sigma-pi network has 3 hidden layers (11-9-5-3-1)
• Might be a problem to do with sparse connectivity
  of solution where input weights need to be
  saved for later layers.
• Potential solution Fix more of the network
  network architecture so the size of the
  chromosome is smaller.

20
Future work

• Expand evolution parameters to allow wider
  variation in the evolved networks (network
  weights, activation functions).
• Try to simulate larger networks.
• Evolve a network that implements backpropagation
• Start small with just the delta-rule for output
  and hidden units.
• Work up to a network that does full
  backpropagation.

21
More future work

• Evolve networks that create their own learning
  rules.
• Use a learning algorithm for training sigma-pi
  networks rather than evolution.
• Create a sigma-pi network that simulates other
  sigma-pi networks.