Hybrid Approaches

1
Hybrid Approaches

• Genetic algorithms and artificial neural networks
  can combined in various ways.
• These are sometimes referred to as hybrid
  approaches

2
Hybrid Approaches

• Example 1
• using a GA to specify' a high performing
  multi-layer perceptron (MLP).
• i.e. specify the network architecture and
  operating parameters.
• compare results with the heuristics reported in
  the literature.
• compare results across different problem domains.
• Issues
• the MLP needs to be represented as a chromosome.
• each gene represents an operating parameter (e.g
  learning rate, momentum, epoch, activation
  function etc) or some component of architecture
  number of layers, number of neurons in a layer
  etc
• the gene values represent values for the
  operating parameters and for the architecture
  components (discrete values).

3
Hybrid Approaches

• an example chromosome for a MLP applied to the
  heart disease problem domain

4
Hybrid Approaches

• The gene values are used after some decoding
  as the parameters for the multi layer perceptron.
• eg 0 21 14 3 4 14 0 5
• In the above example the value for gene3 (i.e.
  14) is not used because there is only one hidden
  layer (from gene1). This chromosome specifies
  the following network
• one hidden layer 0.
• 22 21 1 neurons in the hidden layer.
• gene3 value 14 not used.
• learning rate 0.15 3 third value in learning
  rate set
• momentum 0.5 4 fourth value in the momentum
  set
• Epoch size training set size/(141)
• Sigmoid activation function 0
• initial weight range 0.05 5 fifth value in
  set

5
Hybrid Approaches

• Some things are kept constant or were problem
  domain dependent
• eg
• number of trials
• number of input neurons
• number of output neurons
• Evaluation of fitness
• the chromosome is decoded into a set of
  parameters to pass to BP
• performance measures ( correct on training and
  testing sets) are returned and used to evaluate
  fitness

6
Hybrid Approaches

• where Ts best test percentage correct, Tt
  best training percentage correct and k is some
  very small constant (eg 0.001)
• fitness is being minimised
• Problem domains
• Iris data set
• Heart disease data set
• Breast cancer data
• US congressional voting records

7
Hybrid Approaches

• Some results
• networks with one hidden layer were favoured.
• low momentum and learning rates (lt 0.5, and
  0.1- 0.25) were favoured across the datasets.
• the hyperbolic tangent activation function was
  favoured in three of the four datasets.
• when the logistic function was favoured (in the
  voting dataset) a higher learning rate was
  favoured (0.6).

8
Hybrid Approaches

• Example 2
• Using a Genetic Algorithm for NN learning
• ie for weight optimization

Bias 0.7
1
5
W13 3.0
W35 0.6
Bias 4.0
3
W14 -1.0
W36 0.2
W23 -0.5
W45 -0.7
2
6
4
W46 1.4
W24 2.0
Bias 5.0
Bias 0.9
9
Hybrid Approaches

• or as weight matrix

each column of the table gives the weighted
connections to a neuron (including the bias
weights) eg in bold for column 3
10
Hybrid Approaches

• how might we encode this as a chromosome?
• there are 12 weights therefore we could have a 12
  gene chromosome, where the chromosome value is
  the weight
• zeros are ignored

For application of the genetic operators
crossover and mutation the genes are grouped
by neuron i.e.
Also note that the NN architecture is fixed, so
the de-coding of the chromosome is the same
for all
11
Hybrid Approaches

• Fitness?
• the chromosome can be decoded, a training
  example(s) presented and an error evaluated.
• the lower the error the more fit the network

12
Hybrid Approaches

• Genetic operators?
• crossover
• a child from two parents
• all weights associated with a given neuron could
  be exchanged
• e.g.

parent 1
parent 2
crossover only shifts the existing weights around
the network
may give, exchanging weights associated with
neuron 4 and 5
13
Hybrid Approaches

• mutation
• mutation may alter an individual weight
• if a weight is mutated as determined by the
  mutation rate and a randomly generated number
  then a small value is added eg randomly
  generated between -1 to 1
• in the above example, if weight 3 mutates and has
  0.5 added to it , then the chromosome becomes

14
Hybrid Approaches

• Example 3
• Using a Genetic Algorithm for input selection
• problem many possible attributes could
  contribute to the classification.
• which subset is required for effective training?
• again an evolutionary approach could be used

15
Hybrid Approaches

• e.g
• x features (attributes)
• use a chromosome with x 1 bit genes
• where
• 1 means the attribute has been selected
• 0 means the attribute has not been selected
• and gene position indicates the attribute
• fitness is how well the attribute subset performs
• some combination of training and testing
  performance

16
Hybrid Approaches
chromosome population
0 1 0 0 0 1 1 1 0 0 0 ltetcgt
ANN
fitness calculation
possible benefits reduced training
time increased classification accuracy