Evolving Neural Networks in Classification

1
Evolving Neural Networks in Classification

• Sunghwan Sohn

2
Outline

• Introduction
• Research Objectives
• Data mining and Computational Intelligence
• CI Tools
• Neural Networks
• Genetic Algorithms
• Evolving Neural Networks
• Experimental Results
• Conclusion

3
Research Objectives

• To develop a hybrid neural system that can be
  used in data mining
• Artificial neural networks and genetic algorithms
  are used
• Another paradigm of modeling classification tools
  

4
Data Mining

• The hottest technology to deal with explosive
  growth of data
• Data mining uses sophisticated statistical
  analysis and modeling techniques to uncover
  patterns and relationships in data

5
CI to Data Mining

• CI is the study or design of intelligent systems
• Artificial neural networks
• Fuzzy logic
• Genetic algorithms
• Each of these methodologies provides a distinct
  method to address problems
• These methodologies can be implemented
  cooperatively to be more intelligent and robust
  systems

6
Neural Networks

• An information-processing paradigm inspired by
  biological nervous systems
• A novel structure composed of a large number of
  highly interconnected processing neurons
• Organized in layers
• Input layer
• Hidden layer
• Output layer
• Layers are made up of a number of interconnected
  neurons that contain an activation function

7
Neural Network Topology
8
Neural Network Learning

• Primary Features
• Learns from its environment
• Improves its performance after iterations of the
  learning process

9
Genetic Algorithms

• First approach to evolutionary computations
• Optimization algorithm based on natural selection
  survival of the fittest
• Use binary-valued strings to represent the
  problem
• String ? Chromosome
• Bit ? Gene

10
Issues of Genetic Algorithms

• Encoding
• A genetic representation of the solution to the
  problem
• Fitness Function
• An evaluation function in terms of their fitness
• Genetic operators
• Alter genetic composition of children
• Crossover and Mutation

11
Encoding
12
Crossover
1-point crossover

• Adapted from http//www.pwr.wroc.pl/AMIGA/ARTech/i
  002/GeneticAlgorithms_1.html

13
Mutation

• Adapted from http//www.pwr.wroc.pl/AMIGA/ARTech/i
  002/GeneticAlgorithms_1.html

14
Genetic Algorithms Procedure

• Adapted from http//www.systemtechnik.tu-ilmenau.d
  e/pohlheim/Papers/mpga_gal95/gal2_1.html

15
Combination of NNs and GAs

• Neural Network Training
• GAs can be used to train the weights of NN and to
  work as a learning algorithm
• Neural Network Architecture
• An individual of the population is translated
  into a network structure
• Neural Network Parameters
• Feature Selection

16
Why Evolving Neural Networks?

• NNs performance significantly depends on their
  architecture
• This architecture is usually found by trial and
  error
• Time consuming
• May not guarantee to find the optimal network

17
Why Evolving Neural Networks?

• GAs to the automatic generation of NNs
• A proper architecture
• Biological inspiration
• Customization for a special objective

18
Combining Multiple Neural Networks

• Biological concepts in network design
• Neuron doctrine by Barlow
• The neural information processing must be based
  at the modular subnetwork level
• Not to rely on a single network's decision but to
  use multiple networks by combining their
  individual information
• Derive more robust decisions

19
Evolving Neural Networks

• The classification tool in data mining
• Combination of NN and GA
• GA to find a good feature subset
• GA to find a proper network architecture
• Combining networks

20
Ensemble of Evolving Neural Networks
21
Design of Evolving Neural Networks

• The individual in GA is translated into a network
  structure
• Feature selection
• Neuron connectivity
• Then trained by backpropagation
• The fitness measure is evaluated for the network
  performance

22
Optimizing a NN architecture Using GA
23
Encoding of Neural Networks
24
Genetic Operators

• Crossover is used to exchange the element values
• The architecture of two neural networks is
  exchanged
• Mutation changes the element value to a new one
• Change the feature selection
• Change the connection link in neural networks

25
Fitness Function
where ? and ? are weight constants CRv
is the correct classification ratio C
is the complexity defined by
connections used Cmax is the maximum
complexity defined by full
connections
26
Experimental Setup

• Single best evolving NN
• Simple problem
• Comparison with classical NN
• Special problems
• Ensemble of evolving NNs
• Utilizing bagging

27
Single evolving NN Data Sets

• Iris Data
• To show how the evolving NN works
• Data from UCI Machine Learning Repository
• To demonstrate the performance of the evolving NN

28
Iris Data
29
Network Topologies Produced by GA
30
UCI Data Sets

• Classification, 5-fold cross validation

31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
Special Problems

• To present feature selection ability of evolving
  NNs
• To demonstrate customization ability of evolving
  NNs
• Ensemble of evolving NNs with averaging

35
Data Sets

• Splice Junction Data
• 3 classes of 1000 samples
• 60 DNA sequence elements (features)
• German Credit Data
• 2 classes of 1000 samples (Good700, Bad300)
• Requires use of a cost matrix
• The problem is to reduce the cost

36
Splice Junction Data

• 33 among 60 features used

Test data error (5-fold cross validation)
37
German Credit Data

• Fitness Function

where CostRatio is the Total Cost / Possible
Maximum Cost
38
German Credit Data
Test data cost (5-fold cross validation)
Trained with samples having the same samples
for both Good and Bad
39
Ensemble of Evolving NNs

• Diversity in ensemble of classifiers is necessary
  to ensure good performance
• Bagging can improve diversity
• Training sets are selected by resampling from the
  original training set
• Classifiers trained with these sets are combined
  by voting
• Selective combining is also tried

40
Ensemble of Evolving NNs

• Each individual classifier is implemented by a GA
• Individual classifier is trained by the different
  training set that is selected by bagging
• Combined by voting for the final decision

41
(No Transcript)
42
Experimental Results

• Classification Error () (5-fold cross
  validation)

43
(No Transcript)
44
(No Transcript)
45
Comparison with Simple Combining Methods
46
Summary

• Evolving NN
• Automatic generation of NN
• Feature selection
• Adaptable topology
• Adjust to a specific problem
• Ensemble of evolving NNs
• Better generalization
• Robust decision

47
Questions?