Neural Networks: An Overview

1
Neural NetworksAn Overview
2
There are many different kinds of Neural Network
they include to name a few
The Back Propagating Networks
Neural NetworksAn Overview
Kohonen (Self-Organising) Networks
Hopfield Networks
The Boltzmann Machine
All Neural Networks use the concept of Neurons
which are gradually trained to give a more
appropriate response to input.
This project has used a Kohonen Network. I shall
explain the properties of such a network and why
it is best for a small scale project such as this.
3
The inputs are added together and
this result is used to generate an output for the
neuron.
What is a Kohonen Network?
So what is a Neuron?

The inputs are generally weighted so
that some inputs to the neuron may have a greater
affect than others.
Neurons have inputs and outputs.
x1
Kohonen (Self-Organising) Networks
w1
w2
U
?
x2
w3
x3
A neuron of such as this one is called an
Adaptive Linear Element (or Adaline) and may be
found own its own
or may be found as part of a larger neural
network where the output of a neuron may be the
input for the next layer.
This project has used a Kohonen Network. I shall
explain the properties of such a network and why
it is best for a small scale project such as this.
4
Kohonen (Self-Organising) Networks
What is a Kohonen Network?
A Kohonen network uses these basic principles
however in a Kohonen network there is only one
layer of neurons which are arranged in a 2D
lattice.
There is an input layer with inputs which feed
into every neuron.
x1
x2
5
Kohonen (Self-Organising) Networks
How does training work?
What is a Kohonen Network?
A Kohonen network uses these basic principles
however in a Kohonen network there is only one
layer of neurons which are arranged in a 2D
lattice.
The key to the neural network are the weights as
it is they that dictate which neuron will win
when the network is presented with a particular
input pattern.
There is an input layer with inputs which feed
into every neuron.
Individual neuron output ?(xi wik) 2 (for all
i) Where wik is the weight between input i and
neuron k.
Each neuron evaluates the sum of the weighted
inputs and uses this as its own output. This is
the Euclidian distance of the neuron from the
input pattern.
The neuron with the smallest Euclidian distance
is the one which best represents the input
pattern and is known as the winner.
6
Kohonen (Self-Organising) Networks
How does training work?
All the weights in the network are initialised to
small random values.
The use of a Neighborhood during training.
An input pattern is presented to the network and
the winning neuron is found.
The weights to the winning node
are updated using the following equation.
wik(t1) wik(t) G(t) (xi(t) wik(t))

This
promotes large changes away from the initial
state of the network but smaller changes later
when the network is some way through training.

This is
referred to as course mapping and fine mapping.
The gain term is a value between 0 and 1 which
starts high and as more patterns are presented to
the network is decreased.
7
Kohonen (Self-Organising) Networks
How does training work?
The use of a Neighborhood during training.
Usually the weights of more neurons than the
winner are changed.
Once the winner has been identified a
neighbourhood of a predefined shape and size is
drawn around it.
8
Kohonen (Self-Organising) Networks
How does training work?
The use of a Neighborhood during training.
A real output lattice will typically have
hundreds of neurons.
Usually the weights of more neurons than the
winner are changed.

All the neurons
in the neighbourhood have their weights updated
to better represent the input pattern. This
promotes clustering of similar results.
Once the winner has been identified a
neighbourhood of a predefined shape and size is
drawn around it.
The neighbourhood size starts very large
(typically the entire network) and is decreased
during training in a similar way to the gain term.
9
Kohonen (Self-Organising) Networks
How does training work?
A real output lattice will typically have
hundreds of neurons.
Then all that is left is to present some input
patterns to the network with known real world
values (such as PCO2) so that the defined areas
of the lattice can be found for the individual
problem and to give the areas real world values.


This self organisation gives the network its
name and leads to the lattice of nodes often
being referred to as a self organising matrix.
Over time the lattice of neurons will organise
its self into areas of neurons which respond more
highly to certain input patterns.
The PCO2 problem on a
302.1
Kohonen network
320.8
295.6
254.3
Now when an input pattern is presented to the
network the winning neuron should be in or close
to a defined area so a value can be assigned.
10
The PCO2 problem on a Kohonen network
The aim of this project is to program, implement
and test a Kohonen network designed to predict
PCO2 levels at the surface of the ocean.
The inputs
Lattice size
The network has been mainly programmed in Fortran
and is made up of many console programs each with
a particular function to perform on the network.
Where ever possible the
programs are arbitrary and so such properties as
the number of input in an input pattern can be
set for each network.
The latter networks have had 5
inputs in each pattern, these are
2 for month
, 1 latitude
, 1 longitude
and 1 sea surface temperature.
These where used as they where thought to have
the greatest affect on the PCO2 concentration.
To encourage
the network to distinguish between months far
apart and close in the calendar and to give a
greater emphasis on the month input it was
included twice in each pattern. However one was
an input based on the Sine of the month and one
on the Cosine making the month cyclical.
11
The PCO2 problem on a Kohonen network
The aim of this project is to program, implement
and test a Kohonen network designed to predict
PCO2 levels at the surface of the ocean.
PCO2 data in the North Atlantic drift area from a
neural network
Lattice size
Typically a square lattice size of 70x70 neurons
was used. Of the 4900 neurons in such a lattice
only 18.06 of them will be given exact PCO2
values to report when they are found to be the
winner.

If
an unclassified neuron wins then a function
must be employed to find the most appropriate
PCO2 value.
As the network must find the output of every
neuron in the lattice to find the winner the size
of the lattice greatly affects the time the
network takes to train or run.
To illustrate the properties of lattice sizes
several sizes have
been tested.
Lattice size Classed Apr May Jun Jul Aug Sep Oct Nov Average
175x175 (30625) 6.13 30.58 4.81 4.75 2.78 2.12 2.73 2.16 2.12 6.5080
70x70 (4900) 18.06 35.68 4.74 6.93 3.33 2.52 3.13 2.24 2.22 7.6000
28x28 (784) 46.68 31.82 6.16 6.57 3.75 2.62 4.27 2.36 2.31 7.4833
11x11 (121) 86.78 11.54 8.86 10.70 4.62 2.91 5.00 4.07 2.82 6.3161
4x4 (16) 100.00 30.49 13.22 14.19 5.58 11.43 10.98 16.52 3.59 13.2505
Using data as model - 11.54 15.84 15.69 10.57 11.53 9.41 11.77 4.31 11.3331
All data is restricted to the north Atlantic
drift area.
12
Created by a 70x70 lattice
The PCO2 problem on a Kohonen network
PCO2 data in the North Atlantic drift area from a
neural network
Using Levitus data (which gives a uniform grid of
latitude and longitude with temperature values
for any month) a map of PCO2 values can be
constructed.
This was done with MatLab.
Each point on the maps has an inside colour set
by the PCO2 value given by the network and an
outside colour set by the data the network was
trained on, this gives a visual grade of how well
the network predicts the PCO2 levels.
The network which created these maps had a
lattice size of 70x70
13
Created by a 175x175 lattice
14
Created by a 28x28 lattice
15
Created by a 11x11 lattice
16
Created by a 4x4 lattice
The PCO2 problem on a Kohonen network
Training method
17
The PCO2 problem on a Kohonen network
Training method
As mentioned before the gain and the
neighbourhood size change during the training
process.
Long Train
3000000 80.00 80.00 100.00
75.00 3000000 65.00 65.00 75.00
60.00 3000000 52.00 52.00 60.00
50.00 3000000 40.00 40.00 50.00
33.00 6000000 30.00 20.00 33.00
20.00 6000000 20.00 10.00
20.00 16.00

Usually the gain is stepped down while the
neighbourhood size gets smaller smoothly.

The training program
looks for a file which tells it how to change
these properties of the network.


The file used in training
the last networks was as follows.
3000000 65.00 65.00 100.00
30.00 3000000 40.00 40.00 30.00
20.00 6000000 20.00 10.00 20.00
16.00
So two more tests where performed this time
changing the training method.
18
The PCO2 problem on a Kohonen network
Training method
Long Train
3000000 80.00 80.00 100.00
75.00 3000000 65.00 65.00 75.00
60.00 3000000 52.00 52.00 60.00
50.00 3000000 40.00 40.00 50.00
33.00 6000000 30.00 20.00 33.00
20.00 6000000 20.00 10.00
20.00 16.00
Quick Train
1000000 65.00 65.00 100.00
35.00 500000 40.00 40.00 35.00
25.00 1500000 20.00 20.00 25.00
18.00
These are designed to do the same as original
training method but with more or less runs i.e.
the patterns are presented to the network more or
less times.
The networks both had a 70x70
output lattice so a comparison with the 70x70
size network trained with the original method is
valid.
Classed Apr May Jun Jul Aug Sep Oct Nov Average
Long 18.00 34.83 4.80 6.08 3.45 2.37 3.37 2.07 2.26 7.4038
Original 18.06 35.68 4.74 6.93 3.33 2.52 3.13 2.24 2.22 7.6000
Quick 19.84 18.38 4.30 4.99 3.13 2.38 3.24 2.14 2.22 5.0963
19
Created by a 70x70 lattice using the Long method
20
Created by a 70x70 lattice using the Orig-inal
method
21
Created by a 70x70 lattice using the Quick method