Neural Networks Introduction and Review

1
Neural Networks- Introduction and Review -

• Wen Yu

2
History of Neural Networks

• Attempts to mimic the human brain date back to
  work in the 1930s, 1940s, 1950s by Alan Turing,
  Warren McCullough, Walter Pitts, Donald Hebb and
  James von Neumann
• 1943 McCulloch-Pitts neuron as comp. elem
• 1948 Wiener cybernatics
• 1949 Hebb learning rule
• 1957 Rosenblatt at Cornell developed Perceptron,
  a hardware neural net for character recognition
• 1959 Widrow and Hoff at Stanford developed
  Adaline for adaptive control of noise on
  telephone lines
• 1960 Widrow-Hoff least mean square algorithm

3
History of Neural Networks

• Recession
• 1969 Minsky-Papert limitations perceptron model
• Linear Separability in Perceptrons

4
History of Neural Networks

• Revival, mathematically tied together many of the
  ideas from previous research.
• 1982 Hopfield recurrent network model
• 1982 Kohonen self-organizing maps
• 1986 Rumelhart et. al. backpropagation
• universial approximation
• Since then, growth has exploded. Over 80 of
  Fortune 500 have neural net RD programs.
• Thousands of research papers
• Commercial software applications

5
Application with neural network

• Forecasting/Market Prediction finance and
  banking
• Manufacturing quality control, fault diagnosis
• Medicine analysis of electrocardiogram data,
  RNA DNA sequencing, drug development without
  animal testing
• Pattern/Image recognition handwriting
  recognition, airport bomb detection
• Optimization without Simplex
• Control process, robotics

6
The Biological Neuron
Neurons are brain cells, it is estimated that
there are 1012 neurons and 1014 synaptic
connections in the human brain
7
Biological Neurons
dendrites
axon
synapses
The information transmission happens at the
synapses.
8
Neural Dynamics
Action potential 100mV Activation threshold
20-30mV Rest potential -65mV Spike time
1-2ms Refractory time 10-20ms
Action potential
Refractory time
9
Key to intelligence

• Synapse weight adjustment.
• Connection strength
• Each neuron receives input from nearly 50,000 to
  80,000 other neurons in the human brain.
• The contribution of the signals depends on the
  strength of the synaptic connection.

10
Simple Neuron

• Nodes have input signals. Dendrites carry an
  impulse to the neuron
• Nodes have one output signal. Axons carry signal
  out of neuron and synapses are local regions
  where signals are transmitted from the axon of
  one neuron to dendrites of another
• Input signal weights are summed at each node.
  Nerve impulses are binary they are go or no
  go. Neurons sum up the incoming signal and fire
  if a threshold value is reached.

11
Artificial neurons
Neurons work by processing information. They
receive and provide information in form of spikes.
x1 x2 x3 xn-1 xn
w1
Output
w2
Inputs
y
w3
.
.
.
wn-1
wn
The McCullogh-Pitts model
12
Binary Neurons
Stimulus
Response
on
Hard threshold
off

• threshold

• ex Perceptrons, Hopfield NNs, Boltzmann Machines
• Main drawbacks can only map binary functions,
  biologically implausible.

13
Analog Neurons
Stimulus
Response
on
Soft threshold
off

• ex MLPs, Recurrent NNs, RBF NNs...
• Main drawbacks difficult to process time
  patterns, biologically implausible.

14
Spiking Neurons
Stimulus

• spike and afterspike potential
• urest resting potential
• e(t,u(t)) trace at time t of input at time t
• threshold
• xj(t) output of neuron j at time t
• wij efficacy of synapse from neuron i to neuron
  j
• u(t) input stimulus at time t

Response
Firing pattern interpretation (spike trains of
units) Timing of spike trains encodes
information (time to first spike, phase of
signal, correlation and synchronicity
15
Spiking Neuron Dynamics
16
Artificial Neural Networks
Output layer
Hidden layers
fully connected
Input layer
sparsely connected
17
Feedforward ANN Architectures

• Information flow unidirectional
• Static mapping yf(x)
• Multi-Layer Perceptron (MLP)
• Radial Basis Function (RBF)
• Kohonen Self-Organising Map (SOM)

18
Recurrent ANN Architectures

• Feedback connections
• Dynamic memory y(t1)f(x(t),y(t),s(t))
  t?(t,t-1,...)
• Jordan/Elman ANNs
• Hopfield
• Adaptive Resonance Theory (ART)

19
Activation functions
Linear
Sigmoid
Hyperbolic tangent
20
Neural network mathematics
Inputs
a
Output
y
f
b
MLP neural network
Gaussian RBF
a
f
y

b
f
21
ANN Capabilities

• Learning
• Approximate reasoning
• Generalisation capability
• Noise filtering
• Parallel processing
• Distributed knowledge base
• Fault tolerance

22
Properties of Neural Networks

• Supervised networks are universal approximators
• Theorem Any limited function can be
  approximated by a neural network with a finite
  number of hidden neurons to an arbitrary
  precision
• Type of Approximators
• Linear approximators for a given precision, the
  number of parameters grows exponentially with the
  number of variables (polynomials)
• Non-linear approximators (NN), the number of
  parameters grows linearly with the number of
  variables
• Knowledge base not transparent (black box)
  (Partially resolved)
• Learning sometimes difficult/slow
• Limited storage capability

23
Learning in biological systems as optimisation

• Learning learning by adaptation
• The young animal learns that the green fruits are
  sour, while the yellowish/reddish ones are sweet.
  The learning happens by adapting the fruit
  picking behavior. The animal likes to eat many
  energy rich, juicy fruits that make its stomach
  full, and makes it feel happy.
• At the neural level the learning happens by
  changing of the synaptic strengths, eliminating
  some synapses, and building new ones. the
  objective of learning in biological organisms is
  to optimise the amount of available resources,
  happiness, or in general to achieve a closer to
  optimal state

24
Learning principle for artificial neural networks

• Maintaining synaptic strength needs energy, it
  should be maintained at those places where it is
  needed, and it shouldnt be maintained at places
  where its not needed
• ENERGY MINIMIZATION
• We need an appropriate definition of energy for
  artificial neural networks, and having that we
  can use mathematical optimisation techniques to
  find how to change the weights of the synaptic
  connections between neurons.
• ENERGY measure of task performance error

25
Neural Networks Learning

• Supervised learning
• Classification
• Control
• Function approximation
• Associative memory
• Unsupervised learning
• Clustering
• Reinforcement learning
• Control

26
Unsupervised Learning

• ANN adapts weights to cluster input data
• Hebbian learning
• Connection stimulus-response strengthened
  (hebbian)
• Competitive learning algorithms
• Kohonen ART
• Input weights adjusted to resemble stimulus

27
Hebbian Learning
General Formulation
Kohonen, Competitive (ART)
Hebbian
llearning coefficient wijconnection from neuron
xj to yi

• Hebb postulate (1948)
• Correlation-based learning
• Connections between concurrently firing neurons
  are strengthened
• Experimentally verified (1973)

28
Supervised Learning

• Teacher presents ANN input-output pairs
• ANN weights adjusted according to error
• Iterative algorithms (e.g. Delta rule, BP rule)
• One-shot learning (Hopfield)
• Quality of training examples is critical

29
Delta Rule

• Least Mean Squares
• Widrow-Hoff iterative delta rule
• Gradient descent of the error surface
• Guaranteed to find minimum error configuration in
  single layer ANNs

30
Gradient Learning
is the learning parameter (usually a constant)
31
Learning with a perceptron
A perceptron is able to learn a linear function.
Perceptron
Data
Error
Learning
32
Learning with RBF neural networks
An RBF neural network learns a nonlinear function.
RBF neural network
Data
Error
Learning
33
Learning with general optimization
MLP neural network with a single hidden layer
34
Learning with general optimization
35
Reinforcement Learning

• Sequential tasks
• Desired action may not be known
• Critic evaluation of ANN behaviour
• Weights adjusted according to critic
• May require credit assignment
• Population-based learning
• Evolutionary Algorithms
• Swarming Techniques
• Immune Networks

36
Learning Summary
Artificial Neural Networks
Unsupervised
Supervised
Unsupervised
Supervised
Elman, Jordan, Hopfield
MLP, RBF
ART
Kohonen, Hebbian