Artificial Neural Networks: An Introduction

1
Artificial Neural Networks An Introduction

• S. Bapi Raju
• Dept. of Computer and
• Information Sciences,
• University of Hyderabad

2
OUTLINE

• Biological Neural Networks
• Applications of Artificial Neural Networks
• Taxonomy of Artificial Neural Networks
• Supervised and Unsupervised Artificial Neural
  Networks
• Basis function and Activation function
• Learning Rules
• Applications
• OCR, Load Forecasting, Condition Monitoring

3
Biological Neural Networks

• Study of Neural Networks originates in biological
  systems
• Human Brain contains over 100 billion neurons,
  number of synapses is approximately 1000 times
  that
• in electronic circuit terms synaptic fan-in
  fan-out is 1000,
• switching time of a neuron is order of
  milliseconds
• But on a face recognition problem brain beats
  fastest supercomputer in terms of number of
  cycles of computation to arrive at answer
• Neuronal Structure

• Cell body
• Dendrites for input
• Axon carries output to other dendrites
• Synapse-where they meet
• Activation signal (voltage) travels along axon

4
Need for ANN

• Standard Von Neumman Computing as existing
  presently has some shortcomings.
• Following are some desirable characteristics in
  ANN
• Learning Ability
• Generalization and Adaptation
• Distributed and Parallel representation
• Fault Tolerance
• Low Power requirements
• Performance comes not just from the computational
  elements themselves but the manner of networked
  interconnectedness of the decision process.

5
VonNeumann versus BiologicalComputer
6
ANN Applications

• Pattern Classification
• Speech Recognition, ECG/EEG classification, OCR

7
ANN Applications

• Clustering/Categorization
• Data mining, data compression

8
ANN Applications

• Function Approximation
• Noisy arbitrary function needs to be approximated

9
ANN Applications

• Prediction/Forecasting
• Given a function of time, predict the function
  values for future time values, used in weather
  prediction and stock market predictions

10
ANN Applications

• Optimization
• Several scientific and other problems can be
  reduced to an optimization problem like the
  Traveling Salesman Problem (TSP)

11
ANN Applications

• Content Based Retrieval
• Given the partial description of an object
  retrieve the objects that match this

12
ANN Applications

• Control
• Model-reference adaptive control, set-point
  control
• Engine idle-speed control

13
Characteristics of ANN

• Biologically inspired computational units
• Also called as Connectionist Models or
  Connectionist Architectures
• Large number of simple processing elements
• Very large number of weighted connections between
  elements. Information in the network is encoded
  in the weights learned by the connections
• Parallel and distributed control
• Connection weights are learned by automatic
  training techniques

14
Artifical Neuron Working Model

• Objective is to create a model of functioning of
  biological neuron to aid computation

• All signals at synapses are summed i.e. all the
  excitatory and inhibitory influences and
  represented by a net value h(.)
• If the excitatory influences are dominant, then
  the neuron fires, this is modeled by a simple
  threshold function ?(.)
• Certain inputs are fixed biases
• Output y leads to other neurons

McCulloch Pitts Model
15
More about the Model

• Activation Functions play a key role
• Simple thresholding (hard limiting)
• Squashing Function (sigmoid)
• Gaussian Function
• Linear Function
• Biases are also learnt

16
Different Kinds of Network Architectures
17
Learning Ability

• Mere Architecture is insufficient
• Learning Techniques also need to be formulated
• Learning is a process where connection weights
  are adjusted
• Learning is done by training from labeled
  examples. This is the most powerful and useful
  aspect of neural networks in their use as Black
  Box classifiers.
• Most commonly an input-output relationship is
  learnt
• Learning Paradigm needs to be specified
• Weight update in learning rules must be specified
• Learning Algorithm specifies step by step
  procedure

18
Learning Theory

• Major Factors
• Learning Capacity This concerns the number of
  patterns that can be learnt and the functions and
  kinds of decision boundaries that can be formed
• Sample Complexity This concerns the number of
  the samples needed to learn with generalization.
  Overfitting problem is to be avoided
• Computational Complexity This concerns the
  computation time needed to learn the concepts
  embedded in the training samples. Generally the
  computational complexity of learning is high.

19
Learning Issues
20
Major Learning Rules

• Error Correction Error signal (dy) used to
  adjust the weights so that eventually desired
  output d is produced

Perceptron Solving AND Problem
21
Major Learning Rules

• Error Correction in Mutlilayer Feedforward
  Network

Geometric interpretation of the role of hidden
units in a 2D input space
22
Major Learning Rules

• Hebbianweights are adjusted by a factor
  proportional to the activities of the neurons
  associated

Orientation Selectivity of a Single Hebbian Neuron
23
Major Learning Rules

• Competitive Learning winner take all

(a) Before Learning (b) After
Learning
24
Summary of ANN Algorithms
25
(No Transcript)
26
Application to OCR System

• The main problem in the Handwritten Letter
  recognition is that characters with variation in
  thickness shape, rotation and different nature of
  strokes need to be recognized as of being in the
  different categories for each letter.
• Sufficient number of sample training data is
  required for each character to train the networks
  

A Sample set of characters in the NIST Data
27
OCR Process
28
OCR Example (continued)

• Two schemes shown at right
• First makes use of the feature extractors
• Second uses the image pixels directly

29
References

• A. K. Jain, J.Mao, K.Mohiuddin, ANN a Tutorial,
  IEEE Computer, 1996 March, pp 31-44 (Figures and
  Tables taken from this reference)
• B. Yegnanarayana, Artificial Neural Networks,
  Prentice Hall of India, 2001.
• Y. M. Zurada, Inroduction to Artificial Neural
  Systems, Jaico, 1999.
• MATLAB neural networks toolbox and manual