Pattern%20Recognition:%20Statistical%20and%20Neural

1
Nanjing University of Science Technology
Pattern RecognitionStatistical and Neural
Lonnie C. Ludeman Lecture 20 Oct 26, 2005
2
Lecture 20 Topics
1. Perceptron Algorithm Revisited 2. Local Delta
Training Algorithm for ANE 3. General Definition
of Neural Networks 4. Basic Neural Network
Structures-Examples 5. Analysis and Synthesis of
Neural Networks
3
Signum Function Activation Training
Algorithm(Perceptron)
Review
y 1 if input vector x is from C1 y -1 if
input vector x is from C2
Weight Update Algorithm
4
Question
How do we train an Artificial Neural Element(ANE)
to do classification ???
Answer
Use the Delta Training Algorithm !!!
5
Given an Artificial Neural Element as follows
Wish to find weight vector such that training
patterns are correctly classified
6
Given
x(p) e x1, x2, , xK d( x(p) )
d(x1), d(x2), , d(xK) Define a performance
measure Ep for sample x(p) and decision d x(p)
as
7
Derivation of Delta weight update Equation
Use the gradient method to minimize Ep
New Weight wk1 in terms of previous weight wk
where the Gradient is
8
Substituting the gradient vector into the weight
update gives the General Local Delta Algorithm
or rewriting gives
w(p1) w(p) dx(p) f(net) f /(net))
x(p)
where net wT(p)x(p)
General Local Delta Algorithm Weight Update
Equation
9
Sometimes called the
Continuous Perceptron Training Algorithm
10
Case 1 Local Delta Algorithm for Training an ANE
with Logistic Activation Function
Given
Solution
11
Substituting the derivative gives the Local
algorithm for the Logistic Activation function as
Local Weight Update Equation for Logistic
Activation Function
12
Case 2 Local Delta Algorithm for for Training an
ANE - Hyperbolic Tangent Activation Function
Given
Solution Taking derivative of the nonlinearity
and substituting into the general update equation
yields the following
Local Weight Update Equation for Hyperbolic
Activation Function
13
Scale Factors for Case 2 Tanh Activation
Function
SF ( dx(p) f(net) )(1 f 2(net) )
dx(p) 1 SF1 ( 1 f(net) )(1 f
2(net) )
dx(p) -1 SF-1 ( -1 f(net) )(1 f
2(net) )
dx(p) 1
dx(p) -1
14
Scale Factors for Case 2 Tanh Activation
Function (desired values 0.9 and -0.9 )
15
Case 3 Local Delta Algorithm for Training an ANE
- Linear Activation Function
Given
SolutionTaking derivative and substituting in
general update equation gives
Local Weight Update Equation for Linear
Activation Function
( Widrow-Hoff Training Rule )
16
General Global Delta Algorithm
Define a performance measure ETOT for all samples
xk and decisions d xk) as
Using Gradient technique gives the Global Delta
Algorithm as
Global Weight Update Equation
17
Definitions
A Neural Network is defined as any connection of
Neural Elements.
An Artificial Neural Network is defined as any
connection of Artificial Neural Elements.
18
Examples of Artificial Neural Networks
Feed Forward Artificial Neural Networks
(a) Two Layer neural Network (b) Special Three
Layer Form Hyperplane-AND-OR
structure (c) General 3-Layer Feedforward
structure and nomenclature
Feedback Artificial Neural Networks (d)
One Layer Hopfield Net (e) Two Layer
Feedback
19
(a) Example - Two Layer Neural Network Using
Signum Nonlinearity
20
(b) Special Hyperplane-AND-OR structure
input
output
Layer 1
Layer 2
Layer 3
y
x
Hyperplanes
Logical AND
Logical OR
21
Building Block- Hyperplane
µ
22
Building Block- AND
µ
-(n-½)
23
Building Block- OR
µ
½
24
(b) Example- Hyperplanes-AND-OR Structure
Hyperplanes Layer
all f() u() unit step
AND Layer
OR Layer
25
(c) General Feedforward Structure
26
(d) Example Feedback Structure one Layer
27
(e) Example Feedback Structure Two Layer
/
28
Definitions
Analysis of Neural Networks-
Given a Neural Network describe the output for
all inputs ( Mathematical or computer generated)
Synthesis of Neural Networks-
Given a list of properties and requirements build
a Neural Network to satisfy the requirements (
Mathematical or computer generated)
29
Example Analyze the following Neural Network
-1
0
1
1
1
0
0
-1
1
Solution
Determine the output y1(2) for all (x1,x2).
(Next Lecture)
30
Example Synthesize a Neural Network
Given the following decision regions build a
neural network to perform the classification
process
Solution Use Hyperplane-AND-OR Structure
(Next Lecture)
31
Summary Lecture 20
1. Perceptron Algorithm Revisited 2. Local Delta
Training Algorithms for ANE 3. General Definition
of Neural Networks 4. Basic Neural Network
Structures-Examples 5. Analysis and Synthesis of
Neural Networks
32
Question How do we train an Artificial Neural
Network to perform the classification problem???
Answer Not a simple answer but we will look at
one way that uses the backpropagation algorithm
to do the Training.
Not Today, we have to wait until Friday.
?????????
33
End of Lecture 20