NEURONAL DYNAMICS 2: ACTIVATION MODELS

1
NEURONAL DYNAMICS 2
ACTIVATION MODELS
2
Chapter 3. Neuronal Dynamics 2 Activation Models
3.1 Neuronal dynamical system
Neuronal activations change with time. The way
they change depends on the dynamical equations as
following
(3-1) (3-2)
3
3.1 ADDITIVE NEURONAL DYNAMICS

• first-order passive decay model

In the absence of external or neuronal stimuli,
the simplest activation dynamics model is
(3-3) (3-4)
4
3.1 ADDITIVE NEURONAL DYNAMICS

• since for any finite initial condition

• The membrane potential decays exponentially
  quickly to its zero potential.

5
Passive Membrane Decay

• Passive-decay rate

scales the rate to
the membranes resting potential.

• solution

Passive-decay rate measures the cell
membranes resistance or friction to current
flow.
6
property

• Pay attention to property

The larger the passive-decay rate,the faster the
decay--the less the resistance to current flow.
7
Membrane Time Constants

• The membrane time constant scales the time
  variable of the activation dynamical system.

• The multiplicative constant model

(3-8)
8
Solution and property

• solution

• property

The smaller the capacitance ,the faster things
change
As the membrane capacitance increases toward
positive infinity,membrane fluctuation slows to
stop.
9
Membrane Resting Potentials

• Definition

Define resting Potential as the activation
value to which the membrane potential
equilibrates in the absence of external or
neuronal inputs
(3-11)

• Solutions

(3-12)
10
Note

• The capacitance appear in the index of the
• solution, it is called time-scaling capacitance.

• It does not affect the asymptotic or
  steady-state solution and does not depend on
  the finite initial condition.

11
Additive External Input

• Add input

Apply a relatively constant numeral input to a
neuron.
(3-13)

• solution

(3-14)
12
Meaning of the input

• Input can represent the magnitude of directly
• experiment sensory information or directly apply
  control information.

• The input changes slowly,and can be assumed
• constant value.

13
3.2 ADDITIVE NEURONAL FEEDBACK

• Neurons do not compute alone. Neuron modify
  their state activations with external input and
  with the feedback from one another.

• This feedback takes the form of path-weighted
  signals from synaptically connected neurons.

14
Synaptic Connection Matrices

• n neurons in field p neurons in field

The ith neuron axon in a synapse
jth neurons in
is constant,can be positive,negative or zero.
15
Meaning of connection matrix

• The synaptic matrix or connection matrix M is
  an
• n-by-p matrix of real number whose entries are
  the
• synaptic efficacies. the ijth synapse is
  excitatory
• if inhibitory if

• The matrix M describes the forward projections
  from
• neuron field to neuron field

• The matrix N describes the backward projections
  
• from neuron field to neuron field

16
Bidirectional and Unidirectional connection
Topologies

• Bidirectional networks

M and N have the same or approximately the
same structure.

• Unidirectional network

A neuron field synaptically intraconnects to
itself.M nxn.

• BAM

M is symmetric, the
unidirectional network is BAM
17
Augmented field and augmented matrix

• Augmented field

M connects to ,N connects to
then the augmented field intraconnects to
itself by the square block matrix B
18
Augmented field and augmented matrix

• In the BAM case,when then
  hence a BAM symmetries an arbitrary
  rectangular matrix M.

• In the general case,

P is n-by-n matrix. Q is p-by-p matrix.
If and only if,
the neurons in are symmetrically
intraconnected
19
3.3 ADDITIVE ACTIVATION MODELS

• Define additive activation model
• np coupled first-order differential equations
  defines the additive activation model

(3-15) (3-16)
20
additive activation model define

• The additive autoassociative model correspond to
  a system of n coupled first-order differential
  equations

(3-17)
21
additive activation model define

• A special case of the additive autoassociative
  model

(3-18) (3-19)
(3-20)
where
is
measures the cytoplasmic resistance
between neurons i and j.
22
Hopfield circuit and continuous additive
bidirectionalassociative memories

• Hopfield circuit arises from if each neuron has
  a strictly increasing signal function and if the
  synaptic connection matrix is symmetric

(3-21)

• continuous additive bidirectional associative
  memories

(3-22)
(3-23)
23
3.4 ADDITIVE BIVALENT FEEDBACK

• Discrete additive activation models correspond
  to neurons with threshold signal function

• The neurons can assume only two value ON and
  OFF.
• ON represents the signal value 1. OFF
  represents 0 or 1.

• Bivalent models can represent asynchronous and
  stochastic behavior.

24
Bivalent Additive BAM

• BAM-bidirectional associative memory

• Define a discrete additive BAM with threshold
  signal functions, arbitrary thresholds and
  inputs,an arbitrary but constant synaptic
  connection matrix M,and discrete time steps k.

(3-24)
(3-25)
25
Bivalent Additive BAM

• Threshold binary signal functions

(3-26)
(3-27)

• For arbitrary real-value thresholds
• for neurons
  for neurons

26
A example for BAM model

• Example
• A 4-by-3 matrix M represents the forward synaptic
  projections from to .
• A 3-by-4 matrix MT represents the backward
  synaptic projections from to .

27
A example for BAM model

• Suppose at initial time k all the neurons in
  are ON.
• So the signal state vector at time k
  corresponds to

• Input

• Suppose

28
A example for BAM model

• firstat time k1 through synchronous
  operation,the result is

• nextat time k1 ,these signals pass
  forward through the
• filter M to affect the activations of the
  neurons.

• The three neurons compute three dot products,or
  correlations.
• The signal state vector multiplies
  each of the three columns of M.

29
A example for BAM model

• the result is

• synchronously compute the new signal state
  vector

30
A example for BAM model

• the signal vector passes backward through the
  synaptic
• filter at time k2

• synchronously compute the new signal state
  vector

31
A example for BAM model
since
then

• conclusion

These same two signal state vectors will pass
back and forth in bidirectional equilibrium
forever-or until new inputs perturb the system
out of equilibrium.
32
A example for BAM model

• asynchronous state changes may lead to
  different bidirectional equilibrium

• keep the first neurons ON,only update
  the second and third neurons. At k,all
  neurons are ON.

• new signal state vector at time k1 equals

33
A example for BAM model

• new activation state vector equals

• synchronously thresholds

• passing this vector forward to gives

34
A example for BAM model

• similarly,
• for any asynchronous state change policy we apply
  to the neurons

• the system has reached a new equilibrium,the
  binary pair
• represents a fixed
  point of the system.

35
conclusion

• conclusion
• Different subset asynchronous state change
  policies applied to the same data need not
  product the same fixed-point equilibrium. They
  tend to produce the same equilibria.
• All BAM state changes lead to fixed-point
  stability.

36
Bidirectional Stability

• definition
• A BAM system is
  Bidirectional stable if all inputs converge to
  fixed-point equilibria.
• A denotes a binary n-vector in
• B denotes a binary p-vector in

37
Bidirectional Stability

• Represent a BAM system equilibrates to
  bidirectional fixed
• point as

38
Lyapunov Functions

• Lyapunov Functions L maps system state variables
  to real
• numbers and decreases with time. In BAM case,L
  maps the
• Bivalent product space to real numbers.

• Suppose L is sufficiently differentiable to apply
  the chain
• rule

(3-28)
39
Lyapunov Functions

• The quadratic choice of L

(3-29)

• Suppose the dynamical system describes the
  passive decay system.

(3-30)

• The solution

(3-31)
40
Lyapunov Functions

• The partial derivative of the quadratic L

(3-32)
(3-33)
(3-34)
or
(3-35)
In either case
(3-36)
At equilibrium
This occurs if and only if all velocities equal
zero
41
conclusion

• A dynamical system is stable if some Lyapunov
  Functions L
• decreases along trajectories.
• A dynamical system is asymptotically stable if
  it strictly
• decreases along trajectories
• Monotonicity of a Lyapunov Function provides a
  sufficient
• not necessary condition for stability and
  asymptotic stability.

42
Linear system stability
For symmetric matrix A and square matrix B,the
quadratic form behaves as a
strictly decreasing Lyapunov
function for any linear dynamical system
if and only if the matrix
is negative definite.
43
The relations between convergence rate and
eigenvalue sign

• A general theorem in dynamical system theory
  relates convergence rate and eigenvalue sign
• A nonlinear dynamical system converges
  exponetially quickly if its system Jacobian has
  eigenvalues with negative real parts. Locally
  such nonlinear system behave as linearly.
• (Jacobian matrix)
• A Lyapunov Function summarizes total system
  behavior.
• A Lyapunov Function often measures the energy of
  a physical sysem. Represents system energy
  decrease
• with dynamical systems

44
Potential energy function represented by
quadratic form
Consider a system of n variables and its
potential-energy function E. Suppose the
coordinate measures the displacement from
equilibrium of ith unit.The energy depends on
only coordinate ,so since E is a
physical quantity,we assume it is sufficiently
smooth to permit a multivariable Taylor-series
expansion about the origin
45
Potential energy function represented by
quadratic form
Where A is symmetric,since
46
The reason of (3-42)follows

• First,we defined the origin as an equilibrium of
  zero potential
• energyso
• Second,the origin is an equilibrium only if all
  first partial
• derivatives equal zero.
• Third,we can neglect higher-order terms for small
  
• displacement,since we assume the higher-order
  products are
• smaller than the quadratic products.

47

• Conclusion
• Bounded decreasing L funcs provide an
  intuitive way to describe global computations
  in nueral networks ad other dynamical system.

48
Bivalent BAM theorem

• The average signal energy L of the forward pass
  of the
• Signal state vector through M,and the
  backward pass
• Of the signal state vector through
  

since
49
Lower bound of Lyapunov function

• The signal is Lyapunov function clearly bounded
  below.
• For binary or bipolar,the matrix coefficients
  define the
• attainable bound

• The attainable upper bound is the negative of
  this expression.

50
Lyapunov function for the general BAM system

• The signal-energy Lyapunov function for the
  general BAM
• system takes the form

Inputs and
and constant vectors of
thresholds the attainable bound of this function
is.
51
Bivalent BAM theorem

• Bivalent BAM theorem.every matrix is
  bidrectionally stable
• for synchronous or asynchronous state changes.

• Proof consider the signal state changes that
  occur from time k to time k1,define the vectors
  of signal state changes as

52
Bivalent BAM theorem

• define the individual state changes as

• We assume at least one neuron changes state from
  k to time k1.
• Any subset of neurons in a field can change
  state,but in only one field at a time.
• For binary threshold signal functions if a state
  change is nonzero,

53
Bivalent BAM theorem
For bipolar threshold signal functions
The energychange
Differs from zero because of changes in field
or in field
54
Bivalent BAM theorem
55
Bivalent BAM theorem
Suppose
Then
This implies so the product is
positive

Another case suppose
56
Bivalent BAM theorem

• This implies so the product is
  positive
  

So for every state change.

• Since L is bounded,L behaves as a Lyapunov
  function for
• the additive BAM dynamical system defined by
  before.
• Since the matrix M was arbitrary,every matrix is
  bidirectionally stable. The bivalent Bam theorem
  is proved.

57
Property of globally stable dynamical system
58
Two insights about the rate of convergence

• First,the individual energies decrease
  nontrivially.the BAM system does not creep
  arbitrary slowly down the toward the nearest
  local minimum.the system takes definite hops into
  the basin of attraction of the fixed point.

• Second,a synchronous BAM tends to converge
  faster than an asynchronous BAM.In another word,
  asynchronous updating should take more iterations
  to converge.

59
Review
1.Neuronal Dynamical Systems
We describe the neuronal dynamical systems by
first-order differential or difference equations
that govern the time evolution of the neuronal
activations or membrane potentials.
60
Review
4.Additive activation models
Hopfield circuit

1. Additive autoassociative model
2. Strictly increasing bounded signal function
   
3. Synaptic connection matrix is symmetric
   .

61
Review
5.Additive bivalent models
Lyapunov Functions Cannot find a lyapunov
function,nothing follows Can find a lyapunov
function,stability holds.
62
Review
A dynamics system is stable , if
asymptotically stable, if
.
Monotonicity of a lyapunov function is a
sufficient not necessary condition for stability
and asymptotic stability.
63
Review
Bivalent BAM theorem. Every matrix is
bidirectionally stable for synchronous or
asynchronous state changes.

• Synchronousupdate an entire field of neurons at
  a time.

• Simple asynchronousonly one neuron makes a
  state-change decision.

• Subset asynchronousone subset of neurons per
  field makes state-change decisions at a time.

64
Chapter 3. Neural Dynamics IIActivation Models
The most popular method for constructing Mthe
bipolar Hebbian or outer-product learning
method binary vector associations bipolar vector
associations
65
Chapter 3. Neural Dynamics IIActivation Models
The binary outer-product law
The bipolar outer-product law
The Boolean outer-product law
66
Chapter 3. Neural Dynamics IIActivation Models
The weighted outer-product law
Where holds.
In matrix notation
Where
67
Chapter 3. Neural Dynamics IIActivation Models
?3.6.1 Optimal Linear Associative Memory Matrices
Optimal linear associative memory matrices
The pseudo-inverse matrix of
68
Chapter 3. Neural Dynamics IIActivation Models
?3.6.1 Optimal Linear Associative Memory Matrices
Optimal linear associative memory matrices
The pseudo-inverse matrix of
If x is a nonzero scalar
If x is a nonzero vector
If x is a zero scalar or zero vector
For a rectangular matrix , if exists
69
Chapter 3. Neural Dynamics IIActivation Models
?3.6.1 Optimal Linear Associative Memory Matrices
Define the matrix Euclidean norm as
Minimize the mean-squared error of forward
recall,to find that satifies the relation
70
Chapter 3. Neural Dynamics IIActivation Models
?3.6.1 Optimal Linear Associative Memory Matrices
Suppose further that the inverse matrix
exists. Then
So the OLAM matrix correspond to
71
Chapter 3. Neural Dynamics IIActivation Models
If the set of vector is
orthonormal
Then the OLAM matrix reduces to the classical
linear associative memory(LAM)
For is orthonormal, the inverse of is
.
72
Chapter 3. Neural Dynamics IIActivation Models
?3.6.2 Autoassociative OLAM Filtering
Autoassociative OLAM systems behave as linear
filters.
In the autoassociative case the OLAM matrix
encodes only the known signal vectors . Then
the OLAM matrix equation (3-78) reduces to
M linearly filters input measurement x to the
output vector by vector matrix
multiplication .
73
Chapter 3. Neural Dynamics IIActivation Models
?3.6.2 Autoassociative OLAM Filtering
The OLAM matrix behaves as a projection
operatorSorenson,1980.Algebraically,this means
the matrix M is idempotent .
Since matrix multiplication is
associative,pseudo-inverse property (3-80)
implies idempotency of the autoassociative OLAM
matrix M
74
Chapter 3. Neural Dynamics IIActivation Models
?3.6.2 Autoassociative OLAM Filtering
Then (3-80) also implies that the additive dual
matrix behaves as a projection
operator
We can represent a projection matrix M as the
mapping
75
Chapter 3. Neural Dynamics IIActivation Models
?3.6.2 Autoassociative OLAM Filtering
The Pythagorean theorem underlies projection
operators.
The known signal vectors span
some unique linear subspace
of
L equals ,
the set of all linear combinations of the m known
signal vectors.
denotes the orthogonal complement space
,the set of all real n-vectors x orthogonal to
every n-vector y in L.
76
Chapter 3. Neural Dynamics IIActivation Models
?3.6.2 Autoassociative OLAM Filtering

1. Operator projects onto L.
2. The dual operator projects
   onto .

Projection Operator and
uniquely decompose every vector x into a
summed signal vector and a noise or novelty
vector
x
77
Chapter 3. Neural Dynamics IIActivation Models
?3.6.2 Autoassociative OLAM Filtering
The unique additive decomposition
obeys a generalized Pythagorean theorem
where defines the
squared Euclidean or norm.
Kohonen1988 calls the novelty
filter on .
78
Chapter 3. Neural Dynamics IIActivation Models
?3.6.2 Autoassociative OLAM Filtering
Projection measures what we know about input
x relative to stored signal vectors

for some constant vector .
The novelty vector measures what is
maximally unknown or novel in the measured
input signal x.
79
Chapter 3. Neural Dynamics IIActivation Models
?3.6.2 Autoassociative OLAM Filtering
Suppose we model a random measurement vector x as
a random signal vector corrupted by an
additive, independent random-noise vector
We can estimate the unknown signal as the
OLAM-filtered output .
80
Chapter 3. Neural Dynamics IIActivation Models
?3.6.2 Autoassociative OLAM Filtering
Kohonen1988 has shown that if the multivariable
noise distribution is radially symmetric, such as
a multivariable Gaussian distribution,then the
OLAM capacity m and pattern dimension n scale
the variance of the random-variable
estimator-error norm
81
Chapter 3. Neural Dynamics IIActivation Models
?3.6.2 Autoassociative OLAM Filtering
1.The autoassociative OLAM filter suppress noise
if , when memory capacity does not
exceed signal dimension. 2.The OLAM filter
amplifies noise if , when capacity
exceeds dimension.
82
Chapter 3. Neural Dynamics IIActivation Models
?3.6.3 BAM Correlation Encoding Example
The above data-dependent encoding schemes add
outer-product correlation matrices.
The following example illustrates a complete
nonlinear feedback neural network in action,with
data deliberately encoded into the system
dynamics.
83
Chapter 3. Neural Dynamics IIActivation Models
?3.6.3 BAM Correlation Encoding Example
Suppose the data consists of two unweighted
binary associations
and defined by the nonorthogonal
binary signal vectors
84
Chapter 3. Neural Dynamics IIActivation Models
?3.6.3 BAM Correlation Encoding Example
These binary associations correspond to the two
bipolar associations and
defined by the bipol ar signal vectors
85
Chapter 3. Neural Dynamics IIActivation Models
?3.6.3 BAM Correlation Encoding Example
We compute the BAM memory matrix M by adding the
bipol ar correlation matrices and
pointwise. The first correlation matrix
equals
86
Chapter 3. Neural Dynamics IIActivation Models
?3.6.3 BAM Correlation Encoding Example
Observe that the i th row of the correlation
matrix equals the bipolar vector
multipled by the i th element of . The j th
column has the similar result. So equals
87
Chapter 3. Neural Dynamics IIActivation Models
?3.6.3 BAM Correlation Encoding Example
Adding these matrices pairwise gives M
88
Chapter 3. Neural Dynamics IIActivation Models
?3.6.3 BAM Correlation Encoding Example
Suppose, first,we use binary state vectors.All
update policies are synchronous.Suppose we
present binary vector as input to the
systemas the current signal state vector at
. Then applying the threshold law (3-26)
synchronously gives
89
Chapter 3. Neural Dynamics IIActivation Models
?3.6.3 BAM Correlation Encoding Example
Passing through the backward filter ,
and applying the bipolar version of the
threshold law(3-27),gives back
So is a fixed point of the BAM
dynamical system. It has Lyapunov energy
, which
equals the backward value
.
has the similar resulta fixed
point with energy .
90
Chapter 3. Neural Dynamics IIActivation Models
?3.6.3 BAM Correlation Encoding Example
So the two deliberately encoded fixed points
reside in equally deep attractors.
Hamming distance H equals distance.
counts the number of slots in which binary
vectors and differ
91
Chapter 3. Neural Dynamics IIActivation Models
?3.6.3 BAM Correlation Encoding Example
Consider for example the input
, which differs from by
1 bit , or . Then
Fig3.2 shows that BAM can return original balance
regardless of the noise. bipolar
92
Chapter 3. Neural Dynamics IIActivation Models
?3.6.4 Memory CapacityDimensionality Limits
Capacity
Synaptic connection matrices encode limited
information.
We sum more correlation matrices ,then
holds more frequently.
After a point,adding additional associations
Does not significantly change the connection
matrix. The system forgetssome patterns. This
limits the memory capacity.
93
Chapter 3. Neural Dynamics IIActivation Models
?3.6.4 Memory CapacityDimensionality Limits
Capacity
Grossbergs sparse coding theorem 1976 says ,
for deterministic encoding ,that pattern
dimensionality must exceed pattern number to
prevent learning some patterns at the expense of
forgetting others.
94
Chapter 3. Neural Dynamics IIActivation Models
?3.6.5 The Hopfield Model
The Hopfield model illustrates an autoassociative
additive bivalent BAM operated serially with
simple asynchronous state changes.
Autoassociativity means the network topology
reduces to only one field, ,of neurons
.The synaptic connection matrix M
symmetrically intraconnects the n neurons in field
95
Chapter 3. Neural Dynamics IIActivation Models
?3.6.5 The Hopfield Model
The autoassociative version of Equation (3-24)
describes the additive neuronal activation
dynamics
(3-87)
for constant input , with threshold signal
function
(3-88)
96
Chapter 3. Neural Dynamics IIActivation Models
?3.6.5 The Hopfield Model
We precompute the Hebbian synaptic connection
matrix M by summing bipolar outer-product(autocorr
elation)matrices and zeroing the main diagonal
(3-89)
where I denotes the n-by-n identity matrix .
Zeroing the main diagonal tends to improve recall
accuracy by helping the system transfer function
behave less like the identity operator.
97
Chapter 3. Neural Dynamics IIActivation Models
?3.7 Additive dynamics and the noise-saturation
dilemma
Grossbergs Saturation Theorem
Grossbergs Saturation theorem states that
additive activation models saturate for large
inputs, but multiplicative models do not .
98
Chapter 3. Neural Dynamics IIActivation Models
The stationary reflectance pattern
confronts the system amid the background
illumination
The i th neuron receives input .Convex
coefficient defines the reflectance
the passive decay rate
the activation bound
99
Chapter 3. Neural Dynamics IIActivation Models
Additive Grossberg model
We can solve the linear differential equation to
yield
For initial condition , as time
increases the activation converges to its
steady-state value
As
100
Chapter 3. Neural Dynamics IIActivation Models
So the additive model saturates.
Multiplicative activation model
101
Chapter 3. Neural Dynamics IIActivation Models
For initial condition ,the
solution to this differential equation becomes
As time increases, the neuron reaches steady
state exponentially fast
(3-96)
as .
102
Chapter 3. Neural Dynamics IIActivation Models
This proves the Grossberg saturation theorem
Additive models saturate ,multiplicative models
do not.
103
Chapter 3. Neural Dynamics IIActivation Models
In general the activation variable can
assume negative values . Then the operating range
equals for .In the
neurobiological literature the lower bound
is usually smaller in magnitude than the upper
bound
This leads to the slightly more general shunting
activation model
104
Chapter 3. Neural Dynamics IIActivation Models
Setting the right-hand side of the above equation
to zero, and we can get the equilibrium
activation value
which reduces to (3-96) if C0.
the neuron generates nonnegative activations.
105
Chapter 3. Neural Dynamics IIActivation Models
?3.8 General Neuronal ActivationsCohen-Grossberg
and multiplicative models
Consider the symmetric unidirectional or
autoassociative case when ,
, and M is constant . Then a neural network
possesses Cohen-Grossberg1983 activation
dynamics if its activation equations have the
form
(3-102)
The nonnegative function
represents an abstract amplification function.
106
Chapter 3. Neural Dynamics IIActivation Models
Grossberg1988has also shown that (3-102)
reduces to the additive brain-state-in-a-box
model of Anderson1977,1983 and the shunting
masking-field model Cohen,1987 upon appropriate
change of variables.
107
Chapter 3. Neural Dynamics IIActivation Models
If , ,
and constant
, where and are
positive constants , and input is constant
or varies slowly relative to fluctuations in
,then (3-102) reduces to the Hopfield
circuit1984
An autoassociative network has shunting or
multiplicative activation dynamics when the
amplification function is linear, and is
nonlinear .
108
Chapter 3. Neural Dynamics IIActivation Models
For instance , if ,
(self-excitation in lateral inhibition) , and
then (3-104) describes the distance-dependent
unidirectional shunting network
109
Chapter 3. Neural Dynamics IIActivation Models
Hodgkin-Huxley membrane equation
, and denote respectively
passive(chloride ) , excitatory (sodium
) , and inhibitory (potassium ) saturation
upper bounds .
110
Chapter 3. Neural Dynamics IIActivation Models
At equilibrium, when the current equals zero ,the
Hodgkin-Huxley model has the resting potential

Neglect chloride-based passive terms.This gives
the resting potential of the shunting model as
111
Chapter 3. Neural Dynamics IIActivation Models
BAM activations also possess Cohen-Grossberg
dynamics, and their extensions
with corresponding Lyapunov function L , as we
show in Chapter 6
112
Chapter 3. Neural Dynamics IIActivation Models
????!