Associative%20Memory%20by%20Recurrent%20Neural%20Networks%20with%20Delay%20Elements

1
Associative Memory by Recurrent Neural Networks
with Delay Elements

• Seiji MIYOSHI Hiro-Fumi YANAI Masato
  OKADA

Kobe City College of Tech. Ibaraki
Univ. RIKEN BSI , ERATO KDB
JAPAN JAPAN
JAPAN
miyoshi_at_kobe-kosen.ac.jp www.kobe-kosen.ac.jp/miy
oshi/
2
Background

• Synapses of real neural systems seem to have
  delays.

• It is very important to analyze associative
  memory model with delayed synapses.

• Computer simulation is powerful method.

There is a Limit on the number of neurons.
However,
Simulating network with large delay steps is
realistically impossible.

• Theoretical and analytical approach is
  indispensable to research on delayed networks.

• Yanai-Kim theory by using Statistical
  Neurodynamics

Good Agreement with computer simulation
Computational Complexity is O(L4t)
3
Objective

• To derive macroscopic steady state equations by
  using discrete Fourier transformation

• To discuss storage capacity quantitatively even
  for a large L limit (L length of delay)

4
Recurrent Neural Network with Delay Elements
Model
5
Model
6
Macrodynamical Equations by Statistical
NeurodynamicsYanai Kim(1995) Miyoshi, Yanai
Okada(2002)
7
Initial Condition of the Network

• One Step Set Initial Condition
• Only the states of neurons are set explicitly.
• The states of delay elements are set to be zero.

• All Steps Set Initial Condition
• The states of all neurons and all delay elements
  are set to be close to the stored pattern
  sequences.
• If they are set to be the stored pattern
  sequences themselves
• Optimum Initial Condition

8
Dynamical Behaviors of Recall Process
All Steps Set Intial Condition Loading
ratea0.5 Length of delay L3
Theory
Simulation(N2000)
9
Dynamical Behaviors of Recall Process
All Steps Set Intial Condition Loading
ratea0.5 Length of delay L2
Theory
Simulation(N2000)
10
Loading rates a - Steady State Overlaps m
Theory
Simulation(N500)
11
Length of delay L - Critical Loading Rate aC
12
Macrodynamical Equations by Statistical
NeurodynamicsYanai Kim(1995) Miyoshi, Yanai
Okada(2002)

• Good Agreement with Computer Simulation

• Computational Complexity is O(L4t)

13
Macroscopic Steady State Equations

• Accounting for Steady State
• Parallel Symmetry in terms of Time Steps
• Discrete Fourier Transformation

14
Loading rates a - Steady State Overlaps m
15
Loading rates a - Steady State Overlaps m
Theory
Simulation(N500)
16
Loading rate a - Steady State Overlap
17
Storage Capacity of Delayed Network
Storage Capacity 0.195 L
18
Conclusions

• Yanai-Kim theory (macrodynamical equations for
  delayed network) is re-derived.

? Computational Complexity is O(L4t)
? Intractable to discuss macroscopic
properties in a large L limit

• Steady state equations are derived by using
  discrete Fourier transformation.

? Computational complexity does not formally
depend on L
? Phase transition points agree with those
under the optimum initial conditions, that is,
the Storage Capacities !

• Storage capacity is 0.195 L in a large L limit.