Recurrent Networks

1
Recurrent Networks

• Partially and Fully recurrent networks feed
  forward vs. relaxation nets
• TDNN Time Delay Neural Networks (snapshot)
• Elman Nets simple standard training
• BPTT Backpropagation through time must pick
  depth
• Real-time recurrent learning, etc.

2
Recurrent Issues

• TDNN Speech example, what are best intervals?
• Elman net While execution can theoretically
  look back arbitrarily far in time, learning
  gradient goes back only 1 step in time.
• Typically an exponential drop-off in effect of
  prior inputs only so much that a few context
  nodes can be expected to remember.
• BPTT Common approach, but choosing proper depth
  is constraining and often difficult speech
  example
• RTRL n4 and n3 versions too expensive in
  practice

3
BPTT Backprop through time

• Train for k steps
• How do you fill in initial activations of context
  nodes
• Where/how to grab time slices from the training
  set
• Teacher forced training Initially force the
  outputs to desired values during training to give
  context nodes a jump-start

4
BPTT - Unfolding in Time
Outputk
Inputk
Output2
Input2
one step time delay
Outputk
Output1
one step time delay
Inputk
Input1