Neural Networks Chapter 6

1
Neural NetworksChapter 6

• Joost N. Kok
• Universiteit Leiden

2
Feedforward networks
3
Feedforward networks
4
Feedforward Networks
5
Feedforward Networks
6
NetTalk

7
NetTalk
8
Feedforward Networks

• A network to pronounce English text
• 7 x 29 input units
• 1 hidden layer with 80 hidden units
• 26 output units encoding phonemes
• Trained by 1024 words with context
• Produces intelligible speech after 10 training
  epochs

9
Feedforward Networks

• Functionally equivalent to DEC-talk
• Rule-based DEC-talk is the result of a decade of
  efforts by many linguists
• NETtalk learns from examples, and requires no
  linguistic knowledge

10
Back-Propagation
11
Back-Propagation
12
Back-Propagation
13
Back-Propagation
14
Back-Propagation

• Initialize the weights to small random values
• Choose a pattern and apply it to the input layer
• Propagate the signal forwards through the network
• Compute the deltas for the output layer

15
Back-Propagation

• Compute the deltas for the preceding layers by
  propagating the errors backwards
• Update all the connections
• Go back to the second step for the next pattern

16
Feedforward Networks
17
Feedforward Networks
18
Navigation of a Car

• Carnegie-Mellon
• 30 times 32 pixel image
• 8 times 32 range finder
• 29 hidden units, 45 output units
• 1200 simulated road images, 40 training cycles
• 5km/hr

19
Feedforward Networks
20
Backgammon

• Score from 100 to 100
• 3000 examples
• 459 inputs
• Two hidden layers of 24 nodes
• Neurogammon vs. Gammontool 59 percent
• Without precomputed features 41 percent
• Without noise 45 percent

21
Feedforward Networks
22
Feedforward Networks
23
(No Transcript)
24
Parity Problem

• Parity Problem Output is on if an odd number of
  inputs is on

25
Back-Propagation
26
Back-Propagation
27
Back-Propagation
28
Back-Propagation
29
Back-Propagation

• The update rule is local
• Incremental weight updating vs. batch mode
• Momentum accelerate the long term trend by a
  factor

30
Back-Propagation

• Adaptive parameters

31
Feedforward Networks

• Process Modeling and Control
• Machine Diagnostics
• Portfolio Management
• Target Recognition
• Medical Diagnosis
• Credit Rating

32
Feedforward Networks

• Targeted Marketing
• Voice Recognition
• Financial Forecasting
• Quality Control
• Intelligent Searching
• Fraud Detection

33
Optimal Network Architectures

• Optimization
• Use as few units as possible
• Improve computational costs and training time
• Improve generalization
• Search through space of possible architectures,
  for example using Back-Propagation and
  Evolutionary Algorithms

34
Optimal Network Architectures

• Construct or modify architecture
• Start with too many nodes and take some away
• Start with too few and add some more

35
Optimal Network Architectures

• Pruning and weight decay

36
Optimal Network Architectures

• Small weights decay more rapidly than large ones

37
Optimal Network Architectures

• We want to remove units use same for all
  connections feeding unit i

38
Optimal Network Architectures

• Start with small network and gradually grow one
  of the appropriate size
• Boolean function from N binary inputs to single
  binary output

39
Optimal Network Architectures
40
Optimal Network Architectures

• Choose hidden units such that
• Same output for all remaining patterns with one
  target
• Opposite output for at least one of the remaining
  patterns with opposite target and remove these
  patterns
• Linearly separable problem

41
Optimal Network Architectures
42
Optimal Network Architectures

• We do the best we can with single node
• Correct with two nodes
• One for wrongly on patterns
• One for wrongly off patterns
• Each additional unit reduces the number of
  incorrectly classified patterns by at least one

43
Optimal Network Architectures
44
Optimal Network Architectures

• Faithful representation two patterns with
  different targets should have different
  representations
• Master unit does as well as possible on the task
• Ancillary units added to obtain faithful
  representation