Neural Networks: An Introduction and Overview

1
Neural Networks An Introduction and Overview

• Jim Ries
• NLM Predoctoral Fellow
• JimR_at_acm.org
• 6/13/2000

2
Introduction

• Provide an intuitive feel for what NNs are and
  problems for which they are an appropriate tool.
• NOT overwhelm you with mathematics.
• Caveat Im not an NN researcher just an
  interested outsider (like most of you).

3
Topics of Discussion

• What are Neural Networks?
• Training
• History
• Alternative Methods
• Applications
• Conclusions
• Questions

4
What are Neural Nets?

• A mechanism for approximating a function, given
  some sample or training data.
• A mechanism for classifying, clustering, or
  recognizing patterns in data.
• These two broad applications are essentially the
  same (e.g., imagine a function that outputs a
  discrete number indicating a cluster).

5
What are Neural Nets? (cont.)

• Rosenblatts Perceptron a network of processing
  elements (PE)

Y1
Yp
a1
am
. . .
x1
x2
x3
xn
. . .
6
What are Neural Nets? (cont.)

• Additional layer(s) can be added

Y1
Yp
a1
am
. . .
h1
hm
. . .
x1
x2
x3
xn
. . .
7
What are Neural Nets? (cont.)
8
What are Neural Nets? (cont.)

• A node (PE) is typically represented as a
  function.
• Simple functions can quickly be trained or
  updated to fit a curve to data, but are unable to
  fit well to complex data (e.g., linear functions
  can never approximate quadratics).
• Universal Approximator! (typically Radial Basis
  Function).

9
Training

• With simple Perceptron model, we can train by
  adjusting the weights on inputs when the output
  does not match test data.
• The amount of adjustment we do at each training
  iteration is called the learning rate.

10
Training (cont.)

• With one or more hidden layers, training requires
  some sort of propagation algortihm.
• Backpropagation is commonly used and is an
  extension to the Minimum Disturbance Algorithm

11
Training (cont.)

• Minimum Disturbance Algorithm
• 1) Apply an example, propagate inputs to output
• 2) Count of incorrect output units
• 3) For output units, do a number of times
• Select unselected units closest to zero
  activation
• Change weights
• if less errors, use new weights, else old
• 4) Repeat step 3 for all layers

12
Training (cont.)

• Overfitting - fits a function to training data,
  but does not approximate real world.
• Ways to avoid overfitting
• Regularization (assumes real function is
  smooth.
• Early stopping
• Curvature-driven

13
History

• Early 1960s - Rosenblatts Perceptron
  (Rosenblatt, F., Principles of Neurodynamics, New
  York Spartan Books, 1962).
• Late 1960s - Minsky (Minsky, M. and Papert, S.,
  Perceptrons, MIT Press, Cambridge, 1969).
• 1970s early 1980s - largely empty of NN
  activity due to Minsky.

14
History (cont.)

• Late 1980s - NN re-emerge with Rumelhart and
  McClelland (Rumelhart, D., McClelland, J.,
  Parallel and Distributed Processing, MIT Press,
  Cambridge, 1988).
• Since PDP there has been an explosion of NN
  literature.

15
Alternative Methods

• Classical statistical methods
• Fail in on-line scenarios
• Not universal approximators (e.g., linear
  regression)
• Assume normal distribution.
• Symbolic approach.
• Expert Systems
• Mathematical Logic (e.g., Prolog)
• Schemas, Frames, or Scripts

16
Alternative Methods (cont.)

• NNs are the Connectionist approach.
• Encoding of data can be a creative endeavor
• Ensemble Approach
• Baysian Networks
• Fuzzy NN

17
Applications

• Control
• Forecasting
• Provide faster approximations compared to exact
  algorithms (e.g., NeuroBlast).
• Compression
• Cognitive Modeling

18
Conclusions

• NNs are useful for a wide variety of tasks, but
  care must be taken to choose the correct
  algorithms for a given problem domain.
• NNs are not a panacea, and other approaches may
  be appropriate for given problems.

19
Questions?