Introduction to Neural Networks

1
Introduction to Neural Networks

• William Dardick
• February 6, 2006

2
Overview

• Biology
• Brief background PDP
• Neural networks
• Statistics
• Learning
• Cognitive Modeling

3
Biological Background

• Your Brain is a neural network
• What is a Neuron?

4
Human Brain

• 100 Billion neurons
• 2000 connections
• 1000 action potentials per second
• Massive Parallelism

5
Real Neurons
Anderson, J.R. (2000).

• Electrical signals from environment or other
  neurons (at dendrites)
• Some increase electrical potential in the cell
  (excitatory)
• Some decrease electrical potential in the cell
  (inhibitory)
• When the cells threshold is reached, it sends a
  signal (through axon) to other neurons or organs
  (e.g., muscle cells)

6
Key concepts from neuron

• Input
• dendrites
• Action potential
• Output
• Axon/Telodendria
• synapse

7
Parallel Distributed ProcessingConnectionism
Theory

• PDP is a new Model
• Artificial Intelligence
• Cognitive Theories of the Mind
• Model of Brain Function
• the notion that intelligence emerges from the
  interactions of large numbers of simple
  processing units. (Rumelhart, McClelland, 1989.)

8
Parallel Distributed Processing

• Parallel Nature to the Processing
• Many things happening at once, not just serial.
• Distributed representation and Distributed
  control
• Information is taken in through out the model
• General processing system
• Information and memory is stored in the process.

9
Neural Networks Basic concepts

• Many types of Networks
• Focus on Multilayer Perceptron
• Parts
• Some Statistics Stuff
• Learning

10
Neural Network Parts

• Vectors
• Units
• Weights
• Connections

11
Vectors

• Describes a pattern of numbers
• Many numbers can be listed under one symbol
• Ex. Height, Weight, Age, Eye color, shoe size
• Ex. Points on a grid

12
Units

• Node/Perceptron/Cells/Processing entities
• Smallest portion of neural network
• Similar to Neuron
• Takes in information from other unit or vector
• performs function on input
• Puts out information to next unit/output

13
Layers of Units

• Input Layer
• Receives information from outside source
• Sends to inside source
• Hidden Layer
• Receives information from inside source
• Sends to inside source
• Output Layer
• Receives information from inside source
• Sends to out side source

14
Weights

• Level of the charge at synapse/connection
• Adjust to get desired output
• Strengthened at synapses, connection points
• allows for pattern to be stored

15
One Perceptron
16
Example

• Vector levels would be the strength of the
  electrical charge.
• This is the weight Size
• This enters the Unit and goes through a function
• It comes out and goes to the next Unit as a new
  electrical charge

17
Connections

• Key portion of parallelism in Neural Networks
• Connects units
• similar to telodendria/dendrites
• One unit has many connection, Input and Output

18
Types of Neural Networks

• Supervised
• Teacher Computer
• Feedback
• Feedforword-only

• Unsupervised
• Fully Autonomous
• Feedback
• Feedforword-only

19
Fully Connected Multilayer Perceptron,
Feedforward Model

• Input, Hidden Layers, Output

20
Fully Connected Recurrent model

• All connections feedforward are used.
• Some connections recursive (feedback) used.

21
Statistical Concepts

• Regression
• Terminology
• Jargon
• Slang
• Lingo
• and other stuff that means the same thing

22
Regression model

• Y A b1Xb2XbiX

23
The Lingo

• See handout for sample terms of Jargon
• Warren S. Sarle

24
Backpropagation

• Hebian learning rule
• Predecessor
• Delta rule
• Generalized Delta Rule

25
Backpropagation

• Randomly assigned weights
• Forward pass
• Backward pass
• iterate

26
Backpropagation

• Repeated cycles over training set,
• Each time increase by an amount the weights for
  inputs that were right and decrease those that
  were wrong, based on partial derivatives.
• Propagate back down through the layers.
• Do until no systematic change in weights, or
  outputs for a criterion set of inputs.
• See a tutorial on machine learning from Leeds
  University http//cbl.leeds.ac.uk/nikos/pail/intm
  l/subsection3.11.4.html

27
Cognitive Theory

• PDP works well as a theory for certain aspects of
  cognition.
• Modeled after the Brain
• Used as model for Brain Function
• Hard Questions of Cognition- inner life
  experience.

28
Interdisciplinary Field

• Psychologists
• Philosophers
• Computer Scientists
• Engineers
• Cognitive Scientists
• Physicists
• Neurologists

• Mathematicians
• Statisticians
• Medical Doctors
• Marketing
• All of these field and many more contribute and
  use PDP/NN.

29
Modeling

• Alternative theoretical framework
• Issues
• Biologically plausible model
• Nontrivial learning
• Account for higher level cognition
• Examples Fundamentals of Neural Network Modeling

30
Model Examples

• Waking and Sleeping States
• Sutton and Hobson
• Lexical Retrieval
• Tranel, Damasio, Damasio
• Alzheimer's Disease
• Chan, Salomon, Butters