NEURAL NETWORKS

1
NEURAL NETWORKS
2
Introduction

• The branch of AI known as "neural networks" or
  "artificial neural networks" has become an
  increasingly important area of AI.
• Neural networks approaches building intelligent
  systems using architectures and processing
  capabilities that mimic some biological processes
  of the human brain.

3
Basics

• Biological system
• Brains are composed of neurons, which have the
  unique characteristic that of all types of cells
  in the body, they do not die.
• biological neural networks
• http//psych.hanover.edu/Krantz/neurotut.html
• good basic tutorial and introduction

4
Portion of a Network Two Interconnected
Biological Cells
5
Biological system

• Human brains are estimated to contain up to 100
  billion neurons (1011) and hundreds of different
  types.
• Neurons are found in groups called networks (now
  you know why this area of AI is called neural
  networks), each containing thousands of highly
  interconnected neurons.

6
Biological system

• dendrites - takes in information, provide inputs
  to the cell, note they have plenty of surface
  area to facilitate connection to axons of other
  cells
• axon protuberance that delivers outputs from
  the neuron to connections with other neurons

7
Biological system

• A neuron does nothing until the collective
  influence of all its inputs reaches a threshold
  level.
• At that point, the neuron produces a
  full-strength output in the form of a narrow
  pulse that proceeds from the cell body, down the
  axon, and into the axons branches.
• it fires Since it fires or does nothing it is
  considered an all or nothing device.

8
Biological system

• synapse gap between the axon and dendrites.
• Stimulation at some synapses will encourage the
  neuron to fire, while stimulation at others
  discourages the neuron from firing.
• Increases or decreases the strength of connection
  and causes excitation or inhibition of a
  subsequent neuron

9
Artificial Systems

• Simulated neurons are viewed as a node connected
  to other nodes via links that correspond to
  axon-synapse-dendrite connections.
• Each link is associated with a weight.
• The weight determines the nature (/-) and
  strength of the nodes influence on another.
• If the influence of all the links is strong
  enough the node is activate (similar to the
  firing of a neuron).

10
Processing Information in an Artificial Neuron
11
Artificial Systems (continued)

• Processing element
• Think of a PE as an artificial neuron.
• Receives inputs, processes the outputs, and
  delivers a single output.
• Inputs can be received as raw data or from
  another PE

12
Artificial Systems (continued)

• Network
• Composed of a collection of PE's grouped in
  layers.
• This example has three layers, the middle layer
  is referred to as the hidden layer.
• Structure
• Several possible structures

13
Artificial Neural Network with One Hidden Layer
14
Processing Information in an Artificial Neural
Network

• Inputs
• Each input is a value of a single attribute.
• If I wanted to predict stock prices, one
  attribute of interest might be "volume" and
  therefore, I would input the volume (number of
  shares sold) on a specific day as one input.

15
Processing Information in an Artificial Neuron
16
Processing Information in a Network

• Outputs
• Solution to the problem. For instance, the
  projected price of the stock

17
Processing Information in an Artificial Neural
Network

• Weights
• Used to express the relative strength of an input
  value or from a connecting PE (i.e., in another
  layer).
• These weights are essential, it is by adjusting
  these weights that a neural network learns.

18
Processing Information in a Network

• Summation function
• Used to compute a single value (weighted average)
  from all the inputs to a particular PE.
• Think of it as the internal stimulation or
  activation level of the neuron

19
Processing Information in an Artificial Neural
Network

• Transformation (Transfer) Function
• based on the summation value,
• the value, the transformation (transfer) function
  produces an output.

20
Summation Function for Single Neuron(a) and
Several Neurons(b)
21
Processing Information in a Network

• There are many possible transformation functions,
  the sigmoid function is popular.
• Sometimes a threshold value is used, which is
  easier to explain and understand.

22
Processing Information in a Network

• For instance, for summation values less than .5 a
  0 might be output, for summation values greater
  than or equal to .5 a 1 might be output

23
Learning Training in Neural Networks

• Neural networks are trained using data referred
  to as a training set.
• The process is one of computing outputs, compare
  outputs with desired answers, adjust weights and
  repeat.

24
Learning Training in Neural Networks

• It is necessary to have a fairly large training
  set, and you need to have the answer for each
  case in the training set.
• Discrepancies between the "right answer" (from
  the training set) and the computed answer are
  measured and based on the error, adjustments made.

25
History of Neural Networks

• Basic research on brains dates back quite far.
• 1791 - Luigi Galvani (from Bologna) stimulated a
  frog's muscles with electricity, leading to the
  discovery that the brain has electrical activity
• 1837 - Gocli observed the structure of neurons
  with axons and connections to dendrites

26
History of Neural Networks

• 1887 - Sherrington Synaptic interconnection
  suggested
• 1920's - discovered that neurons communicate via
  chemical impulses called neurotransmitters.
• 1930's - research on the chemical processes that
  produce the electrical impulses.

27
History of Neural Networks

• 1943 - McCullock and Pitts showed that a NN could
  be used to code logical relationships such as
• "x AND y" or "x OR y"
• 1950's - Hodgkin and Huxley were awarded the
  Nobel Prize for work developing the model and
  recording the electrical signal of the brain at
  the cellular level

28
History of Neural Networks

• 1969 - Minsky and Papert wrote Perceptrons
• showed that one-layer neural networks could not
  handle statements such as
• (x AND NOT y) OR (y AND NOT x)
• Based on this finding, they conjectured that
  multi-level NN's would not perform better
• Result funding for NN research dried up, for
  about 10 years

29
History of Neural Networks

• 1987 - Robert Hecht-Nielsen mathematically
  disproved Minsky's and Papert's conjecture
  regarding multi-layer neural networks not being
  able to perform better than one-layer neural
  networks.
• Since then, this area has been subject to more
  research.

30
Basic Network Structures

• associative - single layer is representative
• hidden layer - can have more than one hidden
  layer, note that it is uni-directional
• double-layer - feeds forward and backward,
  develops its own categories for representing the
  data

31
Neural Network Structures
32
Artificial Neural Network Develop-ment Process
Get More, Better Data
Refine Structure
Select Another Algorithm
Reset
Reset
33
Developing Neural Networks

• Step 1
• collect data
• Step 2
• separate data into training and test sets,
  usually random separation
• ensure that application is amenable to a NN
  approach

34
Developing Neural Networks

• Step 3
• define a network structure
• Step 4
• select a learning algorithm
• affected by the available tools shells available
  
• Step 5
• set parameter values
• affects the length of the training period

35
Developing Neural Networks

• Step 6
• transform Data to Network Inputs
• data must be NUMERIC, may need to preprocess the
  data, e.g., normalize values for a range of 0 to
  1
• Step 7
• start training
• determine and revise weights, check points
• Step 8
• stop and test iterative process

36
Developing Neural Networks

• Step 9
• implementation
• stable weights obtained
• begin using the system

37
Example - Financial Market Analysis

• Karl Bergerson of Neural Trading Co uses Neural,
  a trading systems with BrainMaker and a C-based
  E.S. for money-management rules.
• Using 9 years of hand-picked financial data,
  trained the NN and ran it against a theoretical
  10,000 investment.
• After 2 years, the fictional account had grown to
  76,034 (660 appreciation).

38
Financial Market Analysis - continued

• When tested on new data, 89 accurate.
• Developer quoted "Neural nets are the best tools
  for pattern recognition, but you can't just dump
  data into one and expect to get wonderful
  results. The most important factor is your
  training data. You have to have your whole act
  together, training, design, and the right tools.

39
Financial Market Analysis - continued

• Some of the attributes used price, volume,
  advance/decline etc.
• The neural network predicts market fluctuation
  and the expert system component flags buying or
  selling opportunities

40
Sales Support

• Veratex Corp. distributes medical and dental
  products.
• They send unsolicited catalogs to physicians and
  dentists.
• When a customer buys something, their name is
  added to the customer database.
• 40 telemarketers then call the names in the
  database for reorders

41
Sales Support - continued

• The problem
• many dormant accounts, i.e., customers who had
  not placed reorders.
• The telemarketers are not trained to prospect for
  new clients and they only have about 20 of their
  time allocated for calling dormant accounts.

42
Sales Support - The Problem (continued)

• The database contains 44,000 customers, these
  represent potential business that is not being
  tapped.
• Further, as the data ages, it becomes less
  reliable (i.e., physicians and dentists move and
  retire).

43
Sales Support - continued

• The solution
• The company hired Churchill Systems to build a
  back-propogation (a learning algorithm) to
  identify those customers in the dormant pool most
  likely to place reorders.
• With this information, telemarketers could focus
  their limited time on customers with the most
  potential.

44
Sales Support - continued

• System was built using NNU400 neural network
  utility (from IBM).
• Inputs consisted of statistical and demographic
  data culled from Dun Bradstreet and other
  sources.
• The network was applied against the customer
  list, giving each customer a numerical rating
  which was put into the customer records and then
  used as a sort key.

45
Sales Support

• Results
• President of Churchill Systems "More Veratex
  accounts were reopened in five months, than
  similar periods (without the network).
• "The patterns and
• interrelationships uncovered by the neural
  network proved to be an extremely valuable
  resource for Veratex marketing analysts."

46
Sales Support - continued

• General comments
• "A lot of people think you can avoid knowledge
  engineering. Forget it - you can't do it. You
  really have to get down to the business problem
  before you can do anything else.
• In fact, Light claims that a large part of the
  neural network's development time entailed
  gathering, cleaning up, and organizing the
  appropriate data.

47
Horse Bloodlines

• University of California at Davis School of
  Veterinary Medicine conducts blood tests to
  confirm the bloodlines of Thoroughbred horses.
• Thoroughbreds cannot be raced unless their
  bloodlines are known.
• To do this, 142 separate reaction tests must be
  run on a blood sample.
• as many as 72,000 tests per day.

48
Horse Bloodlines - continued

• The problem how to automate this function so
  that a technician didn't have to perform this
  job.
• A neural network was trained to read these tests
  starting April 1987, was pilot tested for about
  one year (1989?, article printed in 1990).

49
Horse Bloodlines - continued

• The neural network was trained to read a blood
  test and determine if a reaction occurred - a
  simple yes or no.
• The neural network must be accurate, the results
  of the lab cannot be questioned or breeders will
  not use the lab.

50
Horse Bloodlines - continued

• Specifically, the network must "look" at a drop
  of blood and decide if the cells in the drop of
  blood have clumped together (agglutinated).
• The system "sees" using a video camera that
  divides the field of vision into 262,144 pixels
  of information.

51
Horse Bloodlines - continued

• Using this information, the developer believed it
  would have taken 28 million years to teach the
  network the concept of "clumpiness".
• It was just too much raw data.

52
Horse Bloodlines - continued

• Lendaris (1970) pioneered the scanning of aerial
  surveillance photographs by computer to detect
  orderly man-made features such as orchards, road
  intersections, etc.
• The contribution that could be applied to the
  blood testing problem was the use of the Fourier
  transform, developed by a 19th-century French
  physicist and mathematician.

53
Horse Bloodlines - continued

• This transformation converts massive amounts of
  data into oscillating waves of energy and can be
  used to highlight sharp gradations, such as the
  edge of a building or the edge of a clump of
  blood.
• The Fourier transformed 262,144 pixels into 48
  data points that the network was easily trained
  to recognize.

54
Horse Bloodlines - continued

• The neural network tool used was supplied by
  Science Application International Corp (of San
  Diego), cost 25,000 and called Delta II.
• It includes both software and an accelerator
  board to enhance a 386 machine.

55
Horse Bloodlines - continued

• The developer of the system to read blood tests
  is skeptical about the commercial development of
  Neural Nets
• "The difficulty is, what will you sell? A neural
  net is just an algorithm - a method of
  calculation like a statistical regression or
  multiplication. It is hard to protect a product
  like that - hard to get a commercial handle on
  it."

56
Horse Bloodlines - continued

• He believes that what will succeed are a variety
  of hardware systems with the neural network
  learning method automated and embedded in the
  hardware.
• This will also alleviate the user from having to
  understand as much about the neural network.

57
KBS vs. Neural Networks
58
(No Transcript)
59
Advantages of Neural Nets

• Able to learn any complex non-linear mapping (31)
• Do not make a priori assumptions about the
  distribution of the data/input-output mapping
  function (30)
• Very flexible with respect to incomplete,
  missing, noisy data, fault tolerant (29)
• Easily updated, suitable to dynamic environments
  (15)
• Overcome some limitations of other statistical
  methods, while generalizing them (15)
• Hidden nodes, in feed-forward, can be regarded as
  latent/unobservable variables (5)
• Can implement on parallel hardware, increasing
  accuracy and learning speed (4)
• Can be highly automated, minimizing human
  involvement (3)
• Specially suited to tackle problems in
  non-conservative domains (3)

60
Disadvantages of Neural Nets

• Lack theoretical background, no explanation,
  black box (28)
• Selection of network topology and parameters
  lacks theoretical background, trial and error
  (21)
• Learning process can be very time consuming (11)
• Can overfit the training data, becoming useless
  for generalization (10)
• No explicit set of rules to select a suitable ANN
  paradigm/learning algorithm (8)
• Too dependent on the quality/ amount of data
  available (6)
• Can get stuck in local optima, narrow valleys
  during training (5)
• Techniques still rapidly evolving and not
  reliable or robust enough yet (3)
• Lack classical statistical properties.
  Confidence intervals and hypothesis testing are
  not available (2)

61
Potential Problems with Neural Networks

• The military has been experimenting with ANN
  techniques for sometime.
• One application of interest was to identify
  objects on the battlefield at night. For
  instance, distinguishing the difference between a
  tank and a rock.
• A scanner showed an automated neural network
  thousands of photographs of tanks, rocks, and
  other battlefield objects.

62
Potential Problems with Neural Networks (cont.)

• After training the ANN could correctly
  distinguish a tank from a rock 100 of the time.
• Later, it was discovered that all the photos of
  the tanks had been taken with the same camera.
• The tank photos were all slightly darker than the
  photos of the other objects.
• What the ANN had really learned was to identify
  the camera used to take the picture, not the
  difference between rocks and tanks!