Perceptrons, Multilayer Networks

1
Lecture 3

• Perceptrons, Multilayer Networks
• (Chapts. 3 5, Wu McLarty)

2
Multilayer networks

• Perceptrons are restricted to data sets which are
  linearly separable.
• For this class of data, a perceptron can be
  trained with a finite number of iterations
• Multilayer neural networks include multiple
  layers of neurons.
• Hidden layers are sandwiched between input and
  output layers.
• Usually feature full connectivity between layers
  - typically hundreds of connections, each with
  its own weight.
• Multilayer networks can be trained to implement
  target functions which define surfaces in the
  descriptor space.

3
A multilayer network
hercules.ece.utexas.edu/course/
ee380l/1999sp/present/pholsiri.ppt
4
Multilayer Terminology

• Often called a feed-forward network, since
  signals flow in only one direction (toward the
  output layer)
• The first layer really serves to distribute the
  inputs to the neurons of the hidden layer this
  is done passively, with no application of a
  transfer function.
• Some disagreement in the literature as to whether
  or not to include the input layer in the count of
  layers.

5
Signal propagation in a three (two?) layer network
h1
W11o
W11h
xi
Ok
OM
xN
WDMo
WNDh
hD
6
Computing outputs
Transfer function
Hidden layer output
Output layer (target function)
7
Learning in multilayers
Minimize an error function
Find weights between layers so that
Where p h or o, and r and s take on all
appropriate values for each layer.
8
Backpropagation

• Calculating the weights needed to locate a
  stationary position (all derivatives 0) entails
  some sort of minimization procedure.
• The most popular method (due to Rumelhart) is
  called backpropagation.
• Intuitively, the method adjusts synaptic weights
  by propagating errors backward, from output to
  inputs, and using the back-propagated errors to
  adjust weights.
• It has a rigorous foundation based on the
  treatment shown in the previous slide.

9
BP Algorithm
For output unit,
Adjust weights to output unit
For hidden unit,
Adjust weights to hidden unit
10
BP Details

• Different algorithms exist for batch and on-line
  training.
• The derivation requires that the transfer
  function (phi) be continuous step functions are
  not allowed when using BP or similar training
  methods.
• The BP algorithm is not guaranteed to converge,
  and if it does converge there is no lower limit
  on the rate.

11
More Details

• Usually, one hidden layer is sufficient to model
  a complex target function however, the number of
  hidden units is an open question.
• The sizes of the input and output layers are
  generally determined by the problem statement
  the number of hidden units is determined
  heuristically.
• In most topologies, the network is fully
  connected (every output of layer j is connected
  to every input of j1).

12
Building Java Classes

• Chapter 4
• Lewis Loftus

13
Custom Classes

• We will first work through the example in Listing
  4.1, where we build a custom class to represent a
  die (as in one member of a pair of dice).
• Distinguish between instance and class variables
  public and private variables and methods.
• Principle Object should be self-governing it
  should be the only entity to modify its own data.

14
Methods Terminology

• Accessor method
• Accesses an instance or class variable
• Mutator method
• Modifys a class instance variable
• Parameters
• The arguments passed to a method distinguish
• Formal parameters
• Actual parameters/arguments

15
Constructor Methods

• Distinguished by having the same name as the
  associated class. Constructors initialize
  instances of a class.
• Constructores must NOT have any return type, not
  even void!

16
More graphics programs

• Take a look at the Smiley face and Splat
  programs. Try them out if you wish.

17
GUI Elements

• Components - objects that have a graphical face
  and can be drawn in a frame.
• Events - objects that correspond to a user
  action. A mouse click generates an event, as does
  dragging the mouse
• Listener - an object that can intercept and
  evaluate events.

18
Simple GUIs

• We will examine the PushCounter and Fahrenheit
  examples from the text (Listings 4.10-11 and
  4.12-13)
• These simple examples actually demonstrate most
  of the principles you will need in your own
  programming work in BI785.