Stat 6601 Project: Neural Networks V

1
Stat 6601 ProjectNeural Networks(VR 6.3)

• Group Members
• Xu Yang
• Haiou Wang
• Jing Wu

2
Definition

• Neural Network
• A broad class of models that mimic functioning
  inside the human brain
• There are various classes of NN models.
• They are different from each other depending on
• (1) Problem types, prediction, Classification ,
  Clustering
• (2) Structure of the model
• (3) Model building algorithm
• We will focus on feed-forward neural network.

3
A bit of biology . . .

• Most important functional unit in human brain a
  class of cells called
• NEURON

Neurons
4
An Artificial Neuron
5
Simplest but most common form (One hidden layer)
6
Choice for Activation function
7
A collection of neurons form a layer
Input Layer - Each neuron gets ONLY one
input, directly from outside
Hidden Layer - Connects Input and Output layers
Output Layer - Output of each neuron
directly goes to outside
x1
x2
x3
x4
8
More general format

• Skip-layer connections

9
Fitting criteria

• Least squares
• Maximum likelihood
• Log likelihood
• One way to ensure f is smooth E?C(f )

10
Usage of nnet in R

• nnet.formula(formula, dataNULL, weights, ...,
  subset, na.actionna.fail, contrastsNULL)
• size number of units in the hidden layer. Can be
  zero if there are skip-layer units.
• Wts initial parameter vector. If missing chosen
  at random.
• linout switch for linear output units. Default
  logistic output units.
• entropy switch for entropy ( maximum
  conditional likelihood) fitting. Default by
  least-squares.
• softmax switch for softmax (log-linear model)
  and maximum conditional.
• skip Logical for links from inputs to outputs.
• formula A formula of the form 'class x1 x2
  ...'
• weights (case) weights for each example - if
  missing defaults to 1.
• rang if Wts is missing, use random weights from
  runif(n, -rang, rang).
• decay Parameter ?.
• maxit maximum of iterations for the optimizer.
• Hess Should the Hessian matrix at the solution
  be returned?
• trace logical for output form the optimizer.

11
An Example

• Code
• library(MASS)
• library(nnet)
• attach(rock)
• area1lt-area/10000 peri1lt-peri/10000
• rock1lt-data.frame(perm, areaarea1, periperi1,
  shape)
• rock.nnlt-nnet(log(perm)area peri shape,
  rock1, size3, decay1e-3, linoutT, skipT,
  maxit1000, hessT)
• summary(rock.nn)

12
Output

• gt summary(rock.nn)
• a 3-3-1 network with 19 weights
• options were - skip-layer connections linear
  output units decay0.001
• b-gth1 i1-gth1 i2-gth1 i3-gth1
• 9.48 -7.39 -14.60 6.94
• b-gth2 i1-gth2 i2-gth2 i3-gth2
• 1.92 -11.87 -2.88 7.36
• b-gth3 i1-gth3 i2-gth3 i3-gth3
• -0.03 -11.12 15.61 4.62
• b-gto h1-gto h2-gto h3-gto i1-gto i2-gto i3-gto
• 2.64 3.89 11.90 -17.76 -0.06 4.73 -0.38
• gtsum((log(perm)-predict(rock.nn))2)
• 1 11.39573

• weights 19
• initial value 1712.850737
• iter 10 value 34.726352
• iter 20 value 32.725356
• iter 30 value 30.677100
• iter 40 value 29.430856
• .
• iter 140 value 13.658571
• iter 150 value 13.248229
• iter 160 value 12.941181
• iter 170 value 12.913059
• iter 180 value 12.904267
• iter 190 value 12.901672
• iter 200 value 12.900292
• iter 210 value 12.899496
• final value 12.899400
• converged

13
Use the same method from previous section to view
the fitted surface

• Code
• Xp lt- expand.grid(area seq(0.1, 1.2, 0.05),
  peri seq(0, 0.5, 0.02), shape 0.2)
• trellis.device()
• rock.grid lt- cbind(Xp, fit predict(rock.nn,Xp))
• S Trellis 3D Plot
• wireframe(fit area peri, rock.grid, screen
  list(z 160, x -60), aspect c(1, 0.5), drape
  T)

14
Output
15
Experiment to show key factor which affects the
degree of fit

• attach(cpus)
• cpus3 lt- data.frame(syct syct-2, mmin mmin-3,
  mmax mmax-4, cach cach/256, chmin
  chmin/100, chmax chmax/100, perf perf)
• detach()
• test.cpus lt- function(fit)
• sqrt(sum((log10(cpus3perf) - predict(fit,
  cpus3))2)/109)
• cpus.nn1 lt- nnet(log10(perf) ., cpus3, linout
  T, skip T, size 0)
• test.cpus(cpus.nn1)
• 1 0.271962
• cpus.nn2 lt- nnet(log10(perf) ., cpus3, linout
  T, skip T, size 4, decay 0.01, maxit
  1000)
• test.cpus(cpus.nn2)
• 1 0.2130121
• cpus.nn3 lt- nnet(log10(perf) ., cpus3, linout
  T, skip T, size 10, decay 0.01, maxit
  1000)
• test.cpus(cpus.nn3)
• 1 0.1960365
• cpus.nn4 lt- nnet(log10(perf) ., cpus3, linout
  T, skip T, size 25, decay 0.01, maxit
  1000)
• test.cpus(cpus.nn4)
• 1 0.1675305