Elements of Statistical Learning Hastie, Tibshirani, Friedman

1
Elements of Statistical LearningHastie,
Tibshirani, Friedman

• Chap 11 Neural networks

2
Projection Pursuit Regression

Approximate
with
unspecified functions
unit p-vectors
Minimize
3
Ridge function

varies in one direction only
4
Fitting PPR

• Given construct (e.g. with splines)
• Given minimize E over (Gauss Newton)

5
Fitting PPR

do

• Given construct (e.g. with splines)
• Given minimize E over (Gauss Newton)

until convergence
6
Fitting PPR

for m 1 M
do

• Given construct (e.g. with splines)
• Given minimize E over (Gauss Newton)

until convergence
add to f
7

Artificial Neural Networks
SVM ICA
8
Multilogit model

9
Multilogit models with bias

1
a0m
X1
a1m
a2m
X2
Zm


apm
Xp
10
Sum of multilogit models

1
1
aij
X1
b0
Z1
b1
X2
b2
Z2
T



bM
ZM
Xp
11
Multilayer Perceptron

1
1
aij
X1
b0
Z1
b1
X2
b2
Z2
Y



bM
ZM
Xp
12
Multilayer Perceptron

1
1
apm
bmk
X1
or
Y1
Z1
X2
Z2
Y2




or
ZM
YK
Xp
13
PPR vs MLP

where
14
Fitting MLP

With training samples
Fit parameters
To minize error criterion
15
Back - propagation

1
1
apm
bmk
X1
Y1
Z1
Gradient descent update
X2
Z2
Y2




ZM
YK
Xp
Gradient expression
16

1
Back - propagation
1
apm
bmk
X1
Y1
Z1
Gradient descent update
X2
Z2
Y2




ZM
YK
Xp
Gradient expression
17
Improvements

• Conjugate gradient
• Momentum
• Variable metric
• Levenberg Marquardt

if has been positive
for a while
First order development
and exact resolution
18
Starting values

is roughly linear when is small
Begin linear then evolve nonlinear
19
Avoiding overfitting

• Early stopping
• Penalization

Error
Stop before getting to global minimum
validation
training
Nb epochs
Error function
20
Avoiding overfitting

• Build several models Mi
• and use validation set

• Pruning
• Using validation set
• AIC, BIC criteria

1
1
apm
bmk
X1
Y1
Z1
X2
Z2
Y2




ZM
YK
Xp
21
Other Issues

• Scaling of the inputs
• Number of hidden units and layers
• Multiple minima

22
Radial Basis Function Networks

Output
with
Hidden layer
. . .
Inputs
Three parameters ci, si, lj
23
Fitting RBFN

ci Vector quantization
si Estimation of variance around ci
li Linear regression
24
Applications

25
Handwritten digit
Number
26
(No Transcript)
27
Optic nerve stimuliVisual sensations
MLP
Neuro-physiological Process
?
noise
28
Absorbance spectrum of juiceSugar concentration
RBFN
Sugar concentration
Nonlinear modeling
Spectra