Dynamics of Learning VQ and Neural Gas

1
Dynamics of Learning VQand Neural Gas

• Aree Witoelar, Michael Biehl
• Mathematics and Computing Science
• University of Groningen, Netherlands
• in collaboration with Barbara Hammer (Clausthal),
  Anarta Ghosh (Groningen)

2
Outline

• Vector Quantization (VQ)
• Analysis of (L)VQ Dynamics
• Results
• Multiple prototypes
• Neural Gas
• Learning Vector Quantization
• Summary

3
Vector Quantization
4
Vector Quantization

• Objective
• representation of (many) data with (few)
  prototype vectors

Find optimal set W for lowest quantization error
distance to nearest prototype
data
5
Example Winner Takes All (WTA)

• prototypes at areas with high density data
• (stochastic) on-line gradient descent with
  respect to a cost function

6
Problems

• Winner Takes All

7
Dynamics of VQ Analysis
8
Modeltwo Gaussian clusters of high dimensional
data with class s 1,-1
Random vectors ? ? RN according to
classes 1, -1
prior prob. p, p- p p- 1
center vectors B, B- ? RN
variance ?, ?-
separation l
only separable in 2 dimensions ? simple model,
but not trivial
9
Online learning
sequence of independent random data
acc. to
? ? RN
ws ? RN
update of prototype vector
learning rate, step size
strength, direction of update etc.
prototypeclass
data class
fs describes the algorithm used
winner
10
1. Define few characteristic quantities of the
system

projections tocluster centers
length and overlapof prototypes
11
average over examples
In the thermodynamic limit N?8 ...
12
3. Derive ordinary differential equations

• 4. Solve for Rss(t), Qst(t)
• dynamics/asymptotic behavior (t ? 8)
• quantization/generalization error
• sensitivity to initial conditions, learning
  rates, structure of data

13
Results
14
Vector Quantization2 prototypes
characteristic quantities
Numerical integration of the ODEs (ws(0)0
p0.6, l1.0, ?1.5, ?- 1.0, ?0.01)
quantization error
15
2 prototypes
Projections of prototypes on the B,B- plane at
t50
Asymptotic positions (t?8) prototypes are on the
B,B- plane
16
Neural Gas a winner take most algorithm3
prototypes
?(t) decreased over time ?(t)?0 identical to WTA
?i2 ?f10-2
17
Sensitivity to initialization
Neural Gas
WTA
RS-
RS-
RS
RS
at t50
at t50
?HVQ0
E(W)
plateau
t
18
Summary of VQ and NG

• WTA
• (eventually) reaches minimum E(W)
• depends on initialization possible large
  learning time
• Neural Gas
• more robust w.r.t. initialization

19
Learning Vector Quantization
20
Learning Vector Quantization (LVQ)

• Objective
• classification of data using prototype vectors

21
LVQ1
update winner towards/ away from data
two prototypes
22
Generalization error
class
misclassified data
eg
t
p0.6, p- 0.4 ?1.5, ?-1.0
23
Optimal decision boundary
(hyper)plane where
equal variance (??-) linear decision boundary
unequal variance ?gt?-
K2
24
LVQ1, three prototypes
? gt?- (?0.81, ?- 0.25)
?1,1,-1
?1,1,-1
eg
eg
p
p

• LVQ1 K3 better
• Optimal K3 better

• LVQ1 K3 worse
• Optimal K3 equal to K2

• more prototypes not always better for LVQ1

• best more prototypes on the class with the
  larger variance

25
Summary

• dynamics of (Learning) Vector Quantization for
  high dimensional data
• Neural Gas
• more robust w.r.t. initialization than WTA
• LVQ1
• more prototypes not always better
• variances matter

Outlook

• study other algorithms e.g. LVQ/-, LFM, RSLVQ
• more complex models
• multi-prototype, multi-class problems

26
Questions
?
27
(No Transcript)
28
example LVQ1
29
Self averaging
Fluctuations decreases with larger degree of
freedom N
At N?8, fluctuations vanish (variance becomes
zero)
Monte Carlo simulations over 100 independent runs
30
LVQ /-
update correct and incorrect winners
ds min dk with ?s sµ
dt min dk with ?t ?sµ
31
Comparison LVQ1 and LVQ 2.1
LVQ2.1 ? performance depends on initial conditions
32
LVQ1, three prototypes
l1.0, ?0.81, ?- 0.25
?1,1,-1
? gt ?-
? lt ?-
eg
eg
p
p
3 prototypes better
3 prototypes worse