DENCLUE 2'0: Fast Clustering based on Kernel Density Estimation

1
DENCLUE 2.0 Fast Clustering based on Kernel
Density Estimation

• Alexander Hinneburg
• Martin-Luther-University Halle-Wittenberg,
  Germany
• Hans-Henning Gabriel
• 101tec GmbH, Halle, Germany

2
Overview

• Density-based clustering and DENCLUE 1.0
• Hill climbing as EM-algorithm
• Identification of local maxima
• Applications of general EM-acceleration
• Experiments

3
Density-Based Clustering

• Assumption
• clusters are regions of high density in the data
  space ,
• How to estimate density?
• parametric models
• mixture models
• non-parametric models
• histogram
• kernel density estimation

4
Kernel Density Estimation

• Idea
• influence of a data point is modeled by a kernel
• density is the normalized sum of all kernels
• smoothing parameter h

Gaussian Kernel
Density Estimate
5
DENCLUE 1.0 Framework

• Clusters are defined by local maxima of the
  density estimate
• find all maxima by hill climbing
• Problem
• const. step size

Gradient
Hill Climbing
const. step size
6
Problem of const. Step Size

• Not efficient
• many unnecessary small steps
• Not effective
• does not converge to a local maximumjust comes
  close
• Example

7
New Hill Climbing Approach

• General approach
• differentiate density estimate and set to zero
• no solution, but can be used for iteration

8
New DENCLUE 2.0 Hill Climbing

• Efficient
• automatically adjusted step size at no extra
  costs
• Effective
• converges to local maximum (proof follows)
• Example

9
Proof of Convergence

• Cast the problem of maximizing kernel denstiy
  as maximizing the likelihood of a mixture
  model

• Introduce hidden variable

10
Proof of Convergence

• Complete likelihood is maximized by EM-Algorithm
• this also maximizes the original likelihood,
  which is the kernel density estimate
• When starting the EM with we do the
  hill climbing for

E-Step
M-Step
11
Identification of local Maxima

• EM-Algorithm iterates until
• reached end point
• sum of k last step sizes
• Assumption
• true local maximum is in a ball of around
• Points with end points
  closerbelong to the same maximum M
• In case of non-unique assignmentdo a few extra
  EM iterations

12
Acceleration

• Sparse EM
• update only the p points with largest posterior
• saves 1-p of kernel computations after first
  iteration
• Data Reduction
• use only p of the data as representative points
• random sampling
• kMeans

13
Experiments

• Comparison of DENCLUE 1.0 (FS) vs. 2.0 (SSA)

• 16-dim. artificial data
• both methods are tuned to find the correct
  clustering

14
Experiments

• Comparison of acceleration methods

15
Experiments

• Clustering quality (normalized mutual
  information, NMI) vs. sample size (RS)

16
Experiments

• Cluster Quality (NMI) of DENCLUE 2.0 (SSA) and
  acceleration methods and k-Means on real data

sample sizes 0.8, 0.4, 0.2
17
Conclusion

• New hill climbing for DENCLUE
• Automatic step size adjustment
• Convergence proof by reduction to EM
• Allows the application of general EM
  accelerations
• Future work
• automatic setting of smoothing parameter h(so
  far tuned manually)

18
Thank you for your attention!