X-means: Extending K-means with Efficient Estimation of the Number of Clusters

1
X-means Extending K-means with Efficient
Estimation of the Number of Clusters

• Dan Phelleg, Andrew Moore
• Carnegie Mellon University
• Published ICML 2000
• Presentation by
• Payam Refaeilzadeh

2
Problems with K-means

• Need to know K
• Searching for K is expensive
• Even K-means with fixed-K scales poorly
• Need to calculate the distance from each point to
  each centroid to find new cluster assignments

3
Remedies

• Forward search for the appropriate value of k in
  a given range
• Recursively split each cluster and use BIC score
  to decide if we should keep each split
• Use kd-trees to accelerate individual rounds of
  K-means

4
Splitting

• Use local BIC score to decide on keeping a split
• Use global BIC score to decide which K to output
  at the end

5
BIC (Bayesian Information Criterion)

• Adjusted Log-likelihood of the model.
• The likelihood that the data is explained by
  the clusters according to the spherical-Gaussian
  assumption of k-means

6
Kd-trees

• Points to be clustered are put into a binary
  hierarchical structure
• Each node represents a subset of points and
  stores
• The minimal hyper-rectangle enclosing all points
  in the subset
• The vector-sum of all the points in the subset
• The number of points in the subset

7
Using kd-trees

• For each centroid store a counter containing the
  vector sum of all the points belonging to it and
  the number of points
• Update the above by scanning the kd-tree only
  once
• Start with the root node and all centroids
• As you walk down the tree centroids start to get
  black-listed (when the points in that node could
  not possibly belong to a centroid)
• When only one centroid remains, the counter for
  that centroid can be updated using the statistics
  stored in the node
• At the end of the scan we have enough info to
  recalculate the centroid coordinates

8
Results