Agglomerative clustering (AC)

1
Agglomerative clustering (AC)
Clustering algorithms Part 2c
Pasi Fränti 25.3.2014 Speech Image Processing
Unit School of Computing University of Eastern
Finland Joensuu, FINLAND
2
Agglomerative clusteringCategorization by cost
function

• Single link
• Minimize distance of nearest vectors
• Complete link
• Minimize distance of two furthest vectors
• Wards method
• Minimize mean square error
• In Vector Quantization, known as Pairwise
  Nearest Neighbor (PNN) method

We focus on this
3
Pseudo code
4
Pseudo code
PNN(X, M) ? C, P FOR i?1 TO N DO pi?i
ci?xi REPEAT a,b ? FindSmallestMergeCost() M
ergeClusters(a,b) m?m-1 UNTIL mM
O(N)
O(N2)
N times
T(N) O(N3)
5
Wards methodWard 1963 Journal of American
Statistical Association
Merge cost
Local optimization strategy

• Nearest neighbor search
• Find the cluster pair to be merged
• Update of NN pointers

6
Example of distance calculations
7
Example of the overall process
M5000
M50
M5000 M4999 M4998 . . . M50 . . M16 M15
M16
M15
8
Detailed example of the process
9
Example - 25 Clusters
MSE 1.01109
10
Example - 24 Clusters
MSE 1.03109
11
Example - 23 Clusters
MSE 1.06109
12
Example - 22 Clusters
MSE 1.09109
13
Example - 21 Clusters
MSE 1.12109
14
Example - 20 Clusters
MSE 1.16109
15
Example - 19 Clusters
MSE 1.19109
16
Example - 18 Clusters
MSE 1.23109
17
Example - 17 Clusters
MSE 1.26109
18
Example - 16 Clusters
MSE 1.30109
19
Example - 15 Clusters
MSE 1.34109
20
Storing distance matrix

• Maintain the distance matrix and update rows for
  the changed cluster only!
• Number of distance calculations reduces from
  O(N2) to O(N) for each step.
• Search of the minimum pair still requires O(N2)
  time ? still O(N3) in total.
• It also requires O(N2) memory.

21
Heap structure for fast searchKurita 1991
Pattern Recognition

• Search reduces O(N2) ? O(logN).
• In total O(N2 logN)

22
Store nearest neighbor (NN) pointersFränti et
al., 2000 IEEE Trans. Image Processing
Time complexity reduces to O(N 3) ? ? (?N 2)
23
Pseudo code
PNN(X, M) ? C, P FOR i?1 TO N DO pi?i
ci?xi FOR i?1 TO N DO NNi?
FindNearestCluster(i) REPEAT a ?
SmallestMergeCost(NN) b ? NNi MergeClusters(C,
P,NN,a,b,) UpdatePointers(C,NN) UNTIL mM
O(N)
O(N2)
O(N)
O(?N)
http//cs.uef.fi/pages/franti/research/pnn.txt
24
Example with NN pointersVirmajoki 2004
Pairwise Nearest Neighbor Method Revisited
25
ExampleStep 1
26
ExampleStep 2
27
ExampleStep 3
28
ExampleStep 4
29
ExampleFinal
30
Time complexities of the variants
31
Number of neighbors (t)
32
Processing time comparison
With NN pointers
33
AlgorithmLazy-PNN
T. Kaukoranta, P. Fränti and O. Nevalainen,
"Vector quantization by lazy pairwise nearest
neighbor method", Optical Engineering, 38 (11),
1862-1868, November 1999
34
Monotony property of merge cost Kaukoranta et
al., Optical Engineering, 1999
Merge costs values are monotonically increasing
d(Sa, Sb) ? d(Sa, Sc) ? d(Sb, Sc) ? d(Sa, Sc) ?
d(Sab, Sc)
35
Lazy variant of the PNN

• Store merge costs in heap.
• Update merge cost value only when it appears at
  top of the heap.
• Processing time reduces about 35.

Method Ref. Time complexity Additional data structure Space compl.
Trivial PNN 10 O(dN3) - O(N)
Distance matrix 6 O(dN2 N3) Distance matrix O(N2)
Kuritas method 5 O(dN2 N2logN) Dist. matrix heap O(N2)
?-PNN 1 O(d?N2) NN-table O(N)
Lazy-PNN 4 O(d?N2) NN-table O(N)
36
Combining PNN and K-means
K-means
37
AlgorithmIterative shrinking
P. Fränti and O. Virmajoki Iterative shrinking
method for clustering problemsPattern
Recognition, 39 (5), 761-765, May 2006.
38
Agglomerative clustering based on merging
39
Agglomeration based on cluster removalFränti
and Virmajoki, Pattern Recognition, 2006
40
Merge versus removal
41
Pseudo code of iterative shrinking (IS)
42
Cluster removal in practice
Find secondary cluster
Calculate removal cost for every vector
43
Partition updates
44
Complexity analysis

• Number of vectors per cluster

If we iterate until M1
Adding the processing time per vector
45
AlgorithmPNN with kNN-graph
P. Fränti, O. Virmajoki and V. Hautamäki, "Fast
agglomerative clustering using a k-nearest
neighbor graph". IEEE Trans. on Pattern Analysis
and Machine Intelligence, 28 (11), 1875-1881,
November 2006
46
Agglomerative clustering with kNN graph
47
Example of 2NN graph
48
Example of 4NN graph
49
Graph using double linked lists
50
Merging a and b
51
Effect on calculationsnumber of steps
STAGE Theoretical Theoretical Theoretical Observed Observed Observed
STAGE ?-PNN Single link Double link ?-PNN Single link Double link
Find pair N 1 1 8 357 3 3
Merge N k2 logN k2 ?k logN 8 367 200 305
Remove last N k logN LogN 8 349 102 45
Find neighbors N kN ?k 8 357 41 769 204
Update costs N (1?) ?  ?/k?logN ?  ?/k?logN 48 538 198 187
TOTAL O(?N2) O(kN2) O(?N?logN) 81 970 42 274 746
52
Processing time as function of k(number of
neighbors in graph)
53
Time distortion comparison
?-PNN (229 s) Trivial-PNN (gt9999 s)
Graph-PNN (1)
MSE 5.36
Graph-PNN (2)

1. Graph created by MSP
2. Graph created by D-n-C

54
Conclusions

• Simple to implement, good clustering quality
• Straightforward algorithm slow O(N3)
• Fast exact (yet simple) algorithm O(tN2)
• Beyond this possible
• O(tNlogN) complexity
• Complicated graph data structure
• Compromizes the exactness of the merge

55
Literature

1. P. Fränti, T. Kaukoranta, D.-F. Shen and
   K.-S. Chang, "Fast and memory efficient
   implementation of the exact PNN", IEEE Trans. on
   Image Processing, 9 (5), 773-777, May 2000.
2. P. Fränti, O. Virmajoki and V. Hautamäki, "Fast
   agglomerative clustering using a k-nearest
   neighbor graph". IEEE Trans. on Pattern Analysis
   and Machine Intelligence, 28 (11), 1875-1881,
   November 2006.
3. P. Fränti and O. Virmajoki, "Iterative shrinking
   method for clustering problems", Pattern
   Recognition, 39 (5), 761-765, May 2006.
4. T. Kaukoranta, P. Fränti and O. Nevalainen,
   "Vector quantization by lazy pairwise nearest
   neighbor method", Optical Engineering, 38 (11),
   1862-1868, November 1999.
5. T. Kurita, "An efficient agglomerative clustering
   algorithm using a heap", Pattern Recognition 24
   (3) (1991) 205-209.

56
Literature

1. J. Shanbehzadeh and P.O. Ogunbona, "On the
   computational complexity of the LBG and PNN
   algorithms". IEEE Transactions on Image
   Processing 6 (4), 614-616, April 1997.
2. O. Virmajoki, P. Fränti and T. Kaukoranta,
   "Practical methods for speeding-up the pairwise
   nearest neighbor method ", Optical Engineering,
   40 (11), 2495-2504, November 2001.
3. O. Virmajoki and P. Fränti, "Fast pairwise
   nearest neighbor based algorithm for multilevel
   thresholding", Journal of Electronic Imaging, 12
   (4), 648-659, October 2003.
4. O. Virmajoki, Pairwise Nearest Neighbor Method
   Revisited, PhD thesis, Computer Science,
   University of Joensuu, 2004.
5. J.H. Ward, Hierarchical grouping to optimize an
   objective function, J. Amer. Statist.Assoc. 58
   (1963) 236-244.