Title: GAKREM A Clustering Algorithm that Automatically Generates a Number of Clusters
1GAKREM A Clustering Algorithm that Automatically
Generates a Number of Clusters
Cao Dang Nguyen August 2007
2Introduction
- Clustering means grouping similar objects into
groups - A cluster is a set of entities which are alike
and entities from different clusters are not
alike - Clustering is a very important data mining
technique - Applications
- Feature selections
- Image segmentation
- Speed recognition
- Information retrieval
- DNA analysis
- Market studies
- Requirements
- Scalability
- Dealing with different types of attributes
- High dimensionality
- Interpretability and usability
3Introduction
- Clustering algorithms
- Agglomerative algorithms
- Divisive algorithms
- K-means
- Probabilistic algorithms
- Neural network (SOM)
- K-means is widely used because of its simplicity
and computational efficiency - The Expectation-Maximization (EM) is an iterative
statistical algorithm to locate a maximum
likelihood estimate of the mixture parameters - The EM and K-means have several drawbacks
- They are very sensitive to initialization
- They depend on user-inputs number of clusters
4Introduction
5Introduction
6Introduction
- To overcome such drawbacks of the EM and K-means,
we propose a novel algorithm, namely, GAKREM - We use genetic algorithm for estimating
parameters and initializing starting points for
the EM algorithm to avoid convergence toward
local optimal points, - The log likelihood of each configuration of
parameters and a number of clusters resulting
from the EM are used as the fitness value for
each individual (candidate) in population - We approximates the log likelihood of the EM for
each configuration by using logarithmic
regression instead of running the EM until it
converges - We use simple K-means algorithm to initially
assign data points to the clusters and to speed
up convergence of the EM for each candidate - The effectiveness of the GAKREM (Genetic
Algorithm K-means Logarithmic Regression
Expectation Maximization) is evaluated by
comparing its performance with the original EM,
K-means and LCV algorithms on several datasets
7Background
- Suppose that Xx1,x2, xN is independent and
identically distributed with distribution p,
therefore - We assume Z X , Y is a complete data set with
X is observed and Y is unknown data and specify a
joint density function - We need to maximize the log-likelihood function
L(? X) of the observed data, where - To maximize L, a Lagrange multiplier is solved
(1)
(2)
(3)
(4)
8Background
- The EM algorithm is a simpler algorithm for
obtaining ML estimate - E-Step to find the expected value of the
complete-data log-likelihood log p(Z ? )
denoted by - M-Step to maximize the expected value computed
in E-step
(5)
(6)
9Background
- In the case of the mixture-density parameter
estimation problem, we assume -
-
-
- where ? ?1, ?K,?1?K such that ??h1 and
?h ?h , ?h and ph(x ?h) is a - density Gaussian distribution function
- The log-likelihood function of the incomplete
data is -
(7)
(8)
10Background
- The unobserved data Y is a set of N labels Yy1,
yN - where yi ,, is a binary vector k
dimensions - i.e. if cluster h computes data point xi then
1 and 0, j?h - The log-likelihood function of the complete data
is -
-
- Given ? (t)?1(t), ?K(t),?1(t)?K(t), we
calculate the expectation of unobserved data Y
(9)
(10)
11Background
-
- Then, we maximize the log-likelihood function of
the complete data
(11)
12GAKREM Algorithm
- Phase I Compute the initial guess of parameters.
- We use GA and simple K-means to guess the
parameter?(0) - Each chromosome j in population is configured as
a binary vector dimension N gj,i, i1..N,
where gi,i1 if data point xi is set up as a
centroid point of a cluster - gi,i1 is the number K of components
- Examples suppose the size of dataset N 10, we
have - Chromosome 1 0001000010 has 2 clusters with
centroid points are 4th, 9th - Chromosome 2 0010001001 has 3 clusters with
centroid points are 3rd, 7th and 10th - To optimize the use of memory and time, we encode
only the positions 1-loci in chromosomes - 4,9 , 3, 7, 10
13(No Transcript)
14- Phase II Evaluate the chromosomes fitness
- We perform a partial EM in r iterations for each
of above candidate and calculate the expected log
likelihood function by using logarithmic
regression E(C)a log(t) b, where -
-
- where Lt is the log-likelihood function of
iteration t, experimentally, we set r5,
t1,2,3,4,5 - Log-likelihood is of no direct use because the
log-likelihood of the data can always be
increased by increasing the number of cluster k - Based on Occams razor entities should not be
multiplied beyond necessity, we suggest the
fitness value of each candidate as - fit (C )E(C) - log(k)
(12)
(13)
15- Phase III Evolutionary generation for finding
optimal fitness value - Step 1 Initialize population and evaluate fitness
value of chromosomes
16- Step 2 Generate new population
17Results
- We have conducted extensive experiments using
GAKREM on two kinds of data manually-generated
datasets and automatically-generated datasets as
used in Xu and Jordan (1996) and Jain and Dubes
(1988) - The probability of mutation was set
experimentally to 0.15 for all experiments - For comparison in determining the best number of
clusters in each generated data set, we have
selected the likelihood cross-validation
technique (Smyth, 1988) implemented in Weka
package - k ? 1
- Repeat
- k ? k1
- Divide dataset D into v-folds (v10)
- For i? 1 to v
- train the model in the ith-fold and test the
model in D \ ith (with k) - lavg ? the average log-likelihood function
- Until lavg not increasing or kkmax
18Results
- For comparison in determining the optimal
log-likelihood function in each generated data
set, we implemented EM and K-means - To test the robustness of the algorithms, we
repeated the experiments 100 times for each
dataset
19 Experiments on manually-generated data
- The result of GAKREM on a 2-cluster mixture
derived from the Old Faithful dataset in Bishop
(1995) - This dataset has two dimensions duration of an
eruption of the geyser in minutes and
interruption time - GAKREM precisely recognizes a 2-cluster mixture,
of course, without the pre-defined number of
clusters
20 Experiments on manually-generated data
- The behavior of GAKREM on the Old Faithful
dataset. It performs an heuristic search on the
global space. At the 1728th generation in this
trial, the optimal fitness value is stable at
-7.5568.
21 Experiments on manually-generated data
GAKREM
LCV
22 Experiments on manually-generated data
Average maximum log-likelihood functions of
GAKREM, EM and K-means testing on 2-, 3-, 4-,
5-,6-, 7-, 8- and 9-cluster datasets,
respectively.
23 Experiments on manually-generated data
K-means
EM
24 Experiments on automatically-generated data
- Accuracy of GAKREM is 97
- Accuracy of LCV is 31
25 Experiments on automatically-generated data
26 Experiments on automatically-generated data
K-means
EM
27(No Transcript)
28Conclusions Future Work
- We presented a powerful new algorithm, called
GAKREM, that combines the best characteristics of
K-means and EM algorithms but avoids their
weaknesses. - We have tested the algorithm extensively, on both
manually and automatically generated datasets - We plan to use GAKREM to discover new pathways in
chromosome 21 proteins by using heterogeneous
data combining micro array data and interaction
data - Demonstration is available at http//isl.cudenver.
edu/GAKREM