Pattern Recognition: Statistical and Neural

1
Nanjing University of Science Technology
Pattern RecognitionStatistical and Neural
Lonnie C. Ludeman Lecture 29 Nov 11, 2005
2
Lecture 29 Topics
1. Review Fuzzy Sets and Fuzzy Partitions 2.
Fuzzy C- Means Clustering Algorithm
Preliminaries 3. Fuzzy C-Means Clustering
Algorithm Details 4. Fuzzy C-Means Clustering
Algorithm Example 5. Comments about Fuzzy
Clustering
3
Example of a Fuzzy Set Define a Fuzzy set A by
the following membership function
Review
Example
Or equivalently
4
Review
S
Function defined on S
5
Membership Function for F1
Review
Domain Pattern Vectors
EXAMPLE
Membership Function for F2
Domain Pattern Vectors
Membership Functions for Fuzzy Clusters
6
Review
A Fuzzy Partition F, of a set S, is defined by
its membership functions for the fuzzy sets Fk
k 1, 2, ... , K
)
Fuzzy Partition
7
where
Review
Each value bounded by 0 and 1 Sum of each columns
values 1 Sum of each row is less than n
8
Conversion of Fuzzy Clustering into Crisp
Clustering
Fuzzy Clusters
Crisp Clustering
All membership values are equal to 1 or 0 Only
one 1 in each column where membership value for a
Fuzzy cluster is a maximum
9
Example Given Fuzzy clusters below convert to
Crisp Clusters
Fuzzy Clusters
Cl1 0.6 0.7 0.3 0.1 0.4 0.2 0.1
Cl2 0.1 0.1 0.3 0.5 0.1 0.7 0.1
Cl3 0.3 0.2 0.4 0.4 0.5 0.1 0.8
Crisp Clusters from Fuzzy Clusters
Cl1 1 1 0 0 0 0
0 Cl2 0 0 0 1 0
1 0 Cl3 0 0 1 0 1
0 1
10
Cl1 1 1 0 0 0 0
0 Cl2 0 0 0 1 0
1 0 Cl3 0 0 1 0 1
0 1
Answer Crisp Clusters
11
Fuzzy C-Means Clustering Preliminary
Given a set S composed of pattern vectors which
we wish to cluster
S x1, x2, ... , xN
Define C Cluster Membership Functions
...
...
12
Define C Cluster Centroids as follows
Let Vi be the Cluster Centroid for Fuzzy Cluster
Cli , i 1, 2, , C
Define a Performance Objective Jm as
where
13
Definitions
A is a symmetric positive definite matrix Ns is
total number of pattern vectors
m Fuzziness Index (m gt1 ) Higher numbers
being more fuzzy
The Fuzzy C-Means Algorithm minimizes Jm by
selecting Vi and i , i 1, 2, , C by an
alternating iterative procedure as described in
the algorithms details
14
Fuzzy C-Means Clustering Algorithm (a) Flow
Diagram
Yes
No
15
Fuzzy C-Means Clustering Algorithm
(b) Details of Steps in the Algorithm
16
Step 1 Initialization
Select initial membership functions such that
This is equivalent to specifying fuzzy clusters
F1, F2, , FC
17
One Method to accomplish this selection is to
chose rki randomly from the open interval (0, 1)
and then Normalize
18
Step 2 Computation of Fuzzy Centroids
Compute the Fuzzy Centroids as
19
Step 3 Compute New Fuzzy Membership Functions
Using the Vi, i 1, 2, , C from step 2
compute i(k)
20
Alternative method for computing the Membership
Functions
where
21
Step 4 Check for Convergence
If membership functions do not change convergence
has occurred If algorithm converges then the i
represent the fuzzy clusters and we Stop If
Convergence has not occurred and the number of
iterations is less than some preassigned maximum
value (MAXIT) then return to step 2 If otherwise
then Stop with no solution.
22
Properties of Fuzzy C-Means Algorithm
Usually convergences Answer not unique as
depends upon initial conditions. Can converge to
a local minimum Can be used to produce hard
clusters
23
Example Application of Fuzzy Clustering
Algorithm
Given the following set of data vectors

1. Perform a Fuzzy Clustering of the data using
   Fuzzy C-Means Algorithm to obtain two fuzzy
   clusters. Try several initial conditions. Is the
   result unique? (use MAXIT1000)
2. Using the results of (a) give a crisp clustering
   of the data.
3. Repeat (a) and (b) for three Fuzzy Clusters.

24
Plot of Data for Fuzzy clustering example
25
(a) Solution for two clusters
Fuzzy cluster membership functions randomly
selected
Calculation of Fuzzy Centroids
26
Calculation of New Membership Functions
Calculation of Performance
Not the same as preceding iteration Membership
function (No convergence)
Number of iterations not greater than 1000
therefore the iterations continue.
27
Results Converge at Iteration 17
Cluster membership Functions
Performance Measure
Cluster Centroids
28
(a) Final Cluster membership Functions
Cl1
Cl2
29
(b) Solution Crisp Clustering
Fuzzy Membership functions
Cl1
Cl2
Crisp Membership functions
Set Assignment
Cl1 x1, x2, x3 Cl2 x4 , x5
30
(c) Solution for three fuzzy clusters
Applying the Fuzzy Clustering Algorithm
convergence was obtained in ?? iterations as
Jm 0.9337
Final Cluster membership functions
Cl1
Cl2
Cl3
31
(c) Solution Crisp Clustering
Membership functions
F1 1, 1, 0, 0, 0 F2 0, 0, 0, 1, 1 F3
0, 0, 1, 0, 0
Set Assignment
Cl1 x1 , x2 Cl2 x4 , x5 Cl3
x3
Crisp Clusters
32
Comments
Fuzzy Clustering Can be used to Produce Hard
Clustering
The larger the value of m the fuzzier the clusters
The Fuzzy algorithm is relatively stable and
usually converges in a reasonable number of
iterations
The Fuzzy algorithm is relatively insensitve to
initial conditions
33
Of the two different fuzzy clusterings given
below, which clustering is the Fuzzier ???
Cl1 0.52 0.51 0.04 0.47 0.46 Cl2
0.48 0.49 0.96 0.5 0.53
1
or
Cl1 0.89 0.85 0.04 0.26 0.15 Cl2
0.11 0.15 0.96 0.74 0.85
2
34
ANSWER 1 is the fuzzier of the two different
clusterings
Cl1 0.52 0.51 0.04 0.47 0.46 Cl2
0.48 0.49 0.96 0.53 0.53
1
Cl1 0.89 0.85 0.04 0.26 0.15 Cl2
0.11 0.15 0.96 0.74 0.85
2
35
Why is 1 the Fuzzier of the two ???
36
Why is 1 the Fuzzier of the two ???
ANSWER Because the cluster membership
functions contain many entries close to 0.5 ( for
the two class case) as opposed to values close to
0 and 1.

For the M class case values close to 1/M would
indicate most fuzziness..

37
Lecture 29 Topics
1. Reviewed Fuzzy Sets and Fuzzy Partitions 2.
Presented Fuzzy C- Means Clustering Algorithm
Preliminaries 3. Gave Fuzzy C-Means Clustering
Algorithm Details 4. Showed an Example of the
Fuzzy C-Means Clustering Algorithm 5. Made a few
comments about Fuzzy Clustering in General.
38
End of Lecture 29