The Art of Generalized Averaging in Computer Vision and Pattern Recognition

1
The Art of Generalized Averaging in Computer
Vision and Pattern Recognition

• Prof. Dr. Xiaoyi Jiang
• Department of Mathematics and Computer Science
• University of Münster
• Germany
• xjiang_at_math.uni-muenster.de

TexPoint fonts used in EMF. Read the TexPoint
manual before you delete this box. AAAAAAA
2
Content

• Intuitive understanding of averaging
• Generalized averaging
• Case studies
• Consensus clustering
• Generalized median string
• Generalized median graph
• Multiple classifier combination
• Generalized median contour
• Segmentation combination

3
Definition

• An average, or central tendency, of a data set
  refers to
• a measure of the "middle", "expected", or
  "representative"
• value of the data set.
• Average of n numbers x1, x2, , xn
• Arithmetic mean
• Median middle number of the numbers when they
  are ranked in order
• Geometric mean, harmonic mean, etc.

4
Image Smoothing Mean Operator
Mean operator Replace each pixel value by the
arthmetic average of its k x k neighborhood Exam
ple Top distorted image Middle k3 Bottom k5
5
Image Smoothing Median Operator
Median operator Replace each pixel value by the
median of its k x k neighborhood Example Very
effective for eliminating salt and pepper noise
6
Generalized Averaging

• Was is the "average" of n clusterings?
• Was is the "average" of n strings?
• Was is the "average" of n graphs?
• Was is the "average" of n classification
  results?
• Was is the "average" of n contours?
• Was is the "average" of n image segmentations?
• In general
• Was is the "average" of n objects of type X?

7
Generalized Averaging

• Informal definition
• Formal definition Generalized median
• Given a set of n patterns C1, C2, ?, Cn in an
  arbitrary space U and a distance function d(p, q)
  to measure the dissimilarity between any two
  patterns p, q ? U, the generalized median is
  defined by

8
Generalized Averaging
C9
C10
C1
C8
C
C7
C2
C3
C5
C6
C4
9
Generalized Median of Numbers

• Given n numbers x1, x2, , xn

10
Generalized Averaging

• Content to follow Case studies
• Consensus clustering
• Generalized median string
• Generalized median graph
• Multiple classifier combination
• Generalized median contour
• Segmentation combination

11
Consensus Clustering

• Clustering the assignment of objects into groups
  (called clusters) so that
• objects from the same
• cluster are more similar to
• each other than objects
• from different clusters
• Unsupervised learning

12
Consensus Clustering

• Consensus clustering
• A number of different (input) clusterings have
  been obtained for a particular dataset
• It is desired to find a single (consensus)
  clustering which is a better fit in some sense
  than the existing clusterings

13
Consensus Clustering

• A. Fred, A.K.Jain Combining Multiple Clusterings
  Using Evidence Accumulation. IEEE Trans. Pattern
  Anal. Mach Intell. 27(6) 835-850 (2005)
• P. Topchy, A.K. Jain, W.F. Punch Clustering
  Ensembles Models of Consensus and Weak
  Partitions. IEEE Trans. Pattern Anal. Mach
  Intell. 27(12) 1866-1881 (2005)
• H.Ayad, M.S. Kamel Cumulative Voting Consensus
  Method for Partitions with Variable Number of
  Clusters. IEEE Trans. Pattern Anal. Mach Intell.
  30(1) 160-173 (2008)

14
Generalized Median String

• Edit distance Dissimilarity of two strings of
  arbitrary
• lengths (based on edit operations, which model
  the
• distortions between two strings).
• Edit operations
• Substitution of symbol a by b a ? b, a, b ? A, a
  ? b
• Insertion of symbol a e ? a, a ? A
• Deletion of symbol a a ? e, a ? A
• Example x INDUSTRY, yINTEREST

15
Generalized Median String

• Fact 1 There are many sequences of edit
  operations which transform x into y
• Fact 2 There is even a trivial sequence of edit
  operations which transform x into y (delete all
  symbols of x and insert all symbols y)
• Which sequence of edit operations is the best one?

16
Generalized Median String

• Cost for edit operations
• Cost for substitution c(a ? b)
• Cost for insertion c(e ? a)
• Cost for deletion c(a ? e)
• The smaller/larger the cost of an edit operation,
  the
• larger/smaller is the probability that the ideal
  patterns are
• distorted this way
• Costs of sequence of edit operations S e1, ,
  et
• Edit distance (Levenshtein distance) of strings x
  and y
• d(x, y) minc(S) S is sequence of edit
  operations from x to y

17
Generalized Median String

• There is an elegant algorithm for computing the
  edit distance (Levenshtein distance) of strings x
  and y with time and space complexity O(x.y)
• This algorithm can be extended to compute the
  generalized median of strings

18
Generalized Median String

• Application Improvement of OCR (Optical
  Character Recognition) performance by combining
  the results of multiple OCR algorithms

19
Generalized Median Graph

• Edit distance of graphs Edit operation deletion,
  insertion, and substitution for nodes and edges
  costs for edit operations

20
Generalized Median String / Graph

• X. Jiang, H. Bunke, and J. Csirik. Median
  strings A review. In Data Mining in Time Series
  Databases (M. Last, A. Kandel, and H. Bunke,
  Eds.), World Scientific, 173192, 2004
• X. Jiang, A. Münger, and H. Bunke. On median
  graphs Properties, algorithms, and applications.
  IEEE Transactions on Pattern Analysis and Machine
  Intelligence, 23(10) 11441151, 2001

21
Multiple Classifier Combination

• Pattern Recognition (Pattern Classification)
• Given
• an input unknown pattern p
• a set of classes C C1, C2, , CN
• Classification problem is to define a decision
  function which maps p to one of the classes in C
• Which class does the unknown pattern p belong
  to?

22
Multiple Classifier Combination

• Most important / successful applications of PR
• OCR Text understanding in scanned and camera
  images
• Speech understanding
• Biometrics
• Industrial quality control
• Spam email filtering
• Detection of duplicate publications, copied
  programs, etc.

23
Multiple Classifier Combination

• Pattern classification is a difficult task
• There is no best classifier for all test cases
• Different classifiers typically show different
  behaviors
• ? Multiple Classifier Systems (MCS)
• Take over the common strengths and ignore the
  weakness of the involved classifiers
• Consult more experts if you are uncertain in your
  decision

24
Multiple Classifier Combination

• Parallel MCS architecture
• Run classifiers K1, K2, , KL parallelly for
  unknown pattern x pass the classification
  results to the combination module

25
Multiple Classifier Combination

• General combination scheme
• si(Ck) score (e.g. probability) from classifier
  Ki for class Ck

26
Multiple Classifier Combination
27
MCS Screen-Rendered Text Understanding
28
MCS Screen-Rendered Text Understanding

• Copy and Paste Tools
• Indexing images (Google)
• Reading generated annotations from medical images
• GUI Testing
• ...
• Reading SPAM-images
• Captchas (Completely Automated Public Turing
  test)

• Applications
• Translation Tools

29
MCS Screen-Rendered Text Understanding

• Main challenges
• Can be of very low resolution (height of letter x
  can be as low as only four pixels)
• Screen-rendered text is often smoothed (making it
  look better to human eyes)
• It is hardly possible to segment very small
  and/or smoothed words into letters (because they
  touch each other)
• The same letter of identical logical description
  (font, size, etc.) can be rendered differently
  within the same document (depending on its
  position and surrounding)
• Screen text can occur on inhomogeneous background

30
MCS Screen-Rendered Text Understanding
S.Wachenfeld, Fleischer, and X. Jiang. A multiple
classifier approach for the recognition of
screen-rendered text. LNCS, Vol. 4673, 921928,
2007
31
MCS Classification of Diatoms
32
MCS Classification of Diatoms
33
MCS Classification of Diatoms
Evaluation Significant improvement by
combination of classifiers (different decision
trees in this case)
34
Generalized Median Contours

• Definition For a given MxN image a contour C
  p1p2,,pM is a sequence of points drawn from the
  top to the bottom, where pi, i 1,,M, is a
  point in the i-th row. The points pi and pi1, i
  1,,M-1, of two successive rows are
  continuous.
• Solution by dynamic programming
• P. Wattuya and X. Jiang.
• A class of generalized median
• contour problem with
• exact solution. LNCS,
• Vol. 4109, 109117, 2006.

35
Generalized Median Contours

• Application of medical image analysis
  Detecting the layer of intima and adventitia for
  computing the intima media thickness

D. Cheng and X. Jiang. Detection of arterial wall
in sonographic artery images using dual dynamic
programming. IEEE Trans. on Information
Technology in Biomedicine, 12(6) 792799, 2008
Region of interest in Common Carotid Artery
B-mode sonographic image (left) and detected
layer of intima and adventitia (right)
36
Generalized Median Contours

• Closed Contours Eye contour
  detection problem in an application of strabismus
  simulation by replacing the iris

remove an iris
simulate strabismus
detect an eye contour
project back into the image space
polar transformation
contour detection
eye contour in polar space
optimal path
37
Generalized Median Contours
20o and 40o to outside 20o and 40o to inside

• X. Jiang, S. Rothaus, K. Rothaus, and D. Mojon.
  Synthesizing face images by iris replacement
  Strabismus simulation. Proc. of first Int. Conf.
  on Computer Vision Theory and Applications,
  4147, Setuba, Portugal, 2006.

38
Segmentation Combination
39
Segmentation Combination

• Why combining multiple segmentations?
• Exploring parameter space without ground truth
  Segmentation algorithms mostly have some
  parameters and their optimal setting per image is
  not a trivial task

Exactly the same parameter set applied to two
images
40
Segmentation Combination

• Segmenter combination
• There exists no universal segmentation algorithm
  that can successfully segment all images. It is
  not easy to know the optimal algorithm for one
  particular image.
• Instead of looking for the best segmenter which
  is hardly possible on a per-image basis, now we
  look for the best segmenter combiner.

41
Segmentation Combination
L.Grady Random walks for image segmentation.
IEEE-TPAMI, 28 17681783, 2006
seeded (labeled) pixels
unseeded (unlabeled) pixel
(a) A two-region image
(b) User-defined seeds for each region
edge weight similarity between two nodes ,
based on e.g., intensity gradient, color changes
low-weight edge (sharp color gradient)
(c) A 4-connected lattice topology
(d) An undirected weighted graph
42
Segmentation Combination
The algorithm labels an unseeded pixel in
following steps STEP1. Calculate the
probability that a random walker starting at an
unseeded pixel x first reaches a seed with label s
0.03
0.10
0.15
0.85
0.90
0.97
0.97
0.90
0.85
0.15
0.10
0.03
0.03
0.15
0.85
0.97
0.97
0.85
0.15
0.03
0.03
0.10
0.15
0.85
0.90
0.97
0.97
0.90
0.85
0.15
0.10
0.03
Probability that a random walker starting from
each unseeded node first reaches red seed
Probability that a random walker starting from
each unseeded node first reaches blue seed
43
Segmentation Combination
STEP2. Label each pixel with the most probable
seed destination
A segmentation corresponding to region boundary
is obtained by biasing the random walker to avoid
crossing sharp color gradients
44
Segmentation Combination
Original
Seeds indicating four objects
Resulting segmentation
Label 1 probabilities Label
2 probabilities Label 3
probabilities Label 4
probabilities
45
Segmentation Combination
graph G
initial seeds final
result (optimal K)
46
Segmentation Combination
ANMI 0.5507
ANMI 0.6169
ANMI 0.6687
ANMI 0.6716
ANMI 0.6790
ANMI 0.7326
ANMI 0.7671
ANMI 0.8435
ANMI 0.8480
ANMI 0.7778
final result (thresholding)
final result (optimal K)
worst input
median input
best input
ANMI Average normalized mutual information (the
larger, the better)
47
Segmentation Combination
Comparison (per image) Worst / best / average
input combination
48
Segmentation Combination
f(n) Number of images for which the combination
result is worse than the best n input
segmentations
Combination segmentation outperforms all 24 input
segmentations in 78 cases. For 70 (210) of all
300 test images, the goodness of our solution is
beaten by at most 5 input segmentations only.
49
Segmentation Combination
Comparison Average performance for all 300 test
images (for each parameter setting)
50
Segmentation Combination
Dream
The dream must go on!
51
Segmentation Combination

• P.Wattuya, K Rothaus, J.-S. Praßni, and X. Jiang.
  A
• random walker based approach to combining
  multiple
• segmentations. Proc. of ICPR, Tampa, Florida,
  2008
• The same applications for generalized median
  contours
• Exploring parameter space without ground truth
• Contour segmenter combination

52
Conclusion

• The intuitive concept of averaging can be
  extended to arbitrary domains, either in an
  informal or a mathematical way
• We discussed a variety of such extensions in
  computer vision and pattern recognition
• These approaches are useful either for helping
  solve difficult problems like unsupervised
  clustering and supervised classification or for
  inferring a representative model out of a set of
  objects

53
Beautiful Average Face

• Averaged faces are considered beautiful

54
Beautiful Average Face

• Why are the resulting average faces generally
  beautiful?
• One reason might be the fact that by calculating
  average proportions unpleasant asymmetries and
  irregularities become levelled out.
• Moreover, by blending together several faces
  wrinkles and pimples gradually disappear. As a
  consequence, the skin looks younger and perfectly
  smooth. 

55
Essence of Averaging
Considering properties (averaging, etc.) of
vector spaces in arbitrary other spaces Three
cobblers combined equal the master mind -
Chinese proverb -
Thanks!