Segmentation and Boundary Detection Using Multiscale Intensity Measurements

1
Segmentation and Boundary Detection Using
Multiscale Intensity Measurements

• Eitan Sharon, Meirav Galun, Ronen Basri, Achi
  Brandt

Dept. of Computer Science and Applied
Mathematics The Weizmann Institute of Science
2
Image Segmentation
3
Local Uncertainty
4
Global Certainty
5
Local Uncertainty
6
Global Certainty
7
Coarse Measurements for Texture
8
A Chicken and Egg Problem
Problem Coarse measurements mix neighboring
statistics
Solution support of measurements is determined
as the segmentation process proceeds
9
Segmentation by Weighted Aggregation

• Normalized-cuts measure in graphs
• Complete hierarchy in linear time
• Use multiscale measures of
• intensity, texture, shape, and boundary
  integrity

10
Segmentation by Weighted Aggregation

• Normalized-cuts measure in graphs
• Complete hierarchy in linear time
• Use multiscale measures of
• intensity, texture, shape, and boundary
  integrity

11
Segmentation by Weighted Aggregation

• Normalized-cuts measure in graphs
• Complete hierarchy in linear time
• Use multiscale measures of
• intensity, texture, shape and boundary
  integrity

12
The Pixel Graph
Couplings Reflect intensity similarity

Low contrast strong coupling
High contrast weak coupling
13
Hierarchical Graph
14
Hierarchyin SWA
15
Normalized-Cut Measure
16
Normalized-Cut Measure
17
Normalized-Cut Measure
18
Normalized-Cut Measure
Minimize
19
Normalized-Cut Measure
Minimize
20
Normalized-Cut Measure
Low-energy cut
Minimize
21
Recursive Coarsening
22
Recursive Coarsening
Representative subset
23
Recursive Coarsening
For a salient segment
, sparse interpolation matrix
24
Weighted Aggregation
aggregate
aggregate
25
Segment Detection
26
SWA
Detects the salient segments
Hierarchical structure
Linear in of points (a few dozen operations per
point)
27
Coarse-Scale Measurements

• Average intensities of aggregates
• Multiscale intensity-variances of aggregates
• Multiscale shape-moments of aggregates
• Boundary alignment between aggregates

28
Adaptive vs. Rigid Measurements
Original
Averaging
Geometric
Our algorithm - SWA
29
Adaptive vs. Rigid Measurements
Original
Interpolation
Geometric
Our algorithm - SWA
30
Recursive Measurements Intensity
intensity of pixel i
aggregate
average intensity of aggregate
31
Use Averages to Modify the Graph
32
Use Averages to Modify the Graph
33
Texture Examples
34
Isotropic and Oriented Filters
A brief tutorial
Textons by K-Means Malik et al IJCV2001
35
Isotropic Texture in SWA
Intensity Variance
Isotropic Texture of aggregate average of
variances in all scales
36
Isotropic Texture in SWA
Intensity Variance
Isotropic Texture of aggregate average of
variances in all scales
37
Isotropic Texture in SWA
Intensity Variance
Isotropic Texture of aggregate average of
variances in all scales
38
Oriented Texture in SWA
with Meirav Galun
Shape Moments

• center of mass
• width
• length
• orientation

Oriented Texture of aggregate orientation,
width and length in all scales
39
Gestalt Perceptual Grouping
Shashua and Ullman ICCV 1988
A brief Tutorial

• Group curves by
• Proximity
• Co-linearity

Sharon, Brandt, Basri PAMI 2000
40
Boundary Integrity in SWA
41
Sharpen the Aggregates

• Top-down Sharpening
• Expand core
• Sharpen boundaries

42
Hierarchyin SWA
43
Experiments

• Our SWA algorithm (CVPR00 CVPR01)
• run-time 5-10 seconds.
• Normalized cuts (Shi and Malik, PAMI00 Malik
  et al., IJCV01)
• run-time about 10-15 minutes.
• Software courtesy of Doron Tal, UC Berkeley.

44
Isotropic Texture - Horse I
Our Algorithm (SWA)
Normalized Cuts
45
Isotropic Texture - Horse II
Our Algorithm (SWA)
Normalized Cuts
46
Isotropic Texture - Tiger
Our Algorithm (SWA)
Normalized Cuts
47
Isotropic Texture - Butterfly
Our Algorithm (SWA)
Normalized Cuts
48
Isotropic Texture - Leopard
Our Algorithm (SWA)
49
Isotropic Texture - Dalmatian Dog
Our Algorithm (SWA)
50
Isotropic Texture - Squirrel
Our Algorithm (SWA)
Normalized Cuts
51
Full Texture - Squirrel
Our Algorithm (SWA)
Normalized Cuts
with Meirav Galun
52
Full Texture - Composition
Our Algorithm (SWA)
with Meirav Galun
53
Full Texture Lion Cub
Our Algorithm (SWA)
with Meirav Galun
54
Full Texture Polar Bear
Our Algorithm (SWA)
with Meirav Galun
55
Full Texture Penguin
Our Algorithm (SWA)
with Meirav Galun
56
Full Texture Leopard
Our Algorithm (SWA)
with Meirav Galun
57
Full Texture - Zebra
Our Algorithm (SWA)
with Meirav Galun
58
Segmentation by Weighted Aggregation

• Efficient approximation to Ncut-like measures
• Recursive computation of multiscale measurements
• Novel adaptive pyramid representing the image

59
Matching Experiment
(Chen Brestel)
60
Matching Experiment
(Chen Brestel)
61
Matching Experiment
(Chen Brestel)
62
Experiments Clustering Silhouettes
Data from Gdalyahu, Weinshall, Werman CVPR 99
(Also Domany, Blatt, Gdalyahu, Weinshall)
63
Descending order of prominence objects are
fully categorized
most prominent
less prominent
64
Bottom-Up and Top-Down