Efficient Spatiotemporal Grouping Using the Nystrm Method

1
Efficient Spatiotemporal Grouping Using the
Nyström Method
Charless Fowlkes, U.C. Berkeley Serge Belongie,
U.C. San Diego Jitendra Malik, U.C. Berkeley
2
Grouping With Pairwise Affinities

• Compute the similarities between pairs of points
  in the image
• Find groups of points which have high similarity
  with each other and low similarity with the rest
  of the image.

Sarkar and Boyer (1996), Shi and Malik (1997),
Perona and Freeman (1998), Gdalyahu, Weinshall,
and Werman (2000), .....
3
Normalized Cuts
1. Compute matrix K which contains the pairwise
similarities
2. Find the leading eigenvectors of the
Normalized Laplacian
3. Segment the image using the leading
eigenvectors
Computational Complexity Need to find
eigenvectors of an NxN matrix where N is the
number of pixels. Other spectral partitioning
techniques have same complexity.
4
Spatiotemporal grouping
Adelson and Bergen (1985), Bolles, Baker and
Marimont (1987), Shi and Malik (1998)
5
Computational Problem

• Hard to exploit pairwise clustering techniques
  since 256x384x30 frames entails 1013 pairwise
  similarities.
• How can we overcome this problem?

6
Coping with the computational burden

• Zero out small entries in the affinity matrix
  Shi and Malik (97,98)
  
• Exploit redundancy between rows of the affinity
  matrix
  (this talk)

7
Outline

• Exploiting Redundancy
• The Nyström approximation
• Application to segmenting video sequences

8
Exploiting Redundancy
9
Exploiting Redundancy
Compute Affinity Matrix
10
Exploiting Redundancy
Compute Leading Eigenvectors of Normalized
Laplacian
11
Exploiting Redundancy
Choose sample points
12
Exploiting Redundancy
Compute strip of K
Compute strip of K
13
Exploiting Redundancy
14
Exploiting Redundancy
Interpolate complete eigenvectors
15
Outline

• Exploiting Redundancy
• The Nyström approximation
• Application to segmenting video sequences

16
Approximating eigenfunctions
We would like to find numerical solutions to
Interpolate eigenfunctions using The Nyström
Extension
E. J. Nyström (1929) Baker (1977) Williams and
Seeger (2001)
17
Matrix Completion
Affinity Matrix
Approximate it with
Approximation Error
18
Approximation Error
19
Extrapolating Eigenvectors
Diagonalize approximate K to get complete
eigenvectors
Just matrix notation for the Nyström extension
20
Nyström-NCuts Algorithm

• Choose sample points in image
• Compute similarities for A and B blocks of K
• Compute row sums to estimate degree
• Normalize A and B blocks by degree
• Compute approximate eigenvectors and
  orthogonalize
• Cluster the embedded points using k-means

21
Outline

• Exploiting Redundancy
• The Nyström approximation
• Application to segmenting video sequences

22
Affinity Function for Video
Pairwise affinity function between pixels in a
video sequence makes of three cues

• Similarity in color
• Proximity in time and space
• Common Fate (similarity in optical flow)

We use squared-exponential kernel with diagonal
weighting
23
Performance

• Segmenting a 5 frame video sequence at 120x150
  resolution (100,000 pixels) takes less than 1
  minute in MATLAB on a PC

24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
Conclusions

• Applied the Nystrom approximation to Normalized
  Cuts
• Exploited redundancy in image sequences in order
  to perform efficient spatiotemporal grouping

42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
K-Way Normalized Cuts
Find the leading eigenvectors of Normalized
Laplacian
Embed data and cluster
46
Sampling the image