Interest Point Descriptors

1
Interest Point Descriptors

• CS491Y/691Y Topics in Computer Vision
• Dr. George Bebis

2
Interest Point Descriptors
Descriptors
3
Scale Invariant Feature Transform (SIFT)

1. Take a 16 x16 window around interest point (i.e.,
   at the scale detected).
2. Divide into a 4x4 grid of cells.
3. Compute histogram of image gradient directions in
   each cell (8 bins each).

16 histograms x 8 orientations
128 features
4
Properties of SIFT

• Highly distinctive
• A single feature can be correctly matched with
  high probability against a large database of
  features from many images.
• Scale and rotation invariant.
• Partially invariant to 3D camera viewpoint
• Can tolerate up to about 60 degree out of plane
  rotation
• Partially invariant to changes in illumination
• Can be computed fast and efficiently.

5
SIFT Computation Steps

• (1) Scale-space extrema detection
• Extract scale and rotation invariant interest
  points (i.e., keypoints).
• (2) Keypoint localization
• Determine location and scale for each interest
  point.
• Eliminate weak keypoints
• (3) Orientation assignment
• Assign one or more orientations to each keypoint.
• (4) Keypoint descriptor
• Use local image gradients at the selected scale.

D. Lowe, Distinctive Image Features from
Scale-Invariant Keypoints, International Journal
of Computer Vision, 60(2)91-110, 2004.
6
Scale-space Extrema Detection

• Harris-Laplace
• Find local maxima of
• Harris detector in space
• LoG in scale

7
1. Scale-space Extrema Detection (contd)

• DoG images are grouped by octaves (i.e.,
  doubling of s0)
• Fixed number of levels per octave

22s0
down-sample
where
2s0
s0
8
1. Scale-space Extrema Detection (contd)

• Images within each octave are
• separated by a constant factor k
• If each octave is divided in s-1 intervals
• ks2 or k21/s

9
SIFT parameters

• Parameters (i.e., scales per octave, s0 etc.)
  were chosen experimentally based on keypoint (i)
  repeatability, (ii) localization, and (iii)
  matching accuracy.
• From Lowes paper
• Number of scales per octave 3
• s0 1.6

10
1. Scale-space Extrema Detection (contd)

• Extract local extrema (i.e., minima or maxima)
  in DoG pyramid.
• Compare each point to its 8 neighbors at the
  same level, 9 neighbors
• in the level above, and 9 neighbors in the
  level below (i.e., 26 total).

11
2. Keypoint Localization

• Determine the location and scale of keypoints to
  sub-pixel and sub-scale accuracy by fitting a 3D
  quadratic polynomial

keypoint location
offset
sub-pixel, sub-scale estimated location
Substantial improvement to matching and stability!
12
2. Keypoint Localization (contd)

• Reject keypoints having low contrast.
• i.e., sensitive to noise

13
2. Keypoint Localization (contd)

• Reject points lying on edges (or being close to
  edges)
• Harris uses the auto-correlation matrix

R(AW) det(AW) a trace2(AW) or
R(AW) ?1 ?2- a (?1 ?2)2
14
2. Keypoint Localization (contd)

• SIFT uses the Hessian matrix.
• i.e., Hessian encodes principal curvatures

a largest eigenvalue (?max) ß smallest
eigenvalue (?min) (proportional to principal
curvatures)
(r a/ß)
(SIFT uses r 10)
Reject keypoint if
15
2. Keypoint Localization (contd)

• (a) 233x189 image
• (b) 832 DoG extrema
• (c) 729 left after low
• contrast threshold
• (d) 536 left after testing
• ratio based on Hessian

16
3. Orientation Assignment

• Create histogram of gradient directions, within a
  region around the keypoint, at selected scale

36 bins (i.e., 10o per bin)

• Histogram entries are weighted by (i) gradient
  magnitude and (ii) a
• Gaussian function with s equal to 1.5 times the
  scale of the keypoint.

17
3. Orientation Assignment (contd)

• Assign canonical orientation at peak of smoothed
  histogram (fit parabola to better localize peak).
• In case of peaks within 80 of highest peak,
  multiple orientations assigned to keypoints.
• About 15 of keypoints has multiple orientations
  assigned.
• Significantly improves stability of matching.

18
4. Keypoint Descriptor

1. Take a 16 x16 window around detected interest
   point.
2. Divide into a 4x4 grid of cells.
3. Compute histogram in each cell.

(8 bins)
16 histograms x 8 orientations
128 features
19
4. Keypoint Descriptor (contd)

• Each histogram entry is weighted by (i) gradient
  magnitude and (ii) a Gaussian function with s
  equal to 0.5 times the width of the descriptor
  window.

20
4. Keypoint Descriptor (contd)

• Partial Voting distribute histogram entries into
  adjacent bins (i.e., additional robustness to
  shifts)
• Each entry is added to all bins, multiplied by a
  weight of 1-d,
• where d is the distance from the bin it belongs.

21
4. Keypoint Descriptor (contd)

• Descriptor depends on two main parameters
• (1) number of orientations r
• (2) n x n array of orientation histograms

rn2
features
SIFT r8, n4
128 features
22
4. Keypoint Descriptor (contd)

• Invariance to linear illumination changes
• Normalization to unit length is sufficient.

128 features
23
4. Keypoint Descriptor (contd)

• Non-linear illumination changes
• Saturation affects gradient magnitudes more than
  orientations
• Threshold entries to be no larger than 0.2 and
  renormalize to unit length

128 features
24
Matching SIFT features

• Given a feature in I1, how to find the best match
  in I2?
• Define distance function that compares two
  descriptors.
• Test all the features in I2, find the one with
  min distance.

I2
I1
25
Matching SIFT features (contd)

•  

26
Matching SIFT features (contd)

• Accept a match if SSD(f1,f2) lt t
• How do we choose t?

27
Matching SIFT features (contd)

• A better distance measure is the following
• SSD(f1, f2) / SSD(f1, f2)
• f2 is best SSD match to f1 in I2
• f2 is 2nd best SSD match to f1 in I2

28
Matching SIFT features (contd)

• Accept a match if SSD(f1, f2) / SSD(f1, f2) lt t
• t0.8 has given good results in object
  recognition.
• 90 of false matches were eliminated.
• Less than 5 of correct matches were discarded

29
Matching SIFT features (contd)

• How to evaluate the performance of a feature
  matcher?

30
Matching SIFT features (contd)

• Threshold t affects of correct/false matches

• True positives (TP) of detected matches that
  are correct
• False positives (FP) of detected matches that
  are incorrect

31
Matching SIFT features(contd)

• ROC Curve
• - Generated by computing (FP, TP) for
  different thresholds.
• - Maximize area under the curve (AUC).

1
http//en.wikipedia.org/wiki/Receiver_operating_c
haracteristic
32
Applications of SIFT

• Object recognition
• Object categorization
• Location recognition
• Robot localization
• Image retrieval
• Image panoramas

33
Variations of SIFT features

• PCA-SIFT
• SURF
• GLOH

34
SIFT Steps - Review

• (1) Scale-space extrema detection
• Extract scale and rotation invariant interest
  points (i.e., keypoints).
• (2) Keypoint localization
• Determine location and scale for each interest
  point.
• Eliminate weak keypoints
• (3) Orientation assignment
• Assign one or more orientations to each keypoint.
• (4) Keypoint descriptor
• Use local image gradients at the selected scale.

D. Lowe, Distinctive Image Features from
Scale-Invariant Keypoints, International Journal
of Computer Vision, 60(2)91-110, 2004.
35
PCA-SIFT

• Steps 1-3 are the same Step 4 is modified.
• Take a 41 x 41 patch at the given scale, centered
  at the keypoint, and normalized to a canonical
  direction.

Yan Ke and Rahul Sukthankar, PCA-SIFT A More
Distinctive Representation for Local Image
Descriptors, Computer Vision and Pattern
Recognition, 2004
36
PCA-SIFT

• Instead of using weighted histograms, concatenate
  the horizontal and vertical gradients (39 x 39)
  into a long vector.
• Normalize vector to unit length.

2 x 39 x 39 3042 vector
37
PCA-SIFT

• Reduce the dimensionality of the vector using
  Principal Component Analysis (PCA)
• e.g., from 3042 to 36
• Some times, less discriminatory than SIFT.

38
SURF Speeded Up Robust Features

• Speed-up computations by fast approximation of
  (i) Hessian matrix and (ii) descriptor using
  integral images.
• What is an integral image?

Herbert Bay, Tinne Tuytelaars, and Luc Van Gool,
SURF Speeded Up Robust Features, European
Computer Vision Conference (ECCV), 2006.
39
Integral Image

• The integral image IS(x,y) of an image I(x, y)
  represents the sum of all pixels in I(x,y) of a
  rectangular region formed by (0,0) and (x,y).
• .

Using integral images, it takes only four array
references to calculate the sum of pixels over a
rectangular region of any size.
40
SURF Speeded Up Robust Features (contd)

• Approximate Lxx, Lyy, and Lxy using box filters.
• Can be computed very fast using integral images!

(box filters shown are 9 x 9 good
approximations for a Gaussian with s1.2)
derivative
approximation
approximation
derivative
41
SURF Speeded Up Robust Features (contd)

• In SIFT, images are repeatedly smoothed with a
  Gaussian and subsequently sub-sampled in order to
  achieve a higher level of the pyramid.

42
SURF Speeded Up Robust Features (contd)

• Alternatively, we can use filters of larger size
  on the original image.
• Due to using integral images, filters of any size
  can be applied at exactly the same speed!

(see Tuytelaars paper for details)
43
SURF Speeded Up Robust Features (contd)

• Approximation of H

Using DoG
Using box filters
44
SURF Speeded Up Robust Features (contd)

• Instead of using a different measure for
  selecting the location and scale of interest
  points (e.g., Hessian and DOG in SIFT), SURF uses
  the determinant of to find both.
• Determinant elements must be weighted to obtain a
  good approximation

45
SURF Speeded Up Robust Features (contd)

• Once interest points have been localized both in
  space and scale, the next steps are
• (1) Orientation assignment
• (2) Keypoint descriptor

46
SURF Speeded Up Robust Features (contd)

• Orientation assignment

Circular neighborhood of radius 6s around the
interest point (s the scale at which the point
was detected)
600 window
x response y response
Haar wavelets (responses weighted with
Gaussian) side length 4s
dx
dy
Can be computed very fast using integral images!
47
SURF Speeded Up Robust Features (contd)

• Orientation assignment
• - The image is convoluted with two first-order
  Haar wavelets.
• - The filter responses at certain sampling points
  around the keypoint are represented as a vector
  in a two-dimensional space.
• - A rotating window of 600 is used to sum up all
  vectors within its range, and the longest
  resulting vector determines the orientation.

48
SURF Speeded Up Robust Features (contd)

• Keypoint descriptor (square region of size 20s)

• Sum the response over each sub-region for dx and
  dy separately.
• To bring in information about the polarity of the
  intensity changes, extract the sum of absolute
  value of the responses too.
• Feature vector size
• 4 x 16 64

4 x 4 grid
49
SURF Speeded Up Robust Features (contd)

• SURF-128
• The sum of dx and dx are computed separately
  for points where dy lt 0 and dy gt0
• Similarly for the sum of dy and dy
• More discriminatory!

50
SURF Speeded Up Robust Features

• Has been reported to be 3 times faster than SIFT.
• Less robust to illumination and viewpoint changes
  compared to SIFT.

K. Mikolajczyk and C. Schmid,"A Performance
Evaluation of Local Descriptors", IEEE
Transactions on Pattern Analysis and Machine
Intelligence, vol. 27, no. 10, pp. 1615-1630,
2005.
51
Gradient location-orientation histogram (GLOH)

• Compute SIFT using a log-polar location grid
• 3 bins in radial direction (i.e., radius 6, 11,
  and 15)
• 8 bins in angular direction
• Gradient orientation quantized in 16 bins.
• Total (2x81)16272 bins ?PCA.

K. Mikolajczyk and C. Schmid,"A Performance
Evaluation of Local Descriptors", IEEE Trans. on
Pattern Analysis and Machine Intelligence, vol.
27, no. 10, pp. 1615-1630, 2005.