Algorithms

1
Algorithms

• Point correspondences
• Salient point detection
• Local descriptors
• Matrix decompositions
• RQ decomposition
• Singular value decomposition - SVD
• Estimation
• Systems of linear equations
• Solving systems of linear equations
• Direct Linear Transform DLT
• Normalization
• Iterative error / cost minimization
• Outliers ? Robustness, RANSAC
• Pose estimation
• Perspective n-point problem PnP

2
Point Correspondences - Example 1
Structure and motion from natural landmarks
Schweighofer ? Stereo reconstruction of Harris
corners
3
Point Correspondences - Example 2
elliptical support
normalization canonical view
correspondence
MikolajczykSchmid
4
Salient points (corners) based on 1st derivatives

• Autocorrelation of 2D image signal Moravec
• Approximation by sum of squared differences (SSD)
  
• Window W
• Differences between grayvalues in W and a window
  shifted by (?x,?y)
• Four different shift directions fi(x,y)
• A corner is detected, when fMoravecgtth

5
Salient points (corners) based on 1st derivatives

• Autocorrelation (second moment) matrix
• Avoids various shift directions
• Approximate I(xw?x,yw?y) by Taylor expansion
• Rewrite f(x,y)

second moment matrix M
6
Salient points (corners) based on 1st derivatives

• Autocorrelation (second moment) matrix
• M can be used to derive a measure of cornerness
• Independent of various displacements (?x,?y)
• Corner significant gradients in gt1 directions ?
  rank M 2
• Edge significant gradient in 1 direction ? rank
  M 1
• Homogeneous region ? rank M 0
• Several variants of this corner detector
• KLT corners, Förstner corners

7
Salient points (corners) based on 1st derivatives

• Harris corners
• Most popular variant of a detector based on M
• Local derivatives with derivation scale sD
• Convolution with a Gaussian with integration
  scale sI
• MHarris for each point x in the image
• Cornerness cHarris does not require to compute
  eigenvalues
• Corner detection cHarris gt tHarris

8
Salient points (corners) based on 1st derivatives

• Harris corners

9
Salient points (corners) based on 2nd derivatives

• Hessian determinant
• Local maxima of det H Beaudet
• Zero crossings of det H DreschlerNagel
• Detectors are related to curvature
• Invariant to rotation
• Similar cornerness measure local maxima of K
  KitchenRosenfeld

10
Salient points (corners) based on 2nd derivatives

• DoG / LoG MarrHildreth
• Zero crossings
• Mexican hat, Sombrero
• Edge detector !
• Lowes DoG keypoints Lowe
• Edge ? zero-crossing
• Blob at corresponding scale local extremum !
• Low contrast corner suppression threshold
• Assess curvature ? distinguish corners from edges
• Keypoint detection

11
Salient points (corners) without derivatives

• Morphological corner detector Laganière
• 4 structuring elements
• , ?, x, ?
• Assymetrical closing

12
Salient points (corners) without derivatives

• SUSAN corners SmithBrady
• Sliding window
• Faster than Harris

13
Salient points (corners) without derivatives

• Kadir/Brady saliency KadirBrady
• Histograms
• Shannon entropy
• Scale selection
• Used in constellation
• model Fergus et al.

14
Salient points (corners) without derivatives

• MSER maximally stable extremal regions Matas
  et al.
• Successive thresholds
• Stability regions
• survive over many
• thresholds

15
Affine covariant corner detectors

• Locally planar patch ? affine distortion
• Detect characteristic scale
• see also Lindeberg, scale-space
• Recover affine deformation that fits local image
  data best

elliptical support
normalization canonical view
correspondence
MikolajczykSchmid
16
Scaled Harris Corner Detector

• Harris Laplace MikolajczykSchmid, Mikolajczyk
  et al.

17
Scaled Hessian Detector

• Hessian Laplace MikolajczykSchmid ,
  Mikolajczyk et al.

18
Harris Affine Detector

• Harris affine MikolajczykSchmid , Mikolajczyk
  et al.

19
Hessian Affine Detector

• Hessian affine MikolajczykSchmid ,
  Mikolajczyk et al.

20
Qualitative comparison of detectors (1)
Harris affine
Harris Laplace
Harris
Hessian affine
Hessian Laplace
21
Qualitative comparison of detectors (2)
morphological
Kadir/Brady
MSER
SUSAN
22
Descriptors (1)

• Representation of salient regions
• descriptive features ? feature vector
• There are many possibilities !
• Categorization vs. specific OR, matching
• Sufficient descriptive power
• Not too much emphasis on specific individuals
• Performance is often category-specific

vs. AR ?
feature vector extracted from patch Pn
23
Descriptors (2)

• Grayvalues
• Raw pixel values of a patch P
• local appearance-based description
• local affine frame LAF ObdrálekMatas for
  MSER
• General moments of order pq
• Moment invariants
• Central moments µpq invariant to translation

24
Descriptors (3)

• Moment invariants
• Normalized central moments
• Translation, rotation, scale invariant moments F1
  ... F7 Hu
• Geometric/photometric, color invariants vanGool
  et al.
• Filters
• local jets KoenderinkVanDoorn
• Gabor banks, steerable filters, discrete cosine
  transform DCT

25
Descriptors (4)

• SIFT descriptors Lowe
• Scale invariant feature transform
• Calculated for local patch P 8x8 or 16x16
  pixels
• Subdivision into 4x4 sample regions
• Weighted histogram of 8 gradient directions 0º,
  45º,
• SIFT vector dimension 128 for a 16x16 patch

Lowe
26
Algorithms

• Point correspondences
• Salient point detection
• Local descriptors
• Matrix decompositions
• RQ decomposition
• Singular value decomposition - SVD
• Estimation
• Systems of linear equations
• Solving systems of linear equations
• Direct Linear Transform DLT
• Normalization
• Iterative error / cost minimization
• Outliers ? Robustness, RANSAC
• Pose estimation
• Perspective n-point problem PnP

27
RQ Decomposition (1)

• Remember camera projection matrix P
• P can be decomposed, e.g. finite projective camera

28
RQ Decomposition (2)

• Unfortunately R refers to upper triangular, Q
  to rotation
• Givens rotations
• How to decompose a given 3 x 3 matrix (say M) ?
• MQx enforcing M32 0, first column of M
  unchanged, last two columns replaced by linear
  combinations of themselves
• MQxQy enforcing M31 0, 2nd column unchanged
  (M32 remains 0)
• MQxQyQz enforcing M21 0, first two columns
  replaced by linear combinations of themselves,
  thus M31 and M32 remain 0
• ? MQxQyQz R, M RQxTQyTQzT , where R is upper
  triangular
• How to enforce
• e.g. M21 0 ?

29
Singular Value Decomposition - SVD

• Given a square matrix A (e.g. 3x3)
• A can be decomposed into
• where U and V are orthogonal matrices, and
• D is a diagonal matrix with
• non-negative entries,
• entries in descending order.
• the column of V corresponding to the smallest
  singular value
• ? the last column of
  V

30
SVD (2)

• SVD is also possible when A is non-square (e.g. m
  x n, mn)
• A can again be decomposed into
• where U is m x n with orthogonal columns
  (UTUInxn),
• D is an n x n diagonal matrix with
• non-negative entries,
• entries in descending order,
• V is an n x n orthogonal matrix.

31
SVD for Least-Squares Solutions

• Overdetermined system of linear equations
• Find least-squares ( algebraic error! ) solution
• Algorithm
• Find the SVD
• Set
• Find
• The solution is

Even easier for Ax0 x is the last column of V
32
Algorithms

• Point correspondences
• Salient point detection
• Local descriptors
• Matrix decompositions
• RQ decomposition
• Singular value decomposition - SVD
• Estimation
• Systems of linear equations
• Solving systems of linear equations
• Direct Linear Transform DLT
• Normalization
• Iterative error / cost minimization
• Outliers ? Robustness, RANSAC
• Pose estimation
• Perspective n-point problem PnP

33
Systems of Linear Equations (1)

• Estimation of
• A homography H
• The fundamental matrix
• The camera projection matrix
• By finding n point correspondences
• between 2 images
• between image and scene
• And solving a system of linear equations
• Typical form

(2n x 9) / (2n x 12) matrix representing
correspondences
9 - vector representing H, F 12 - vector
representing P
34
Systems of Linear Equations (2)

• How to obtain ?
• Homography H
• 3 x 3 matrix, 8 DoF, non-singular
• at least 4 point correspondences are required
• Fundamental matrix F
• 3 x 3 matrix, 7 DoF, rank 2
• at least 7 point correspondences are required
• Camera projection matrix P
• 3 x 4 matrix, 11 DoF, decomposition into K, R, t
• at least 5-1/2 (6) point correspondences are
  required

why ?
35
Homography Estimation (1)
Point correspondences
Equation defining the computation of H
Some notation
Simple rewriting
36
Homography Estimation (2)
Ai is a 3 x 9 matrix , h is a 9-vector

• the system describes 3 equations
• the equations are linear in the unknown h
• elements of Ai are quadratic in the known point
  coordinates
• only 2 equations are linearly independent
• thus, the 3rd equation is usually omitted
  Sutherland 63

Ai is a 2 x 9 matrix , h is a 9-vector
37
Homography Estimation (3)
1 point correspondence defines 2 equations H has
9 entries, but is defined up to scale ? 8 degrees
of freedom ? at least 4 point correspondences
needed

• General case
• overdetermined
• n point correspondences
• 2n equations, A is a 2n x 9 matrix

4 x 2 equations
38
Camera Projection Matrix Estimation
homography H
projection matrix P
very similar !

• n point correspondences ? 2n equations that are
  linear in elements of P
• A is a 2n x 12 matrix, entries are quadratic in
  point coordinates
• p is a 12-vector
• P has only 11 degrees of freedom
• a minimum of 11 equations is required ? 5-1/2
  (6) point correspondences

39
Fundamental (Essential) Matrix Estimation (1)

• solving is different from
  solving
• each correspondence gives only
  one equation in the coefficients of F !
• for n point matches we again obtain a set of
  linear equations (linear in f1-f9)

40
Fundamental (Essential) Matrix Estimation (2)

• F is a 3 x 3 matrix, has rank 2, F 0 ? F has
  only 7 degrees of freedom
• at least 7 point correspondences are required to
  estimate F

• Back to the solution of systems of linear
  equations !
• ? similar systems of equations, but different
  constraints ?

41
SVD for Least-Squares Solutions

• Overdetermined system of linear equations
• Find least-squares ( algebraic error! ) solution
• Algorithm
• Find the SVD
• Set
• Find
• The solution is

Even easier for Ax0 x is the last column of V
? This is also called direct linear transform
DLT ?
42
Relevant Issues in Practice

• Poor condition of A ? Normalization
• Algebraic error vs.
• geometric error, ? Iterative minimization
• nonlinearities (lens dist.)
• Outliers ? Robust algorithms
• (RANSAC)