Instructor: Mircea Nicolescu

1
CS 791EComputer Vision

• Instructor Mircea Nicolescu
• Lecture 19

2
2D Geometric Transformations

• General form of a transformation matrix

• Affine transformations

• Involve translations, rotations, scale, and shear
• Preserve parallelism of lines but not lengths and
  angles

3
2D Geometric Transformations

• Similarity transformations

• Involve rotation, translation, uniform scaling
• Preserve angles and length ratios

• Rigid transformations

• Involve only translations and rotations
• Preserve areas, angles and lengths

4
2D Geometric Transformations

• Rigid transformations cont.

• Property the upper 2x2 submatrix is orthonormal

• Example

5
2D Geometric Transformations

• Shear transformations

• Change the shape of the object.

• Shear along the y-axis

• Shear along the x-axis

6
3D Geometric Transformations

• Coordinate systems

• Right-handed vs. left-handed systems

• Positive rotation angles for right-handed systems

7
3D Geometric Transformations

• Homogeneous coordinates of a 3D point

• Idea add a third coordinate (x, y, z) ? (xh,
  yh, zh, w)

• Homogenize (xh, yh, zh, w)

• In general (x, y, z) ? (xw, yw, zw, w) (i.e.,
  xhxw, yhyw, zhzw)
• w can assume any value (w ? 0), for example, w
  1
• (x, y, z) ? (x, y, z, 1) (no division required
  when you homogenize)
• (x, y, z) ? (2x, 2y, 2z, 2) (division required
  when you homogenize)
• Each point (x, y, z) corresponds to a line in the
  4D-space of homogeneous coordinates

8
3D Geometric Transformations

• Translation

9
3D Geometric Transformations

• Scaling

10
3D Geometric Transformations

• Rotation

• Rotation about the z-axis

11
3D Geometric Transformations

• Rotation

• Rotation about the x-axis

12
3D Geometric Transformations

• Rotation

• Rotation about the y-axis

13
3D Geometric Transformations

• Change of coordinate systems

• Suppose that you know the coordinates of P3 in
  the xyz system and you need its coordinates in
  the RxRyRz system.

• You need to recover the transformation T from
  RxRyRz to xyz.
• Apply T on P3 to compute its coordinates in the
  RxRyRz system.

14
3D Geometric Transformations

• Change of coordinate systems cont.

• Assume that ux, uy, uz are the unit vectors in
  the xyz coordinate system.
• Assume that rx, ry, rz are the unit vectors in
  the RxRyRz coordinate system (note rx, ry, rz
  are represented in the xyz coordinate system).
• Find a mapping rz ? uz, rx ? ux, and ry ? uy

15
3D Geometric Transformations

• Change of coordinate systems cont.

• Verify

16
Singular Value Decomposition (SVD)

• Definition

• Any real mxn matrix A can be decomposed uniquely
  as

A UDVT

• U is mxn and column orthogonal (its columns are
  eigenvectors of AAT)

(AAT UDVT VDUT UD2UT )

• V is nxn and orthogonal (its columns are
  eigenvectors of ATA)

(ATA VDUT UDVT VD2VT )

• D is nxn diagonal (non-negative real values
  called singular values)

D diag(s1, s2, , sn) ordered so that s1 ? s2 ?
? sn (if s is a singular value of A, its square
is an eigenvalue of ATA)
17
Singular Value Decomposition (SVD)

• Definition cont.

• If U (u1 u2 . . . un) and V (v1 v2 . . . vn),
  then

(actually, the sum goes from 1 to r where r is
the rank of A)

• Example

• The eigenvalues of AAT, ATA are

18
Singular Value Decomposition (SVD)

• Example cont.

• The eigenvectors of AAT, ATA are

• The expansion of A is

• Important note that the second eigenvalue is
  much smaller than the first if we neglect it
  from the above summation, we can represent A by
  introducing relatively small errors

19
Singular Value Decomposition (SVD)

• Computing the rank using SVD

• The rank of a matrix the number of non-zero
  singular values.

• Computing the inverse of a matrix using SVD

• A square matrix A is nonsingular iff si ? 0 for
  all i

• If A is a nxn nonsingular matrix, then its
  inverse is given by

• If A is singular or ill-conditioned, then we can
  use SVD to approximate its inverse by the
  following matrix

20
Singular Value Decomposition (SVD)

• The condition of a matrix

• Consider the system of linear equations

• If small changes in b can lead to relatively
  large changes in the solution x, then A is
  ill-conditioned.

• The ratio given below is related to the condition
  of A and measures the degree of singularity of A
  (the larger this value is, the closer A is to
  being singular)

21
Singular Value Decomposition (SVD)

• Least squares solutions of mxn systems

• Consider the over-determined system of linear
  equations

• Let r be the residual vector for some x

• The vector x which yields the smallest possible
  residual is called a least-squares solution (it
  is an approximate solution).

22
Singular Value Decomposition (SVD)

• Least squares solutions of mxn systems cont.

• Although a least-squares solution always exist,
  it might not be unique
• The least-squares solution x with the smallest
  norm x is unique and is given by

• Example

23
Singular Value Decomposition (SVD)

• Computing A using SVD

• If ATA is ill-conditioned or singular, we can use
  SVD to obtain a least squares solution as follows

• Least squares solutions of nxn systems

• If A is ill-conditioned or singular, SVD can give
  us a workable solution in this case too

24
Singular Value Decomposition (SVD)

• Homogeneous systems

• If b0 then the linear system is called
  homogeneous

• The minimum-norm solution in this case is x0
  (trivial solution).
• For homogeneous linear systems, the meaning of a
  least-squares solution is modified by imposing
  the constraint

• This is a "constrained" optimization problem

25
Singular Value Decomposition (SVD)

• Homogeneous systems cont.

• The minimum-norm solution for homogeneous systems
  is not always unique.

• Special case rank(A) n 1 (m ? n 1, sn0)
• solution is x avn (a is a constant)
• (vn is the last column of V corresponds to
  the smallest s)

• General case rank(A) n k (m ? n k,
  sn-k1sn0)
• solution is x a1vn-k1 a2vn-k2 akvn
  
• (ai is a constant with a12 a22 ak2 1)

26
Reference Frames

• Five reference frames are typically used for
  general problems in 3D scene analysis

27
Reference Frames

• Object Coordinate Frame

• This is a 3D coordinate system xb, yb, zb
• It is used to model ideal objects in both
  computer graphics and computer vision.
• It is needed to inspect an object (e.g., to check
  if a particular hole is in proper position
  relative to other holes)
• Object coordinates do not change regardless how
  the object is placed in the scene.
• Notation (Xb, Yb, Zb)T

28
Reference Frames

• World Coordinate Frame

• This is a 3D coordinate system xw, yw, zw
• The scene consists of object models that have
  been placed (rotated and translated) into the
  scene, yielding coordinates in the world
  coordinate system.
• It is needed to relate objects in 3D (e.g., the
  image sensor tells the robot where to pick up a
  bolt and in which hole to insert it).
• Notation (Xw, Yw, Zw)T

29
Reference Frames

• Camera Coordinate Frame

• This is a 3D coordinate system (xc, yc, zc axes)
• Its purpose is to represent objects with respect
  to the location of the camera.
• Notation (Xc, Yc, Zc)T

• Image Plane Coordinate Frame (CCD plane)

• This is a 2D coordinate system (xf, yf axes)
• Describes coordinates of 3D points projected on
  the image plane.
• The projection of A is point a.
• Notation (x, y)T

30
Reference Frames

• Pixel Coordinate Frame

• This is a 2D coordinate system (r, c axes)
• Each pixel in this frame has integer coordinates.
• Point A gets projected to image point (ar, ac)
  where ar and ac are integer row and column.
• Notation (xim, yim)T

31
Reference Frames

• Transformations between frames

32
Projection

• Pinhole camera and perspective projection

• This is the simplest imaging device which,
  however, captures accurately the geometry of
  perspective projection.
• Rays of light enters the camera through an
  infinitesimally small aperture.
• The intersection of the light rays with the image
  plane form the image of the object.
• Such a mapping from three dimensions onto two
  dimensions is called perspective projection.

33
Projection

• A simplified geometric arrangement

• In general, the world and camera coordinate
  systems are not aligned.
• To simplify the derivation of the perspective
  projection equations, we will make the following
  assumptions
• the center of projection coincides with the
  origin of the world.
• the camera axis (optical axis) is aligned with
  the worlds z-axis.
• avoid image inversion by assuming that the image
  plane is in front of the center of projection.

34
Projection

• Terminology

• The model consists of a plane (image plane) and a
  3D point O (center of projection).
• The distance f between the image plane and the
  center of projection O is the focal length (the
  distance between the lens and the CCD array).
• The line through O and perpendicular to the image
  plane is the optical axis.
• The intersection of the optical axis with the
  image place is called principal point or image
  center.
• (note the principal point is not always the
  "actual" center of the image)

35
Projection

• The equations of perspective projection

• Using the following similar triangles

36
Projection

• The equations of perspective projection cont.

• Using matrix notation

• Verify the correctness of the above matrix
  (homogenize using w Z)

37
Projection

• Properties of perspective projection

• Many-to-one mapping

• The projection of a point is not unique (any
  point on the line OP has the same projection).

38
Projection

• Properties of perspective projection cont.

• Scaling/Foreshortening

• The distance to an object is inversely
  proportional to its image size.

• When a line (or surface) is parallel to the image
  plane, the effect of perspective projection is
  scaling.
• When an line (or surface) is not parallel to the
  image plane, we use the term foreshortening to
  describe the projective distortion (the dimension
  parallel to the optical axis is compressed
  relative to the frontal dimension).

39
Projection

• Properties of perspective projection cont.

• Effect of focal length

• As f gets smaller, more points project onto the
  image plane (wide-angle camera).
• As f gets larger, the field of view becomes
  smaller (more telescopic).

• Lines, distances, angles

• Lines in 3D project to lines in 2D.
• Distances and angles are not preserved.
• Parallel lines do not in general project to
  parallel lines (unless they are parallel to the
  image plane).

40
Projection

• Properties of perspective projection cont.

• Vanishing point

• parallel lines in space project perspectively
  onto lines that on extension intersect at a
  single point in the image plane called vanishing
  point or point at infinity.
• (alternative definition) the vanishing point of a
  line depends on the orientation of the line and
  not on the position of the line.
• the vanishing point of any given line in space is
  located at the point in the image where a
  parallel line through the center of projection
  intersects the image plane.

41
Projection

• Properties of perspective projection cont.

• Vanishing line

• the vanishing points of all the lines that lie on
  the same plane form the vanishing line.
• also defined by the intersection of a parallel
  plane through the center of projection with the
  image plane.

42
Projection

• Orthographic Projection

• It is the projection of a 3D object onto a plane
  by a set of parallel rays orthogonal to the image
  plane.
• It is the limit of perspective projection as f ?
  ? (f / Z ? 1)

43
Projection

• Orthographic Projection cont.

• Using matrix notation

• Verify the correctness of the above matrix
  (homogenize using w1)

• Properties of orthographic projection

• Parallel lines project to parallel lines.
• Size does not change with distance from the
  camera.

44
Projection

• Weak Perspective Projection

• Perspective projection is a non-linear
  transformation.
• We can approximate perspective by scaled
  orthographic projection (linear transformation)
  if

45
Projection

• Weak Perspective Projection cont.

• The term f / is a scale factor now (every
  point is scaled by the same factor).
• Using matrix notation

• Verify the correctness of the above matrix
  (homogenize using w )

46
Camera Parameters

• Assumptions made so far

• Equations derived so far are valid only when
• all distances are measured in the cameras
  reference frame.
• the image coordinates have their origin at the
  principal point.

• In general, the world and pixel coordinate
  systems are related by a set of physical
  parameters such as

• the focal length of the lens
• the size of the pixels
• the position of the principal point
• the position and orientation of the camera

47
Camera Parameters

• Camera parameters

• Two types of parameters need to be recovered in
  order for us to reconstruct the 3D structure of a
  scene from the pixel coordinates of its image
  points

• Extrinsic camera parameters the parameters that
  define the location and orientation of the camera
  reference frame with respect to a known world
  reference frame.
• Intrinsic camera parameters the parameters
  necessary to link the pixel coordinates of an
  image point with the corresponding coordinates in
  the camera reference frame.

48
Camera Parameters

• Extrinsic camera parameters

• Identify uniquely the transformation between the
  unknown camera reference frame and the known
  world reference frame.
• Typically, determining these parameters means
• finding the translation vector between the
  relative positions of the origins of the two
  reference frames.
• finding the rotation matrix that brings the
  corresponding axes of the two frames into
  alignment (i.e., onto each other)

49
Camera Parameters

• Extrinsic camera parameters cont.

• Using the extrinsic camera parameters, we can
  find the relation between the coordinates of a
  point P in world (Pw) and camera (Pc) coordinates

50
Camera Parameters

• Intrinsic camera parameters

• These are the parameters that characterize the
  optical, geometric, and digital characteristics
  of the camera
• the perspective projection (focal length f ).
• the transformation between image plane
  coordinates and pixel coordinates.
• the geometric distortion introduced by the optics.

• From Camera Coordinates to Image Plane Coordinates

• Apply perspective projection

51
Camera Parameters

• Intrinsic camera parameters cont.

• From Image Plane Coordinates to Pixel Coordinates

• where (ox, oy) are the coordinates of the
  principal point (in pixels, ox N/2, oy M/2 if
  the principal point is the center of the image)
  and sx, sy correspond to the effective size of
  the pixels in the horizontal and vertical
  directions (in millimeters).

52
Camera Parameters

• Intrinsic camera parameters cont.

• Using matrix notation

• Relating pixel coordinates to world coordinates

53
Camera Parameters

• Intrinsic camera parameters cont.

• Image distortions due to optics

• Assume radial distortion

• where (xd, yd) are the coordinates of the
  distorted points (r2 xd2 yd2)
• k1 and k2 are intrinsic parameters too but will
  not be considered here...

54
Camera Parameters

• Combine extrinsic with intrinsic camera parameters

• The matrix containing the intrinsic camera
  parameters

• The matrix containing the extrinsic camera
  parameters

• Using homogeneous coordinates

55
Camera Parameters

• Combine extrinsic with intrinsic camera
  parameters cont.

• Homogenization is needed to obtain the pixel
  coordinates

• M is called the projection matrix (it is a 3 x 4
  matrix).
• Note the relation of 3D points and their 2D
  projections can be seen as a linear
  transformation from the projective space (Xw, Yw,
  Zw, 1)T to the projective plane (xh, yh, w)T.

56
Camera Parameters

• The perspective camera model (using matrix
  notation)

• Assuming ox oy 0 and sx sy 1

• Verify the correctness of the above matrix

• After homogenization (we get the same equations
  as before)

57
Camera Parameters

• The weak perspective camera model (using matrix
  notation)

• Verify the correctness of the above matrix

• After homogenization

58
Camera Parameters

• The affine camera model

• The entries of the projection matrix are totally
  unconstrained (except for the zeros on last row)

• The affine model does not appear to correspond to
  any physical camera.
• Leads to simple equations and appealing geometric
  properties.
• Does not preserve angles but does preserve
  parallelism.

59
Camera Calibration

• Goal
• Produce an estimate of the extrinsic and
  intrinsic camera parameters.

• Procedure
• Given the correspondences between a set of point
  features in the world (Xw, Yw, Zw) and their
  projections in an image (xim, yim), compute the
  intrinsic and extrinsic camera parameters.

60
Camera Calibration

• Establishing the correspondences
• Calibration methods rely on one or more images of
  a calibration pattern
• a 3D object of known geometry.
• located in a known position in space.
• that generates image features which can be
  located accurately.

61
Camera Calibration

• Consider this calibration pattern
• consists of two orthogonal grids.
• equally spaced black squares drawn on white,
  perpendicular planes.
• assume that the world reference frame is centered
  at the lower left corner of the left grid, with
  axes parallel to the three directions identified
  by the calibration pattern.

• given the size of the planes, their angle, the
  number of squares etc. (all known by
  construction), the coordinates of each vertex can
  be computed in the world reference frame using
  trigonometry.
• the projection of the vertices on the image can
  be found by intersecting the edge lines of the
  corresponding square sides (or through corner
  detection).

62
Camera Calibration

• Methods

• Direct parameter calibration.
• Direct recovery of the intrinsic and extrinsic
  camera parameters.
• Recovery of camera parameters through the
  projection matrix

• Estimate the elements of the projection matrix.
• Compute the intrinsic/extrinsic parameters as
  closed-form functions of the entries of the
  projection matrix.