Cameras, lenses and sensors

1
Cameras, lenses and sensors

• Marc Pollefeys
• COMP 256

2
Cameras, lenses and sensors

• Camera Models
• Pinhole Perspective Projection
• Affine Projection
• Camera with Lenses
• Sensing
• The Human Eye

Reading Chapter 1.
3
Images are two-dimensional patterns of brightness
values.
Figure from US Navy Manual of Basic Optics and
Optical Instruments, prepared by Bureau of Naval
Personnel. Reprinted by Dover Publications, Inc.,
1969.
They are formed by the projection of 3D objects.
4
Animal eye a looonnng time ago.
Photographic camera Niepce, 1816.
Pinhole perspective projection Brunelleschi,
XVth Century. Camera obscura XVIth Century.
5
Distant objects appear smaller
6
Parallel lines meet

• vanishing point

7
Vanishing points
VPL
H
VPR
VP2
VP1
To different directions correspond different
vanishing points
VP3
8
Geometric properties of projection

• Points go to points
• Lines go to lines
• Planes go to whole image
• or half-plane
• Polygons go to polygons
• Degenerate cases
• line through focal point yields point
• plane through focal point yields line

9
Pinhole Perspective Equation
10
Affine projection models Weak perspective
projection
is the magnification.
When the scene relief is small compared its
distance from the Camera, m can be taken
constant weak perspective projection.
11
Affine projection models Orthographic projection
When the camera is at a (roughly constant)
distance from the scene, take m1.
12
Planar pinhole perspective
Orthographic projection
Spherical pinhole perspective
13
Limits for pinhole cameras
14
Camera obscura lens
?
15
Lenses
Snells law n1 sin a1 n2 sin a2
Descartes law
16
Paraxial (or first-order) optics
Snells law n1 sin a1 n2 sin a2
Small angles n1 a1 ? n2a2
17
Thin Lenses
spherical lens surfaces incoming light ?
parallel to axis thickness ltlt radii same
refractive index on both sides
18
Thin Lenses
http//www.phy.ntnu.edu.tw/java/Lens/lens_e.html
19
Thick Lens
20
The depth-of-field
?
21
The depth-of-field




?
22
The depth-of-field
decreases with d, increases with Z0
strike a balance between incoming light and
sharp depth range
?
23
Deviations from the lens model
3 assumptions
1. all rays from a point are focused onto 1 image
point
2. all image points in a single plane
3. magnification is constant deviations from
this ideal are aberrations
?
24
Aberrations
2 types
1. geometrical
2. chromatic
geometrical small for paraxial rays
study through 3rd order optics
chromatic refractive index function of
wavelength
?
25
Geometrical aberrations

• spherical aberration
• astigmatism
• distortion
• coma

aberrations are reduced by combining lenses
?
26
Spherical aberration
rays parallel to the axis do not converge outer
portions of the lens yield smaller focal lenghts
?
27
Astigmatism

• Different focal length for inclined rays

28
Distortion

• magnification/focal length different
• for different angles of inclination

pincushion (tele-photo)
barrel (wide-angle)
Can be corrected! (if parameters are know)
29
Coma

• point off the axis depicted as comet shaped blob

30
Chromatic aberration
rays of different wavelengths focused in
different planes cannot be removed
completely sometimes achromatization is achieved
for more than 2 wavelengths
?
31
Lens materials
reference wavelengths ?F 486.13nm ?d
587.56nm ?C 656.28nm lens characteristics
1. refractive index nd 2. Abbe number Vd (nd -
1) / (nF - nC) typically, both should be
high allows small components with sufficient
refraction notation e.g. glass BK7(517642) nd
1.517 and Vd 64.2
?
32
(No Transcript)
33
Darkening of images near the corner
Vignetting
The white openings in the top illustrations
denote the entrance pupil, which is the image of
the aperture stop seen through all lens elements
in front of it and from a position on the optical
axis. The bottom illustrations show the lens from
the semifield angle. Here, the white openings
correspond to the clear aperture for light that
is heading for the image corner.
Figure from http//www.vanwalree.com/optics/vignet
ting.html
34
Photographs (Niepce, La Table Servie, 1822)
Milestones Daguerreotypes (1839) Photographic
Film (Eastman,1889) Cinema (Lumière
Brothers,1895) Color Photography (Lumière
Brothers, 1908) Television (Baird, Farnsworth,
Zworykin, 1920s)
Collection Harlingue-Viollet.
CCD Devices (1970) more recently CMOS
35
Cameras
we consider 2 types
1. CCD
2. CMOS
?
36
CCD
separate photo sensor at regular positions no
scanning charge-coupled devices (CCDs) area
CCDs and linear CCDs 2 area architectures
interline transfer and frame transfer
photosensitive
storage
?
37
The CCD camera
38
CMOS

• Same sensor elements as CCD
• Each photo sensor has its own amplifier
• More noise (reduced by subtracting black image)
• Lower sensitivity (lower fill rate)
• Uses standard CMOS technology
• Allows to put other components on chip
• Smart pixels

39
CCD vs. CMOS

• Recent technology
• Standard IC technology
• Cheap
• Low power
• Less sensitive
• Per pixel amplification
• Random pixel access
• Smart pixels
• On chip integration with other components

• Mature technology
• Specific technology
• High production cost
• High power consumption
• Higher fill rate
• Blooming
• Sequential readout

40
Color cameras

• We consider 3 concepts
• Prism (with 3 sensors)
• Filter mosaic
• Filter wheel
• and X3

41
Prism color camera

• Separate light in 3 beams using dichroic prism
• Requires 3 sensors precise alignment
• Good color separation

42
Prism color camera
43
Filter mosaic

• Coat filter directly on sensor

Demosaicing (obtain full colour full resolution
image)
44
Filter wheel

• Rotate multiple filters in front of lens
• Allows more than 3 colour bands

Only suitable for static scenes
45
Prism vs. mosaic vs. wheel
Wheel 1 Good Average Low Motion 3 or more
approach sensors Separation Cost Framerate Artef
acts Bands
Prism 3 High High High Low 3 High-end cameras

Mosaic 1 Average Low High Aliasing
3 Low-end cameras
Scientific applications
46
new color CMOS sensorFoveons X3
smarter pixels
better image quality
three layers of pixels. The layers of pixels are
embedded in silicon to take advantage of the fact
that red, green, and blue light penetrate silicon
to different depths
47
Reproduced by permission, the American Society of
Photogrammetry and Remote Sensing. A.L. Nowicki,
Stereoscopy. Manual of Photogrammetry, Thompson,
Radlinski, and Speert (eds.), third edition,
1966.
The Human Eye
Helmoltzs Schematic Eye
48
The distribution of rods and cones across the
retina
Reprinted from Foundations of Vision, by B.
Wandell, Sinauer Associates, Inc., (1995). ?
1995 Sinauer Associates, Inc.
Rods and cones in the periphery
Cones in the fovea
Reprinted from Foundations of Vision, by B.
Wandell, Sinauer Associates, Inc., (1995). ?
1995 Sinauer Associates, Inc.
49
Geometric camera model
perspective projection
(Man Drawing a Lute, woodcut, 1525, Albrecht
Dürer)
50
Models for camera projection
the pinhole model revisited
center of the lens center of projection
notice the virtual image plane this is called
perspective projection
?
51
Perspective projection
Zc
u
v
Xc
Yc

• origin lies at the center of projection
• the Zc axis coincides with the optical axis
• Xc-axis ?? to image rows, Yc-axis ?? to columns

?
52
Pseudo-orthographic projection
If Z is constant ? x kX and y kY, where kf/Z
i.e. orthographic projection a scaling
Good approximation if ƒ/Z constant, i.e. if
objects are small compared to their distance
from the camera
?
53
Pictoral comparison
Pseudo - orthographic
Perspective
?
54
Projection matrices
the perspective projection model is incomplete
what if
1. 3D coordinates are specified in a world
coordinate frame
2. Image coordinates are expressed as row
and column numbers
We will not consider additional refinements, such
as radial distortions,...
?
55
?
56
Projection matrices
Image coordinates are to be expressed as pixel
coordinates
with
? (x0, y0) the pixel coordinates of the principal
point ? fx the number of pixels per unit length
horizontally ? fy the number of pixels per unit
length vertically ? s indicates the skew
typically s 0
NB3 kx,ky,s,x0 and y0 are called internal
camera parameters
NB7 fully calibrated means internally and
externally calibrated
NB4 when they are known, the camera is
internally calibrated
NB5 vector C and matrix R? SO (3) are
the external camera parameters
NB2 ky/kx is called the aspect ratio
NB1 often only integer pixel coordinates matter
NB6 when these are known, the camera
is externally calibrated
?
57
Projection matrices
Exploiting homogeneous coordinates
?
58
Projection matrices
We define
yielding
for some non-zero ?? ?
?
59
Next class Radiometry lights and surfaces