Images

1
Images

• - photometric aspects of image formation
• gray level images
• linear/nonlinear filtering
• edge detection
• corner detection

2
Image
Brightness values
I(x,y)
3
Image model
Mathematical tools
Analysis Linear algebra Numerical methods Set
theory, morphology Stochastic methods Geometry,
AI, logic
Analog intensity function Temporal/spatial
sampled function Quantization of the gray
levels Point sets Random fields List of image
features, regions
4
Basic Photometry
Radiometric model of image formation
5
Basic ingredients
Radiance amount of energy emitted along certain
direction
Iradiance amount of energy received along
certain direction
BRDF bidirectional reflectance distribution
portion of the energy coming from
direction reflected to direction
Lambertian surfaces the appearance depends
only on radiance, not
on the viewing direction
Image intensity for a Lambertian surface
6
Images

• Images contain noise sources sensor quality,
  light
• fluctuations, quantization effects

7
Image Noise Models

• Additive noise most commonly used

• Multiplicative noise

• Impulsive noise (salt and pepper)

• Noise models gaussian, uniform
• Noise Amount SNR ?s/ ?n

8
Image filtering

• How can we reduce the noise in the image
• Image acquisition noise due to light fluctuations
  and sensor noise can be reduced by acquiring a
  sequence of images and averaging them
• Computation of simple features
• First stage of visual processing

9
Image Processing
1D signal and its sampled version
f f(1), f(2), f(3), , f(n) f 0, 1, 2,
3, 4, 5, 5, 6, 10
10
Discrete time system

• Maps 1 discrete time signal to another

fx
gx
h

• Special class of systems linear ,
  time-invariant systems

Superposition principle
Shift (time) invariant shift in input causes
shift in output
11
Convolution sum
unit impluse if x 0 its 1 and zero
everywhere else
Every discrete time signal can be written as a
sum of scaled and shifted impulses
The output the linear system is related to the
input and the transfer function via convolution
Convolution sum
Notation
12
Averaging filter
Original image
Smoothed image
13
Averaging filter
and 0 everywhere else
Box filter
Ex. cont.
Averaging filter center pixel weighted more
14
Convolution in 2D
g
h
1
1
1
1
1
1
1/9
1
1
1
1/9.(10x1 11x1 10x1 9x1 10x1 11x1
10x1 9x1 10x1)
1/9.( 90) 10
15
Example
X
X
X
X
X
X
11
10
0
0
1
10
4
7
1
10
X
X
10
1
1
0
9
11
O
X
X
10
10
9
2
0
1
I
X
X
9
9
10
11
11
10
X
X
10
9
9
99
11
11
F
X
X
X
X
X
X
9
9
10
11
10
10
1
1
1
1
1
1
1/9
1
1
1
1/9.(10x1 0x1 0x1 11x1 1x1 0x1 10x1
0x1 2x1)
1/9.( 34) 3.7778
16
Example
X
X
X
X
X
X
11
10
0
0
1
10
4
7
1
10
X
X
10
1
1
0
9
11
O
X
X
10
10
9
2
0
1
I
X
X
9
9
10
11
11
10
X
X
20
10
9
9
99
11
11
X
X
X
X
X
X
9
9
10
11
10
10
1/9
1/9.(10x1 9x1 11x1 9x1 99x1 11x1 11x1
10x1 10x1)
1/9.( 180) 20
17
How big should the mask be?

• The bigger the mask,
• more neighbors contribute
• bigger noise spread.
• more blurring.
• more expensive to compute

Limitations of averaging

• Signal frequencies shared with noise are lost
• Impulsive noise is diffused but not removed

18
Frequency Domain Interpretation

• The secondary lobes of the sinc let noise into
  the filtered image.

19
Gaussian Filter

• Better option for blurring
• The coefficients are samples of a 1D Gaussian.
• Gives more weight at the central pixel and less
  weights to the neighbors.
• The further away the neighbors, the smaller the
  weight.
• Gaussian filter

Samples from the continuous Gaussian

• Gaussian filter is the only one that has the
  same shape
• in the space and frequency domains.

• There are no secondary lobes i.e. a truly
  low-pass filter

20
How big should the mask be?

• The std. dev of the Gaussian ? determines the
  amount of smoothing.
• The samples should adequately represent a
  Gaussian
• For a 98.76 of the area, we need
• m 5?
• 5.(1/?) ? 2? ? ? ? 0.796, m ?5

5-tap filter
gx 0.136, 0.6065, 1.00, 0.606, 0.136
21
Image Smoothing

• Convolution with a 2D Gaussian filter
• Gaussian filter is separable, convolution can be
  accomplished as two 1-D convolutions

22
Non-linear Filtering

• Replace each pixel with the MEDIAN value of all
  the pixels in the neighborhood
• Non-linear
• Does not spread the noise
• Can remove spike noise
• Expensive to run

23
Example
O
median
sort
10,11,10,9,10,11,10,9,10
9,9,10,10,10,10,10,11,11
24
Example
11
10
0
0
1
X
X
X
X
X
10
X
1
1
10
11
1
1
0
10
X
10
9
X
O
10
10
9
2
0
1
X
X
I
9
9
10
11
11
X
X
10
10
9
9
99
11
11
X
X
9
9
10
11
10
10
X
X
X
X
X
X
median
sort
11,10,0,10,11,1,9,10,0
0,0,1,9,10,10,10,11,11
25
Example
11
10
0
0
1
X
X
X
X
X
10
X
10
1
1
0
10
X
9
11
X
O
10
10
9
2
0
1
X
X
I
9
9
9
10
11
11
X
X
10
10
9
9
99
11
11
X
X
10
9
9
10
11
10
10
X
X
X
X
X
X
median
sort
10,9,11,9,99,11,11,10,10
9,9,10,10,10,11,11,11,99
26
Image Features

• Local, meaningful, detectable parts of the image.
• We will look at edges and corners

27
Image Features Edges, Corners

• Look for detectable, meaningful parts of the
  image
• Edges are detected at places where the image
  values exhibit sharp variation

28
Edge detection (1D)
F(x)
Edge sharp variation
x
F (x)
Large first derivative
x
29
Digital Approximation of 1st derivatives
30
Edge Detection (2D)
Vertical Edges
Horizontal Edges
31
Noise cleaning and Edge Detection
E(x,y)
I(x,y)
We need to also deal with noise -gt Combine Linear
Filters
32
Noise Smoothing Edge Detection
Convolve with
Noise Smoothing
Vertical Edge Detection
This mask is called the (vertical) Prewitt Edge
Detector Outer product of box filter 1 1 1T
and -1 0 1
33
Noise Smoothing Edge Detection
Convolve with
Horizontal Edge Detection
Noise Smoothing
This mask is called the (horizontal) Prewitt Edge
Detector
34
Sobel Edge Detector
Convolve with
Gives more weight to the center pixels
and
35
Example
36
Image Derivatives
We know better alternative to smoothing Smooth
using Gaussian filter
g(x) is a 1-D Gaussian filter, g(x,y) 2-D
Gaussian filter
Taking a derivative linear operation (take the
derivative of the filter)
37
Gaussian and its derivative
38
Vertical edges
First derivative
39
Horizontal edges
Gradient Magnitude

• Image Gradient

40
Gradient Orientation
41
Orientation histogram
42
Corner detection

• A point on a line is hard to match.

• Intuition
• Right at corner, gradient is ill defined.
• Near corner, gradient has two different values.

43
Formula for Finding Corners
We look at matrix
Gradient with respect to x, times gradient with
respect to y
Sum over a small region, the hypothetical corner
Matrix is symmetric
44
First, consider case where

• This means all gradients in neighborhood are
• (k,0) or (0, c) or (0, 0) (or
  off-diagonals cancel).
• What is region like if
• ?
• ?
• and ?
• and ?

45
General Case
In general case from linear algebra, it follows
that because C is symmetric
With R being a rotation matrix. So every case can
be intrepreted like one on last slide.
46
Corner Detection

• Filter image.
• Compute magnitude of the gradient everywhere.
• We construct C in a window.
• Compute eigenvalues l1 and l2.
• If they are both big, we have a corner
• Or if smalest eigenvalue ? of C is bigger than ?
  - mark pixel as candidate feature point

47
Point Feature ExtractionHarris Corner Detector

• Alternatively feature quality function (Harris
  Corner Detector)

48
Harris Corner Detector - Example
49
Edge Detection
gradient magnitude
original image

• Compute image derivatives
• if gradient magnitude gt ? and the value is a
  local max. along gradient
• direction pixel is an edge candidate
• how to detect one pixel thin edges ?

50
Canny Edge Detector

• The magnitude image Es has the magnitudes of the
  smoothed gradient.
• Sigma determines the amount of smoothing.
• Es has large values at edges
• Find local maxima

51
Nonmaximum supression

• The inputs are Es Eo Magnitude and
  orientation
• Consider 4 directions D 0,45,90,135 wrt x
• For each pixel (i,j) do
• Find the direction d?D s.t. d? Eo(i,j) (normal to
  the edge)
• If Es(i,j) is smaller than at least one of its
  neigh. along d
• IN(i,j)0
• Otherwise, IN(i,j) Es(i,j)
• The output is the thinned edge image IN

52
Thresholding

• Edges are found by thresholding the output of
  NONMAX_SUPRESSION
• If the threshold is too high
• Very few (none) edges
• High MISDETECTIONS, many gaps
• If the threshold is too low
• Too many (all pixels) edges
• High FALSE POSITIVES, many extra edges

53
Hysteresis Thresholding
Strong edges reinforce adjacent weak edges
54
Finding lines in an image

• Option 1
• Search for the line at every possible
  position/orientation
• What is the cost of this operation?
• Option 2
• Use a voting scheme Hough transform

55
Other edge detectors - second-order derivative
filters (1D)

• Peaks of the first-derivative of the input
  signal, correspond to zero-crossings of the
  second-derivative of the input signal.

56
Edge Detection (2D)
1D
2D
I(x)
I(x,y)
57
Notes about the Laplacian

• ?2I(x,y) is a SCALAR
• ? Can be found using a SINGLE mask
• ? Orientation information is lost
• ?2I(x,y) is the sum of SECOND-order derivatives
• But taking derivatives increases noise
• Very noise sensitive!
• It is always combined with a smoothing operation
• Filter Laplacian of Gaussian LOG filter

58
1D Gaussian
2D
Mexican Hat
59
Derivative of Gaussian Filters
Measure the image gradient and its direction at
different scales (use a pyramid).
60
The Laplacian Pyramid

• Building a Laplacian pyramid
• Create a Gaussian pyramid
• Take the difference between one Gaussian pyramid
  level and the next (before subsampling)
• Properties
• Also known as the difference-of-Gaussian
  function, which is a close approximation to the
  Laplacian
• It is a band pass filter - each level represents
  a different band of spatial frequencies
• Reconstructing the original image
• Reconstruct the Gaussian pyramid starting at top
  layer

61
Gaussian pyramid
62
Laplacian Pyramid (note top image is from
Gaussian)
63
Add more oriented filters (Malik Perona, 1990)
64
Alternative Gabor filters
Gabor filters Product of a Gaussian with sine or
cosine Top row shows anti-symmetric (or odd)
filters, bottom row the symmetric (or even)
filters. No obvious advantage to any one type of
oriented filters.
65
An edge is not a line...

• How can we detect lines ?