Image Processing

1
Image Processing High Dynamic Range Display
TexPoint fonts used in EMF. Read the TexPoint
manual before you delete this box. AAAAAAA
2
Admin stuff

• Start homework early!!!!
• Will have homework 1 graded by Wednesday
• Come and see me about projects!!!!

3
Overview of today

• Linear filtering
• Blurring
• Sharpening
• Edge detection
• Wiener denoising
• Non-linear filtering
• Median filter
• Bilateral filter
• Cross-bilateral filter
• Application - High dynamic range

4
Overview of today

• Linear filtering
• Blurring
• Sharpening
• Edge detection
• Wiener denoising
• Non-linear filtering
• Median filter
• Bilateral filter
• Cross-bilateral filter
• Application - High dynamic range

5
Linear filtering

• Discrete convolution in spatial domain
• Have interpretation in frequency domain
• Might use freq. domain for speed
• Want intensity to stay same, so
• Integer arithmetic can improve speed

Filter Kernel
6
Blurring

• Used for softening appearance
• Convolve with gaussian filter
• Same as mult. by gaussian in freq. domain, so
  reduces high-frequency content
• Greater the spatial width, smaller the Fourier
  width, more blurring occurs and vice versa
• How to find blurring filter?

Slide credit Ravi Ramamoorthi
7
Blurring Filter

• In general, for symmetry f(u,v) f(u) f(v)
• You might want to have some fun with asymmetric
  filters
• We will use a Gaussian blur
• Blur width sigma depends on kernel size n
  (3,5,7,11,13,19)

Frequency
Spatial
Slide credit Ravi Ramamoorthi
8
Discrete Filtering, Normalization

• Gaussian is infinite
• In practice, finite filter of size n (much less
  energy beyond 2 sigma or 3 sigma).
• Must renormalize so entries add up to 1
• Simple practical approach
• Take smallest values as 1 to scale others, round
  to integers
• Normalize. E.g. for n 3, sigma ½

Slide credit Ravi Ramamoorthi
9
Blurring
Slide credit Ravi Ramamoorthi
10
Blurring
11
Blurring
12
Blurring
13
Blurring
Slide credit Ravi Ramamoorthi
14
Sharpening
2.0
0
Sharpened original
original
Slide credit Bill Freeman
15
Sharpening example
1.7
11.2
8
8
coefficient
-0.25
-0.3
original
Sharpened (differences are accentuated
constant areas are left untouched).
Slide credit Bill Freeman
16
Sharpening Filter

• Unlike blur, want to accentuate high frequencies
• Take differences with nearby pixels (rather than
  avg)

17
Sharpening
before
after
Slide credit Bill Freeman
18
Edge Detection
19
Edge Detection
20
Edge Detection

• Complicated topic subject of many PhD theses
• Here, we present one approach (Sobel edge
  detector)
• Step 1 Convolution with gradient (Sobel) filter
• Edges occur where image gradients are large
• Separately for horizontal and vertical directions
• Step 2 Magnitude of gradient
• Norm of horizontal and vertical gradients
• Step 3 Thresholding
• Threshold to detect edges

Slide credit Ravi Ramamoorthi
21
Details

• Step 1 Convolution with gradient (Sobel) filter
• Edges occur where image gradients are large
• Separately for horizontal and vertical directions
• Step 2 Magnitude of gradient
• Norm of horizontal and vertical gradients
• Step 3 Thresholding

Slide credit Ravi Ramamoorthi
22
Wiener denoising derivation

• See http//www.cs.dartmouth.edu/farid/tutorials/fi
  p.pdf
• Pages 57?59

23
Wiener denoising
White Gaussian noise power(assumed to be known)
From J.Portilla ICIP01
24
(No Transcript)
25
Overview of today

• Linear filtering
• Blurring
• Sharpening
• Edge detection
• Wiener denoising
• Non-linear filtering
• Median filter
• Bilateral filter
• Cross-bilateral filter
• Application - High dynamic range

26
Median filter
Replace each pixel by the median over N pixels (5
pixels, for these examples). Generalizes to
rank order filters.
Median(1 7 1 5 1) 1 Mean(1 7 1 5 1) 2.8
Spike noise is removed
In
Out
5-pixel neighborhood
Monotonic edges remain unchanged
In
Out
27
Median filtering results
Best for salt and pepper noise
http//homepages.inf.ed.ac.uk/rbf/HIPR2/mean.htmg
uidelines
28
A Gentle Introductionto Bilateral Filteringand
its Applications

• Fixing the Gaussian Blur the Bilateral Filter
• Sylvain Paris MIT CSAIL
• Fredo- Durand MIT CSAIL

29
Blur Comes from Averaging across Edges

output
input


Same Gaussian kernel everywhere.
30
Bilateral FilterNo Averaging across Edges
Aurich 95, Smith 97, Tomasi 98

output
input


The kernel shape depends on the image content.
31
Bilateral Filter Definitionan Additional Edge
Term
Same idea weighted average of pixels.
32
Illustration a 1D Image

• 1D image line of pixels
• Better visualized as a plot

pixelintensity
pixel position
33
Gaussian Blur and Bilateral Filter
Gaussian blur
p
q
space
space
Bilateral filterAurich 95, Smith 97, Tomasi 98
p
range
q
space
range
normalization
space
34
Bilateral Filter on a Height Field
output
input
reproducedfrom Durand 02
35
Space and Range Parameters

• space ss spatial extent of the kernel, size of
  the considered neighborhood.
• range sr minimum amplitude of an edge

36
Influence of Pixels
Only pixels close in space and in range are
considered.
space
range
p
37
Exploring the Parameter Space
sr ? (Gaussian blur)
sr 0.1
sr 0.25
input
ss 2
ss 6
ss 18
38
Varying the Range Parameter
sr ? (Gaussian blur)
sr 0.1
sr 0.25
input
ss 2
ss 6
ss 18
39
input
40
sr 0.1
41
sr 0.25
42
sr ?(Gaussian blur)
43
Varying the Space Parameter
sr ? (Gaussian blur)
sr 0.1
sr 0.25
input
ss 2
ss 6
ss 18
44
input
45
ss 2
46
ss 6
47
ss 18
48
How to Set the Parameters

• Depends on the application. For instance
• space parameter proportional to image size
• e.g., 2 of image diagonal
• range parameter proportional to edge amplitude
• e.g., mean or median of image gradients
• independent of resolution and exposure

49
Bilateral Filter Crosses Thin Lines

• Bilateral filter averages across features
  thinner than 2ss
• Desirable for smoothing more pixels more
  robust
• Different from diffusion that stops at thin lines

close-up
kernel
50
Bilateral Filtering Color Images
input
For gray-level images
intensity difference
scalar
output
For color images
color difference
3D vector (RGB, Lab)
51
Hard to Compute

• Nonlinear
• Complex, spatially varying kernels
• Cannot be precomputed, no FFT
• Brute-force implementation is slow gt 10min

52
Basic denoising
Noisy input
53
Basic denoising
Bilateral filter
54
Basic denoising
Bilateral filter
Median 5x5
55
Basic denoising
Bilateral filter
56
Basic denoising
Bilateral filter
Bilateral filter higher sigma
57
Denoising

• Small spatial sigma (e.g. 7x7 window)
• Adapt range sigma to noise level
• Maybe not best denoising method, but best
  simplicity/quality tradeoff
• No need for acceleration (small kernel)
• But the denoising feature in e.g. Photoshop is
  better

58
Ordinary Bilateral Filter

• Bilateral ? two kinds of weights, one image A

Image A
c
s
59
Joint or Cross Bilateral Filter

• NEW two kinds of weights, two images

B Clean,strong (Flash image)
A Noisy, dim(ambient image)
c
c
s
s
60
Image A Warm, shadows, but too Noisy(too dim
for a good quick photo)
61
Image B Cold, Shadow-free, Clean(flash simple
light, ALMOST no shadows)
62
MERGE BEST OF BOTH applyCross Bilateral or
Joint Bilateral
63
(it really is much better!)
64
Overview of today

• Linear filtering
• Blurring
• Sharpening
• Edge detection
• Wiener denoising
• Non-linear filtering
• Median filter
• Bilateral filter
• Cross-bilateral filter
• Application - High dynamic range

65
Real world dynamic range

• Eye can adapt from 10-6 to 106 cd/m2
• Often 1 10,000 in a scene

10-6
106
Real world
High dynamic range
66
Picture dynamic range

• Typically 1 20 or 150
• Black is 50x darker than white

10-6
106
Real world
10-6
106
Picture
Low contrast
67
Multiple exposure photography

• Merge multiple exposure to cover full range
• We obtain one single image with floats per pixel
• But we still cant display it

10-6
106
High dynamic range
Real world
HDR
Merge
68
HDR image using multiple exposure

• Given N photos at different exposure
• Recover a HDR color for each pixel

69
If we know the response curve

• Just look up the inverse of the response curve
• But how do we get the curve?

Pixel value
scene value
70
Calibrating the response curve

• Two basic solutions
• Vary scene luminance and see pixel values
• Assumes we control and know scene luminance
• Vary exposure and see pixel value for one scene
  luminance
• But note that we can usually not vary exposure
  more finely than by 1/3 stop
• Best of both
• Vary exposure
• Exploit the large number of pixels

71
The Algorithm
Image series
1
1
1
1
1
2
2
2
2
2
3
3
3
3
3
Dt 1/100 sec
Dt 1 sec
Dt 1/1000 sec
Dt 10 sec
Dt 1/10 sec
Pixel Value Z f(Exposure)
Exposure Radiance Dt
log Exposure log Radiance log Dt
Slide adapted from Alyosha Efros who borrowed it
from Paul Debevec? t don't really correspond to
pictures. Oh well.
72
Response curve

• Exposure is unknown, fit to find a smooth curve

Assuming unit radiance for each pixel
After adjusting radiances to obtain a smooth
response curve
3
2
Pixel value
Pixel value
1
log Exposure
log Exposure
Slide stolen from Alyosha Efros who stole it from
Paul Debevec
73
Reconstructed radiance map
Slide stolen from Fredo Durand who stole it from
Alyosha Efros who stole it from Paul Debevec
74
Problem Contrast reduction

• Match limited contrast of the medium
• Preserve details

10-6
106
High dynamic range
Real world
10-6
106
Picture
Low contrast
75
Tone mapping

• Input high-dynamic-range image
• (floating point per pixel)

76
Naïve technique

• Scene has 110,000 contrast, display has 1100
• Simplest contrast reduction?

77
Naïve Gamma compression

• X -gt Xg (where ?0.5 in our case)
• But colors are washed-out. Why?

Input
Gamma
78
Gamma compression on intensity

• Colors are OK, but details (intensity
  high-frequency) are blurred

Gamma on intensity
Intensity
Color
79
Oppenheim 1968, Chiu et al. 1993

• Reduce contrast of low-frequencies (log domain)
• Keep high frequencies

Reduce low frequency
Low-freq.
High-freq.
Color
80
The halo nightmare

• For strong edges
• Because they contain high frequency

Reduce low frequency
Low-freq.
High-freq.
Color
81
Bilateral filtering to the rescue

• Large scale bilateral (log intensity)
• Detail residual

Durand Dorsey 2002
Output
Large-scale
Detail
Color
82
Contrast reduction
Contrast too high!
83
Contrast reduction
Intensity
Color
84
Contrast reduction
Large scale
Intensity
Bilateral Filter (in log domain!)
Spatial sigma 2 image size Range sigma 0.4 (in
log 10)
Color
85
Contrast reduction
Large scale
Intensity
Detail
Bilateral Filter
Color
Detail log intensity large scale(residual)
86
Contrast reduction
Large scale
Large scale
Intensity
Reducecontrast
Detail
Bilateral Filter
Color
87
Contrast reduction
Large scale
Large scale
Intensity
Reducecontrast
Detail
Bilateral Filter
Detail
Preserve!
Color
88
Contrast reduction
Large scale
Large scale
Intensity
Reducecontrast
Detail
Bilateral Filter
Detail
Preserve!
Color
Color
89
Contrast reduction in log domain

• Set target large-scale contrast (e.g. log10 10)
• In linear output, we want 110 contrast for large
  scale
• Compute range of input large scale layer
• largeRange max(inLogLarge) min (inLogLarge)
• Scale factor k log10 (10) / largeRange
• Normalize so that the biggest value is 0 in log

outLog inLogDetail inLogLarge k
max(inLogLarge)