Defocus Difference Matting

1
Defocus Difference Matting

• Wojciech Matusik

Morgan McGuire
Brown University Graphics Group
Mitsubishi Electric Research Laboratory
2
Studio Matting Problem

• Remove background
• Film Television
• Studio environment
• Controlled set
• Controlled lighting
• Sub-pixel precision (matte)
• Need perfect results
• Hardest test case fine hair structure

3
Blue-Screen Matting

• Benefits
• De facto standard
• Controlled background acceptable in studio
• Very fast
• Drawbacks
• Foreground cannot contain blue
• Reflected blue light
• Fresnel effect--blue borders

4
Related Work

• Difference Matting Qian and Sezan 99
• Pre-recorded static background
• Triangulation Matting Smith and Blinn 96
• Two different static backgrounds
• Only works for images
• Defocus Video Matting McGuire et al. 05
• Arbitrary unknown dynamic background
• Defocus allows cameras to simultaneously see same
  foreground, different background

5
Wide-Field Background
6
Narrow-Field Background
7
Wide-Field Image
8
Narrow-Field Image
9
Result
10
New Background
11
Outline

• Camera design
• Solving compositing equation for ?
• Robustness to ill-conditioning
• Background design

12
Beam-Splitters
10
90
13
Time-Multiplexing

• Aperture that changes from f /12 to f /1.6 at 120
  Hz
• Not possible with physical iris
• Momentum!
• Not possible with LCD
• Not enough contrast (110, need 1500)
• Cant refresh fast enough
• Solution ferro-electric liquid crystal and
  polarizers

14
Time-Multiplexing
V. Polarizer
H. Polarizer
FELC OFF
Imager
0.6 - 10 kHz Strobe
X
Pinhole Aperture
X
V
15
Time-Multiplexing
V. Polarizer
H. Polarizer
FELC ON
Imager
X
Pinhole Aperture
V
16
Time-Multiplexing
V. Polarizer
H. Polarizer
FELC ON
Imager
X
Pinhole Aperture
V
17
Design Tradeoffs
Calibration
Size
Motion
Beam-splitter
Hard
2x
Perfect
Time-multiplexing
None
Normal Lens
Requires optical flow
18
Algorithm
19
Defocus Composite
Wide-field Composite
Narrow-field Composite

• h is aperture mask (blur kernel)

20
Solve for ?
Known values
21
Solution
22
Solution

• Ill-conditioned when denominator is zero
• Interpolate at these pixels from neighbors

23
Solution
24
Solution

• Sums and differences of known images
• Very fast to compute

25
Background Design
26
Background Design

• Average color white
• Avoid blue-spill
• Red green out of phase
• Minimize ill-conditioning
• Green blue in phase
• Cameras much more sensitive to red

27
Results
28
Narrow-Field Image
29
Result
30
Narrow-Field Image
31
Result
32
New Background
33
Wide-Field Image
34
Result
35
Future Work

• Real-time implementation
• Video
• Camera design
• Time multiplexing
• Beam-splitters
• Automatic calibration

36
(No Transcript)