Boundary Matting for View Synthesis

1
Boundary Matting for View Synthesis
2nd Workshop on Image and Video Registration,
July 2, 2004
Samuel W. Hasinoff Sing Bing Kang
Richard Szeliski
Interactive Visual Media Group Microsoft
Research sbkang,szeliski_at_microsoft.com
Dept. of Computer Science University of
Toronto hasinoff_at_cs.toronto.edu
2
Motivation
Superior view synthesis 3D editing from N-view
stereo

• Two major limitations even with perfect stereo!
• Resampling blur
• Boundary artifacts

• Key approach occlusion boundaries as 3D curves
• More suitable for view synthesis
• Boundaries estimated to sub-pixel

3
Matting from Stereo

• Matting problem Unmix the foreground background

underdetermined

• Triangulation matting
• (Smith Blinn, 1996)
• multiple backgrounds
• fixed viewpoint object

B2
B3
B1
F
4
Occlusion Boundaries in 3D

• Model boundaries as 3D splines (currently linear)
• Assumptions
• boundaries are relatively sharp
• relatively large-scale objects
• no internal transparency

3D world
view 2 (reference)
view 1
view 3
5
Geometric View of Alpha

• alpha depends only on projected 3D curve, x
• integration over each pixel

F
B
pixel j
alpha ? partial pixel coverage on F side
simulate blurring by convolving with 2D Gaussian
6
Related Work

• Natural image matting Chuang et al., 2001
• based on color statistics

- single image - user-assisted

• Intelligent scissors Mortenson, 2000
• geometric view of alpha

7
Related Work

• Bayesian Layer estimation Wexler and Fitzgibbon,
  2002
• matting from multiple images using triangulation
  priors

• - requires very high-quality stereo
• alpha calculated at pixel level, only for
  reference
• not suitable for view synthesis

8
Boundary Matting Algorithm

• find occlusion boundary in reference view
• backproject to 3D using stereo depth
• project to other views
• initial guess for Bi and F
• optimize matting

3D world
optimize
view 1
view 3
view 2 (reference)
9
Initial Boundaries From Stereo

• Find depth discontinuities
• Greedily segment longest
• four-connected curves
• Spline control points evenly spaced along curve
• Tweak - snap to strongest nearby edge

10
Background Estimation

• Use stereo to grab corresponding background-depth
  pixels from nearby views (if possible)
• Color consistency check to avoid mixed pixels

B1
B2
B3
F
11
Foreground Estimation

• Invert matting equation, given 3D curve and B
• Aggregate F estimates over all views

12
Optimization

• Objective Minimize inconsistency with matting
• over curve parameters, x, and foreground
  colors, F
• Pixels with unknown B not included
• Non-linear least squares, using forward
  differencing for Jacobian

13
Additional Penalty Terms

• Favor control points at strong edges
• define potential field around each edgel

• Discourage large motions (gt2 pixels)
• helps avoid degenerate curves

14
Naïve object insertion (no matting)
15
Object insertion with Boundary Matting
16
Naïve object insertion (no matting)
Object insertion with Boundary Matting
17
Naïve object insertion (no matting)
Object insertion with Boundary Matting
boundaries calculated with subpixel accuracy
18
Samsung commercial sequence
19
Naïve object insertion (no matting)
Object insertion with Boundary Matting
20
Boundary Matting
Naïve method
21
Boundary Matting
Naïve method
22
Synthetic Noise
boundary matting
boundary matting (sigma 13)
boundary matting (sigma 26)
composite
background
no matting
23
Concluding Remarks

• Boundary Matting
• better view synthesis
• refines stereo at occlusion boundaries
• subpixel boundary estimation
• Future work
• incorporate color statistics
• extend to dynamic setting

24
Pixel-level Matting for View Synthesis?

• resampling for view synthesis can lead to
  blurring
• artifacts at boundaries.
• this example can be represented exactly using a
• sub-pixel boundary model instead