A Stereo Approach that Handles the Matting Problem via Image Warping

1
A Stereo Approach that Handles the Matting
Problem via Image Warping

• Michael Bleyer1, Margrit Gelautz1, Carsten
  Rother2, Christoph Rhemann1
• 1Vienna University of Technology, Austria
• 2Microsoft Research Cambridge, UK
• IEEE Conference on Computer Vision and Pattern
  Recognition (CVPR) 2009

2
Synergies between Stereo and Matting
Left Camera
Right Camera
(Right Camera)
3
Synergies between Stereo and Matting
Left Camera
Right Camera
Good Match (Colors consistent)
(Right Camera)
4
Synergies between Stereo and Matting
Left Camera
Right Camera
Transparencies due to Lense Blur, Discretization,
etc.
(Right Camera)
(Right Camera)
5
Synergies between Stereo and Matting
Left Camera
Right Camera
Mixed Color m ap cp aq cq
(Right Camera)
(Right Camera)
6
Synergies between Stereo and Matting
Left Camera
Right Camera
(Right Camera)
7
Synergies between Stereo and Matting

• Pixel mattes lead to color inconsistencies near
  disparity borders
• Overcome this problem
• Solve stereo and matting problems
  simultaneously
• Disparity information provides two instead of
  one mixed colors for computing alpha gt Less
  ambiguity

Left Camera
Right Camera
(Right Camera)
8
Previous Work on Stereo Matting

• Baker et al., CVPR98, Szeliski and Golland,
  ICCV98
• Early work on stereo matting
• Zitnick et al., SIGRAPH04
• Matting in postprocessing step
• Hasinoff et al., CVIU06
• 3D curve fitting to precomputed disparity borders
• Xiong and Jia, CVPR07
• Exploits synergies
• Does not work for more than two depth layers
• Taguchi et al., CVPR08
• Works for multiple depth layers
• Does not exploit problem synergies

9
Contributions

• Combined stereo and matting approach
• Can handle multiple disparity layers
• Still exploits problem synergies
• New assumption of constant solidity

10
Combined Stereo and Matting Approach
11
Overlapping segments

• Color segmentation of left image
  (oversegmentation)

(Left Image)
(Segment Boundaries)
12
Overlapping segments

• Color segmentation of left image
  (oversegmentation)

For each segment S
(Left Image)
(Segment Boundaries)
13
Overlapping segments
Morphological Dilation

• Color segmentation of left image
  (oversegmentation)

For each segment S
(Left Image)
(Segment Boundaries)
14
Overlapping segments
Transparent Region

• Color segmentation of left image
  (oversegmentation)

For each segment S
(Left Image)
(Segment Boundaries)
15
Overlapping segments
Non-Transparent Region

• Color segmentation of left image
  (oversegmentation)

For each segment S
(Left Image)
(Segment Boundaries)
16
Overlapping segments
Overlapping Segment

• Color segmentation of left image
  (oversegmentation)

For each segment S
(Left Image)
(Segment Boundaries)
17
Overlapping segments
Overlapping Segment

• Color segmentation of left image
  (oversegmentation)

• Find the following parameters via energy
  minimization
• Disparity plane (for each overlapping segment)
• Alpha value (for each pixel)
• True Color (for each pixel)

For each segment S
(Left Image)
(Segment Boundaries)
18
How to measure the Goodness of Alphas, True
Colors and Disparities?
(Left Buffer)
19
How to measure the Goodness of Alphas, True
Colors and Disparities?
True Color (Blue), Alpha (0.4)
(Left Buffer)
20
How to measure the Goodness of Alphas, True
Colors and Disparities?
Disparity
(Left Buffer)
21
How to measure the Goodness of Alphas, True
Colors and Disparities?
X-Coordinate
(Left Buffer)
22
How to measure the Goodness of Alphas, True
Colors and Disparities?
Cell
(Left Buffer)
23
How to measure the Goodness of Alphas, True
Colors and Disparities?
(Artificial Left Image)
Color Composition
(Left Buffer)
24
How to measure the Goodness of Alphas, True
Colors and Disparities?
Very similar if Alphas and True Colors are
correct
(Real Left Image)
(Artificial Left Image)
Color Composition
(Left Buffer)
25
How to measure the Goodness of Alphas, True
Colors and Disparities?
Image Warping
(Left Buffer)
26
How to measure the Goodness of Alphas, True
Colors and Disparities?
Image Warping
(Left Buffer)
(Right Buffer)
27
How to measure the Goodness of Alphas, True
Colors and Disparities?
(Artificial Right Image)
Image Warping
Color Composition
(Left Buffer)
(Right Buffer)
28
How to measure the Goodness of Alphas, True
Colors and Disparities?
Very similar if Alphas, True Colors and
Disparities are correct
(Real Right Image)
(Artificial Right Image)
Image Warping
Color Composition
(Left Buffer)
(Right Buffer)
29
How to measure the Goodness of Alphas, True
Colors and Disparities?
Most critical questionHow are alpha-values
affected by image warping?
Very similar if Alphas, Colors and Disparities
have been computed correctly
(Real Right Image)
(Artificial Right Image)
Image Warping
Color Composition
(Left Buffer)
(Right Buffer)
30
How is Alpha affected by Image Warping?

• Assumption of Xiong and Jia, CVPR 2007
• Alpha remains constant for foreground pixels
• Problems
• No information about background pixels (also need
  to be warped)
• Not necessarily true if more than two layers
• Our assumption
• Solidity of a pixel remains constant
• More powerful
• Holds for all pixels
• Holds in the n-layer case

31
What is Solidity?

• Solidity of pixel p is the percentage to which p
  occludes pixels of lower disparities.
• Solidity op is computed by
• with ap and dp being the alpha value and
  disparity of pixel p.

32
What is Solidity?

• Solidity of pixel p is the percentage to which p
  occludes pixels of lower disparities.
• Solidity op is computed by
• with ap and dp being the alpha value and
  disparity of pixel p.

33
What is Solidity?

• Solidity of pixel p is the percentage to which p
  occludes pixels of lower disparities.
• Solidity op is computed by
• with ap and dp being the alpha value and
  disparity of pixel p.

34
What is Solidity?

• Solidity of pixel p is the percentage to which p
  occludes pixels of lower disparities.
• Solidity op is computed by
• with ap and dp being the alpha value and
  disparity of pixel p.

35
What is Solidity?

• Solidity of pixel p is the percentage to which p
  occludes pixels of lower disparities.
• Solidity op is computed by
• with ap and dp being the alpha value and
  disparity of pixel p.

36
Warping with Transparent Pixels (Example 1)
Left Buffer
Right Buffer
37
Warping with Transparent Pixels (Example 1)
Left Buffer
Right Buffer
38
Warping with Transparent Pixels (Example 1)
Left Buffer
Right Buffer
39
Warping with Transparent Pixels (Example 1)
Left Buffer
Right Buffer
40
Warping with Transparent Pixels (Example 1)
Left Buffer
Right Buffer
41
Warping with Transparent Pixels (Example 1)
Left Buffer
Right Buffer
42
Warping with Transparent Pixels (Example 1)
Left Buffer
Right Buffer
Alpha is different across views
43
Warping with Transparent Pixels (Example 1)
Left Buffer
Right Buffer
Assumption of constant foreground alpha violated
44
Warping with Transparent Pixels (Example 2)
Left Buffer
Right Buffer
Occlusions
Occlusion if ?a lt 1 in a cell of the right buffer
45
Energy Function

• For each pixel of the left buffer, find a true
  color, alpha value and disparity so that energy E
  is minimized.
• Data terms of E
• Color difference between artificial left and real
  left images
• Color difference between artificial right and
  real right images
• Prior Knowledge in E
• Infinite penalty if ?a ? 1 (left buffer)
• Penalize neighboring pixels of different alphas
  (Linear smoothness term)
• Penalize neighboring segments carrying different
  disparity planes (Potts model)
• See paper for Optimization Strategy

46
Results
47
Computed Disparity Maps
(Tsukuba)
(Venus)
(Teddy)
(Cones)
48
Disparity Errors gt 1 Pixel
(Tsukuba)
(Venus)
(Teddy)
(Cones)
49
Assignment of Pixels to Disparity Planes
(Tsukuba)
(Venus)
(Teddy)
(Cones)
50
Assignment of Pixels to Disparity Planes
(Tsukuba)
(Venus)
(Teddy)
(Cones)
51
Assignment of Pixels to Disparity Planes
(Tsukuba)
(Venus)
(Teddy)
(Cones)
52
Artificial Right Views
(Tsukuba)
(Venus)
(Teddy)
(Cones)
53
Artificial Right Views
(Tsukuba)
(Venus)
(Teddy)
(Cones)
54
Quantitative Results Middlebury Ranking
Our method takes the 6th rank of 60 submissions
55
Quantitative Results Middlebury Ranking
Our method takes the 6th rank of 60 submissions
56
Quantitative Results Middlebury Ranking
Our method takes the 6th rank of 60 submissions
57
Application Example Novel Viewpoint Generation
(Zoomed-in view)
(Result without using matting information)
(Novel view generated using our matting and
disparity results)
58
Application Example Novel Viewpoint Generation
(Zoomed-in view)
(Result without using matting information)
(Novel view generated using our matting and
disparity results)
59
Application Example Depth Segmentation
(Segmented objects pasted against a white
background)
60
Conclusions

• Combined stereo and matting approach takes
  advantage of problem synergies
• Proposed the assumption of constant solidity
• Good-quality disparity results
• Matting results look visually satisfying

61
Energy Optimization

• Two step procedure
• Optimize disparity planes (fixed alphas and true
  colors)
• Greedy search strategy
• Optimize alphas and true colors (fixed disparity
  planes)
• Belief Propagation (most similar Wang, ICCV05,
  Wang, CVPR07)
• Iterate a few times