High-Quality Video View Interpolation

1
High-Quality Video View Interpolation

• Larry Zitnick
• Interactive Visual Media Group
• Microsoft Research
• Sing Bing Kang, Matt Uyttendaele, Simon Winder,
  Rick Szeliski

2
Current practice
free viewpoint video
Many cameras
vs.
Motion Jitter
3
Current practice
free viewpoint video
Many cameras
vs.
Motion Jitter
4
Video view interpolation
Fewer cameras
and
Smooth Motion
Automatic
Real-time rendering
5
3D video
Virtualized Reality TM
Light Field
Image centric
Geometry centric
Lumigraph
Light field
Fixed geometry
View-dependent geometry
View-dependent texture
Sprites with depth
Layered depth Image
Polygon rendering texture mapping
Interpolation
Warping
6
System overview
Video Capture
Video Capture
OFFLINE
Compression
Representation
Stereo
File
ONLINE
Selective Decompression
Render
7
cameras
cameras
hard disks
controlling laptop
concentrators
8
Calibration
Zhengyou Zhang, 2000
9
Input videos
10
System overview
Video Capture
Video Capture
OFFLINE
Compression
Representation
Stereo
Stereo
File
ONLINE
Selective Decompression
Render
11
Key to view interpolation Geometry
Stereo Geometry
Image 1
Image 2
Camera 1
Camera 2
Virtual Camera
12
Image correspondence
Image 1
Image 2
Leg
Correct
Incorrect
Wall
13
Why segments?

• Better delineation of boundaries.

Tao, Sawhney, Kumar, ICCV'01
14
Why segments?

• Larger support for matching.

Handle gain and offset differences without global
model (Kim, Kolmogorov and Zabih, 2003.)
15
Why segments?

• More efficient.

786,432 pixels vs. 1000 segments Compute
disparities per segment rather than per pixel.
16
Segmentation Important properties

• Not too large, not too small
• As large as possible while not spanning multiple
  objects.

17
Segmentation Important properties

• Stable Regions

Mixed pixels
18
Segmentation

• Many methods will work
• Graph-based (Felzenszwalb and Huttenlocher, 2004)
• Mean Shift (Comaniciu, et al. 2001)
• Min-cut (Boykov et al. 2001)
• Others

19
Segmentation Our Approach

• First average

then segment.
Anisotropic smoothing
20
Segmentation Result
Close-up
21
Matching segments

• Many measures will work
• SSD
• Normalized correlation
• Mutual information
• Depends on color balancing and image quality.

22
Matching segments Important properties

• Never remove correct matches.
• Remove as many false matches as possible
• Use global methods to remove remaining false
  positives.

23
Matching segments Our approach

• Create gain histogram

0.8
1.25
Good match
0.8
1.25
Bad match
24
Local matching
Low texture
25
Global regularization

• Create MRF (Markov Random Field)

R
Q
P
S
T
U

• Number of states number of depth levels

Each segment is a node
26
Global regularization
Likelihood (data term)
Prior (regularization term)
Disparity
Images
27
Global regularization
R
Q
P
S
T
U
colorA colorB ? zA zB
28
Global regularization
A
Disparity
Bias towards similar disparity
Reduced border
Different colors
29
Multiple disparity maps

• Compute a disparity map for each image.

We want the disparity maps to be consistent
across images
30
Consistent disparities
A
R
Q
P
A
S
T
U
zA zP, zQ, zS
31
Consistent disparities
Not occluded
Occluded
32
Iteratively solve MRF
33
Depth through time (video)
34
Matting
Background Surface
Interpolated view without matting
Foreground Surface
Background
Background
Alpha
Strip Width
Foreground
Foreground
Bayesian Matting Chuang et al. 2001
Camera
35
Rendering with matting
Matting
No Matting
36
System overview
Video Capture
OFFLINE
Compression
Representation
Representation
Stereo
Stereo
File
ONLINE
Selective Decompression
Render
37
Representation
Main
Background
Boundary
Strip Width
Foreground
Main Layer
Color
Depth
38
System overview
Video Capture
OFFLINE
Compression
Representation
Compression
Representation
Stereo
File
ONLINE
Selective Decompression
Render
39
Compression

t
t1
t2
40
Temporal Compression

t
t1
t2
41
Spatial Compression

t
t1
t2
42
Spatial Compression



Camera 2
Predicted Camera 2
Camera 1
Difference
43
(No Transcript)
44
Combined Compression

t
t1
t2
45
Boundary layer coding
Color

• Depth
• Color Texture
• Alpha Matte

Alpha
Depth
Use our own shape coding method similar to MPEG-4
46
System overview
Video Capture
OFFLINE
Compression
Representation
Compression
Stereo
File
File
ONLINE
Selective Decompression
Selective Decompression
Render
Render
47
Rendering

Camera 1
Camera 2
Virtual Camera
48
Rendering the main layer (Step 1)
Depth
Color
Video of background depth
Video of background color
Projected
Color Buffer
Vertex Shader
Pixel Shader
Position,
Texture Coord
GPU
Z-Buffer
49
(No Transcript)
50
(No Transcript)
51
Rendering the main layer (Step 2)
Main Layer Depth
Projected
Color Buffer
Locate Depth Discontinuities
Pixel Shader
Generate Erase Mesh
GPU
CPU
Z-Buffer
52
Rendering boundary layer
Boundary Depth
Boundary RGBA
Projected Main Layer
Projected
Color Buffer
Vertex Colors
Compositing
Generate Boundary Mesh
GPU
CPU
Z-Buffer
53
Compositing views
Camera 1
Camera 2
Weights based on proximity to virtual viewpoint
Final composite
Final Result
Pixel Shader
GPU
54
DEMO
55
Massive Arabesque videoclip
56

• Outlook Discussion

57
Camera arrays

• Easier to capture the world then recreate it.
• Always get the right shot. Onsite filming
• Camera position
• Motion blur
• Depth of field
• Lighting
• Allow everyone to be the director.

58
Looking forward

• 3D movies
• Digital theaters are becoming more common.
• More movies will be shot with stereo camera
  pairs.

59
Looking forward

• Cameras will become less expensive.
• Actors will become more expensive.

60
Looking forward

• Combining the real world with CG.
• More information easier to add special effects.
• Some natural phenomena are hard to synthesize
  smoke, water, human motion.
• What does it mean to get the right shot?

61
Hard problems

• Complex scenes
• Reflective objects
• Translucent objects
• Plants
• Real-time manipulation
• Large spaces
• Many users
• Integrating the two worlds

62

• End