Temporally Consistent Disparity Maps from Uncalibrated Stereo Videos

1
Temporally Consistent Disparity Maps from
Uncalibrated Stereo Videos

• Michael Bleyer and Margrit Gelautz
• Institute for Software Technology and Interactive
  Systems
• Vienna University of Technology, Austria
• International Symposium on Image and Signal
  Processing and Analysis (ISPA) 2009

2
Autostereoscopic Displays

• Allow good-quality 3D viewing without glasses
• Novel viewpoint functionality
• Trend towards 3D cinema
• Currently hindered by the difficulty of content
  production

( )

• Philips WoWvx autostereoscopic display

3
2D Depth Format

• 2D Depth

• Easy to generate for artificial content
• How can we get this information for real movies?

4
Depth Map via Dense Stereo Matching

• (Left Image)

(Right Image)
(Disparity Map)
5
Depth Map via Dense Stereo Matching
Our task Convert a sequence of stereo pairs into
the 2D plus depth format

• (Left Image)

(Right Image)
(Disparity Map)
6
Related Work

• Commercial side
• Philips Blue Box
• Research side
• Tons of papers on stereo matching Scharstein,
  IJCV02
• Very little work on temporal stereo
• Novel-viewpoint system Zitnick, SIGRAPH04
• BP-based temporal stereo Larsen, ICCV07
• Stequel matching Sizintsev, CVPR09

7
Overview of our Method
Input
1. Scene Segmentation
2. Rectification
3. Stereo Matching
4. Temporal Smoothing
Output
8
Overview of our Method
Input
1. Scene Segmentation
2. Rectification
3. Stereo Matching
4. Temporal Smoothing
Output
9
Overview of our Method
Input
1. Scene Segmentation
2. Rectification
3. Stereo Matching
4. Temporal Smoothing
Output
10
Overview of our Method
Input
1. Scene Segmentation
2. Rectification
3. Stereo Matching
4. Temporal Smoothing
Output
11
Overview of our Method
Input
1. Scene Segmentation
2. Rectification
3. Stereo Matching
4. Temporal Smoothing
Output
12
Scene Segmentation (1)

• User specifies shot boundaries
• Why doing scene segmentation?
• Rectification
• Calibration parameters assumed to be constant
  within a scene
• Stereo matching
• Setting of disparity range
• Temporal smoothing
• Do not smooth over shot boundaries

13
Scene Segmentation (2)
(User interface)
14
Rectification

• Corresponding pixels need to lie on the same
  scanline
• Practically, never fulfilled for uncalibrated
  stereo cameras
• Uncalibrated Rectification
• Compute a sparse set of correspondences (SURF
  features)
• Input for rectification method of Fusiello,
  ICPR08

15
Stereo Matching - Method

• Dynamic Programming (DP) stereo matcher Bleyer
  and Gelautz, VISSAP08
• Applies DP on tree structures
• State-of-the-art results
• Very fast (ltlt 1 sec)
• Modifications
• Radiometric distortions
• Sub-pixel accuracy

(Tree structures)
16
Stereo Matching User Interface (1)
(Defining the disparity range)
17
Stereo Matching User Interface (2)
(Tuning the stereo parameters)
18
Temporal Smoothing

• Disparity flickering problem
• Median filtering along optical flow vectors

19
Temporal Smoothing

• Disparity flickering problem
• Median filtering along optical flow vectors

Frame i
Pixel pi
20
Temporal Smoothing

• Disparity flickering problem
• Median filtering along optical flow vectors

Frame i
Frame i-1
Pixel pi
Pixel pi-1

• Step 1 Corresponding pixel - previous frame
  (optical flow)

21
Temporal Smoothing

• Disparity flickering problem
• Median filtering along optical flow vectors

Frame i
Frame i1
Pixel pi
Pixel pi1

• Step 2 Corresponding pixel - next frame
  (optical flow)

22
Temporal Smoothing

• Disparity flickering problem
• Median filtering along optical flow vectors

Frame i
Frame i1
Frame i-1
Pixel pi
Pixel pi1
Pixel pi-1

• Step 3 Record disparity values for all 3 pixels
• Step 4 Median of disparity values

23
User Interface

• All operations can be accessed via an easy-to-use
  GUI

24
Results (1)

• 2 test sequences
• Length of 5 minutes
• 720x288 / 720x576 pixels per image
• Computation time
• 1 second per frame
• Quality of results
• Good-quality disparity results (disparity
  borders, untextured regions)
• Temporally consistent
• Look accurate on the Philips WoWvx display

25
Results (2)

• Videos
• Temporal smoothing
• Cave Exploration

(Left image)
(Without smoothing)
(With smoothing)
26
Conclusions

• Method for generating temporal-smooth disparity
  sequences
• Good quality of results
• Fast computation time
• Easy to apply also for technically unskilled
  users (GUI)