Omnidirectional%20Stereo%20Vision

1
Omnidirectional Stereo Vision
Capstone 2004
Lecture 18

• Zhigang Zhu
• Computer Science Department
• The City College, CUNY
• zhu_at_cs.ccny.cuny.edu
• http//www-cs.engr.ccny.cuny.edu/zhu/

2
Acknowledgements

• Collaborators at UMass
• Edward Riseman
• Allen Hanson
• Deepak Karuppiah
• Howard Schultz
• Supported by
• NSF Environmental Monitoring
• DARPA/ITO Mobile Autonomous Robot S/W
• China NSF Scene Modeling
• Paper (with references)
• http//www-cs.engr.ccny.cuny.edu/zhu/zOmniStereo
  01.pdf

3
The Class of Omnistereo(omnidirectional stereo
vision)

• Omnidirectional Vision How to look
• Viewer-centered outward looking
• Object-centered inward looking
• Omnistereo Vision How many viewpoints
• Binocular/N-Ocular a few (2 or more) fixed
• Circular Projection many inside a small area
• Dynamic Omnistereo a few, but configurable
• Object-centered many, in a large space

4
Important Issues of Omnistereo

• What this lecture is about
• Omnistereo Imaging principle for sensor designs
• Epipolar geometry for correspondence
• Depth error characterization in both direction
  and distance
• Other important issues not in this talk
• Sensor designs
• Calibration methods
• Correspondence algorithms

5
Omni Imaging Representation

• Omnidirectional (panoramic) Imaging
• Catadioptric Camera (single effective viewpoint)
• ParaVision by RemoteReality, PAL, and many
• Image Mosaicing
• Rotating camera, translating camera, arbitrary
  motion
• Omnidirectional Representation
• Cylindrical Representation
• Spherical Representation

6
Panoramic Camera
Panoramic Annular Lens (PAL) By Pal Greguss
7
Panoramic Mosaics from a Rotating Camera
(ICMCS99)
8

• 1st frame Cylindrical Panorama

• connecting frame

• conic mosaic

• head-tail stitching

• panorama

9
Cylindrical Projection
Image projection (f, v) of a 3D point P (X,Y,Z)
Distance
Cylindrical image
Vertical axis
10
Binocular / N-Ocular Omnistereo A few fixed
viewpoints

• Three configurations
• Horizontally-aligned binocular (H-Bi) omnistereo
• Vertically-aligned binocular (V-Bi) omnistereo
• N-ocular omnistereo trinocular case
• Issues
• Distance error in the direction of 360 degrees
• Distance error versus distance
• Epipolar geometry

11
H-Bi Omnistereo depth error
From Image pair (f1, v1), (f2, v2) to a 3D
point P (X,Y,Z)
Triangulation
- Fixed baseline B - Horizontal disparity
(vergent angle)
Depth Error

• Depth accuracy is non-isotropic max vergent only
  when f2 90
• Not make full use of the 360 viewing
• Depth error proportional to Depth2 / Baseline

12
H-Bi Omnistereo singularity case
Zero Vergent angle when f1f20 or 180 degree
Distance Ratio Method
- Visible Epipoles the images of the camera
centers in the others could be visible! -
Vertical disparity and vertical epipolar lines
13
H-Bi Omnistereo Epipolar geometry
Given point (f2, v2), search for (f1, v1)
-The epipolar curves are sine curves in the
non-singularity cases and - The epipolar lines
are along the v direction in the singularity cases
14
V-Bi Omnistereo
From Image pair (f1, v1), (f2, v2) to a 3D
point P (X,Y,Z)
- Vertical baseline Bv - Vertical disparity v -
Same as perspective stereo

• Depth accuracy isotropic in all directions
• - Depth error proportional to square of distance
• Epipolar lines are simply vertical lines
• - But NO stereo viewing without 3D reconstruction

15
N-Ocular Omnistereo
Why more viewpoints ?
Every point of the 360 FOV from the center of the
sensor-triangle can be covered by at least two
pairs of rays from different cameras with good
triangulations

• depth accuracy is still not isotropic, but is
  more uniform in directions
• - one pair of stereo match can be verified using
  the second pair
• - However no gain in epipolar geometry

16
Circular Projection Omnistereo Many viewpoints on
a viewing circle

• Omnivergent Stereo (Shum et al ICCV99)
• every point in the scene is imaged from two
  cameras that are vergent on that point with
  maximum vergence angle and
• stereo recovery yields isotropic depth resolution
  in all directions.
• Solution Circular Projection/ Concentric Mosiacs
• A single off-center rotating camera (Peleg CVPR
  99, Shum ICCV99)
• Full optical design (Peleg PAMI 2000)
• My catadioptric omnistereo rig

17
Circular Projection principle
Many viewpoints on a viewing circle
A virtual camera moving in a viewing circle
captures two set of rays on a plane tangent to
the viewing circle the left-eye in clockwise
direction, and the right-eye in counterclockwise
direction
18
Circular Projection geometry
Max vergent angles for left and right rays
baseline
disparity
P 3D space point r radius of the viewing
circle f1,f2 viewing directions of left and
right rays f vergent angle (angular
disparity) B baseline length (lt 2r) D
distance (OP)
19
Circular Projection properties

• Depth estimation is isotropic
• Same depth error in all directions
• Make full use of the 360 viewing
• Depth error proportional to depth2/baseline
• Same as H-Bi Omnistereo
• limited baseline (B lt 2r)
• Horizontal Epipolar lines
• Superior than H-Bi Omnistereo when a single
  viewing circle for left and right omni-images
• Extension to Concentric Mosaics with viewing
  circles of different radii?

20
Circular Projection Implementation
Cameras Single? Multiple? Standard? Special?

• Requirements Two sets of rays 180o apart
• Methods
• 1 Two Rectilinear Cameras
• 2 An Omnidirectional camera
• Question Can we do it with a single rectilinear
  camera?

21
Circular Projection Implementation (I)
Single camera approach

• Rotate a rectilinear camera off its optical
  center
• Take two columns with angular distance 2b ltlt 180o
• Viewing circle smaller than circular path of the
  optical center
• Stretching your arm out, camera viewer may be too
  far from your eyes

22
Circular Projection Implementation (2)
Catadioptric approach

• Rotate a pair of mirror with a camera around its
  optical center
• Look outward at the scene through two slit
  windows
• Larger viewing circle since mirrors enlarge the
  viewing angle
• Camera viewer right in front of your eyes

23
Dynamic Ominstereoa few viewpoints moving freely
(OmniVision2000)

• Requiements
• Optimal configuration for any given point in the
  world
• Change the vergent angle and the baseline freely
• Issues
• Dynamic Calibration
• View Planning

24
Dynamic Ominstereo depth error

• Question 1 Vergent angle
• Max vergent angle (f2 90o)
• Question 2 Baseline
• The larger the better?
• The error in estimating the baseline

25
Dynamic Ominstereo mutual calibration

• Sensors as calibration targets
• Make use of the visible epipoles
• Known target geometry
• Cylindrical body of the moving platform

26
Mutual calibration and human tracking an example
Pano 1
Image of the 2nd robot
Images of a person
Pano 2
Image of the 1st robot
Results B 180 cm, D1 359 cm, D2 208 cm
27
Dynamic Ominstereo Optimal view

• Baseline error proportional to B2
• Larger baseline, even larger error
• Overall distance error is min if
• Best baseline and max vergent angle
• Distance error with optimal configuration
  proportional to D1.5

28
Dynamic Ominstereo Optimal view application

• Track a single target by two robots
• One stationary, one moving
• Omnistereo head with reconfigurable vergent and
  baseline

29
Dynamic Ominstereo error simulation

• Student project in the spring of 2003
• Java Applet

• http//www-cs.engr.ccny.cuny.edu/zhu/omnistereo/s
  imulation/

30
Comparisons

• Four Cases
• Fixed viewpoint omnistereo
• One fixed, one circular projection
• Both circular projection
• Dynamic omnistereo
• Java Interactive Simulations
• http//www-cs.engr.ccny.cuny.edu/zhu/omnistereo/e
  rrormaps/

31
Java Interactive Simulations

• http//www-cs.engr.ccny.cuny.edu/zhu/omnistereo/e
  rrormaps/

32
Object-Centered OmniStereo

• Looking inward rather than Looking outward
• Modeling objects rather than scenes
• Many viewpoints over a large space

Modeling the Earth
33
Omni modeling of an object

• Inward-Looking Rotation
• Many viewpoints over a large circle
• Circular projection viewing circle within the
  object
• Can rotate the (small) object (e.g. human)
  instead moving the camera

34
Omni modeling of the earth

• Modeling the earth
• Airplane flying along great circles
• Taking the leading and trailing edge of each
  frame
• Data amount
• 1017 pixels if 10 cm2/pixel
• 1015 pixels if 1 m2/pixel
• 1012 1 Tera 1000 Giga
• Modeling a small area
• Rotation can be approximated as translation
• Parallel-perspective stereo mosaics
• Virtual flying through

35
Parallel-perspective stereo mosaics

• Ideal model Sensor motion is 1D translation,
  Nadir view
• Two virtual Pushbroom cameras
• Real Applications
• Airborne camera (Umass, Sarnoff..)
• Ground Vehicles (Tsinghua, Osaka)

36
Re-Organizing the images.
Stereo pair with large FOVs and adaptive
baselines
37
Re-Organizing the images.
Stereo pair with large FOVs and adaptive
baselines
38
Re-Organizing the images.
Stereo pair with large FOVs and adaptive
baselines
39
Re-Organizing the images.
Stereo pair with large FOVs and adaptive
baselines
40
Re-Organizing the images.
Stereo pair with large FOVs and adaptive
baselines
41
Re-Organizing the images.
Stereo pair with large FOVs and adaptive
baselines
42
Recovering Depth from Mosaics

• Parallel-perspective stereo mosaics
• Depth accuracy independent of depth
  (in theory)

Adaptive baseline
displacement
disparity
Fixed !
43
Stereo mosaics of Amazon rain forest

• 166-frame telephoto video sequence -gt 7056944
  mosaics

Left Mosaic
Right Mosaic
Depth Map
44
Stereo viewing

• Red Right view Blue/Green Left view

45
Accuracy of 3D from stereo mosaics(ICCV01,
VideoReg01)

• Adaptive baselines and fixed disparity -uniform
  depth resolution in theory and accuracy
  proportional to depth in practice
• 3D recovery accuracy of parallel-perspective
  stereo mosaics is comparable to that of a
  perspective stereo with an optimal baseline

46
Conclusions
Config. View-points Epipolar Geometry Error in direction Error in Distance
Binocular 2, fixed Sine curve Non-isoptric ? D2/B
Dynamic 2, free Sine curve Optimal for target ? D1.5
VCP viewer-centered Many, small circle Horizontal line isoptric ? D2 /2r
OCP object-centered Many, large circle Horizontal line isoptric ? D2/B
PPP Para-perspective Many, on a line Horizontal line uniform everywhere uniform or ?D