Feature Based Image Mosaicing

1
Feature Based Image Mosaicing

• Satya Prakash Mallick

2
Introduction

• Mosaicing methods can be classified broadly into
• Direct Method
• Uses information from all pixels.
• Iteratively updates an estimate of homography so
  that a particular cost function is minimized.
• Sometimes Phase-Correlation is used to estimate
  the a few parameters of the homography.
• Feature Based Method
• A few corresponding points are selected on the
  two images and homography is estimated using
  these reliable points only.

3
My Choice?

• Feature Based Method Because
• They are in general more accurate.
• Can Handle large disparities.
• Convergence Direct methods, may not converge to
  the optimal solution is the presence of local
  minima.
• For reliable performance direct methods rely on
  feature based initialization.

4
Problems in Image Mosaicing

1. Global Alignment essentially means recovery of
   underlying homography.
2. Local Alignment Correct the local mismatches
   left after global alignment.
3. Image Blending a decision has to be made as to
   what color the overlapping region should take.
4. Image Warping After calculation of homography, a
   decision has to be made as to how to warp one
   image w.r.t the other.

5
Problems in Image Mosaicing

• Automatic selection of images to blend.
• Auto-exposure compensation
• Camera error compensation.
• I am looking at the first four problems.

6
The Algorithm ( Overview )

1. Detect corners in both the images
2. Solve for correspondence.
3. Use RANSAC to estimate homography.
4. Refine the estimate of homography using a
   non-linear method.
5. Make a decision on how to blend the images.
6. Warp the one image with respect to the other
   taking the blending decision into consideration

7
Solving for correspondences

• Corners were detected in both the images using
  Harris corner detector.
• Correspondences are solved using a version of
  Zhangs relaxation algorithm
• Matching Through Correlation
• Disambiguating Matches Through Relaxation

8
Solving for correspondences

• Matching through correlation

9
Solving for correspondences

• Disambiguating matches through relaxation
• Show correspondence results

10
Estimation of homography

• RANSAC was implemented for estimating the
  homography relating the two scenes. Results show
  a comparison between using RANSAC and Least
  squares.
• The error function being minimized is

11
Image Blending

• Weighted Image Blending

12
Image Blending

• Lightest Path Cut The idea is to take the
  difference of the images in the overlapping
  region and cut the region along that curve of
  minimum intensity. The implementation was done
  using dynamic programming.
• Note Optimal implementation of the algorithm is
  non-trivial and can be a good problem to look
  into.

13
Results of Blending

• Try to find the curve along which the image
  overlap was cut! ALL THE BEST!

14
Homography Refinement

• In the Weighted image blending example we saw,
  there was some blurring at the edges of the
  overlap.
• The obvious question is
• Are we sure that the homography estimate is
  good enough or can it be further refined?

15
Homography Refinement

• The Homography was refined using Newtons
  non-linear optimization technique. Its safe to
  use Newtons method because we are already close
  to the solution.
• The vector X shown is made of inliers got using
  RANSAC

16
Homography Refinement

• Results The blurring near the edge of
  overlapping region is almost gone.

17
Limitations and Mistakes

• My biggest mistake I assumed that, to make a
  mosaic with many images, it is sufficient to keep
  on adding images to the main mosaic. The mosaic
  doesnt do well, if we try to blend more than
  four images. So there should have been a global
  refinement on different homographies

18
Conclusions

• Lessons to be learnt
• Non-Linear optimization of homography is not an
  overkill. After RANSAC, it is usually very fast (
  Most of the time, my results converged after 2-3
  iterations only)
• Its a good idea to blend images along the
  lightest curve along the difference image.
  However, the image quilting algorithm cannot be
  directly used in that.
• To blend a large number of images, there is
  something more to be done than finding
  pair-wise homography of images.

19
Conclusion

• Image warping should always be done backward with
  bilinear interpolation.

20
References

• Hartley Hartley, R. Zisserman, A. (2000)
  Multiple View GeometryCambridge University
  Press, UK.
• Shum Shum, H. Szeliski, R. (1998)
  Construction and refinement of panoramic mosaics
  with global and local alignment. IEEE Int'l Conf.
  Computer Vision, pp. 953-958.
• Faugeras Zoghlami, I. Faugeras,O.
  Deriche,R. (1997) Using geometric corners to
  build a 2d mosaic from as set of images.Computer
  Vision and Pattern Recognition, pp 421-425.
• Zhang Zhang, Z. Deriche, R. Faugeras, O
  Luong, Q. A robust technique for matching two
  uncalibrated images through the recovery of the
  unknown epipolar geometry (1995) Artificial
  Intelligence Journal, 7887-119, October 1995
• Harris Harris, C. Stephens, M. A combined
  corner and edge detector.(1998) Proc. of 4th
  Alvey Vision Conf.,147-151.
• Szeliski Szeliski, R. Image Mosaicing for
  Tele-Reality Applications.(1994). Digital
  Equipment Corporation, Cambridge, USA.
• Davis Davis, J. Mosaics of scenes with moving
  objects.(1998).Computer Vision and Pattern
  Recognition
• Capel Capel,D Zisserman,A. Automated
  mosaicing with super-resolutionzoom.(1998).Compute
  r Vision and Pattern Recognition

21

• THANKS !