Gradient Domain Image Blending and Implementations on Mobile Devices

1
Gradient Domain Image Blending and
Implementations on Mobile Devices

• Yingen Xiong and Kari Pulli
• Nokia Research Center, Palo Alto, USA

2
Outlines

• Introduction
• Background
• What is the problem? Why we need image blending?
• Related work
• Gradient domain image blending
• Optimal seam finding
• Transition smoothing
• Implementations
• Global image blending
• Sequential image blending
• Applications and results
• Environments
• Applications
• Conclusions

3
Outlines

• Introduction
• Background
• What is the problem? Why we need image blending?
• Related work
• Gradient domain image blending
• Optimal seam finding
• Transition smoothing
• Implementation
• Global image blending
• Sequential image blending
• Applications and result analysis
• Environments
• Applications
• Conclusions

4
Introduction---Background
high-quality color displays
real-time hardware-accelerated 3D graphics
5
Mobile Panorama Imaging System
6
Outlines

• Introduction
• Background
• What is the problem? Why we need image blending?
• Related work
• Gradient domain image blending
• Graph cut optimization for optimal seam finding
• Poisson blending for transition smoothing
• Implementation
• Global image blending
• Sequential image blending
• Applications and result analysis
• Environments
• Applications
• Conclusions

7
What is the Problem? Why We Need Blending?

• A panoramic image is created by an image sequence

8
What is the Problem? Why We Need Blending?

• A simple pasting may produce visible artificial
  edges

Artifacts
9
What is the Problem? Why We Need Blending?

• Image blending can create seamless stitching

10
What is the Problem? Why We Need Blending?

• Alpha blending is not enough
• When source images are very different in colors
  and luminance.
• To make the result image look like a real one
  image after blending

11
What is the Problem? Why We Need Blending?

• Ghosting problems
• Object motion
• Alignment errors

Moving objects
Background

Ghosting problems
12
Outlines

• Introduction
• Background
• What is the problem? Why we need image blending?
• Related work
• Gradient domain image blending
• Graph cut optimization for optimal seam finding
• Poisson blending for transition smoothing
• Implementation
• Global image blending
• Sequential image blending
• Applications and result analysis
• Environments
• Applications
• Conclusions

13
Introduction---Related Work

• Two categories for image stitching
• Optimal seam finding
• Gracias et al. 2009, Milgram 1975, Efros and
  Freeman 2001, Davis 1998, Agarwala et al. 2004
• Transition smoothing
• Burt and Adelson 1983, Agarwala et al. 2004,
  Levin et al. 2004, Szeliski et al. 2008, Kazhdan
  and Hoppe 2008, Perez et al. 2003, Jia et al 2006
• Combination of optimal seam finding and
  transition smoothing

14
Outlines

• Introduction
• Background
• What is the problem? Why we need image blending?
• Related work
• Gradient domain image blending
• Optimal seam finding
• Transition smoothing
• Implementation
• Global image blending
• Sequential image blending
• Applications and result analysis
• Environments
• Applications
• Conclusions

15
Work Flow of the Gradient Domain Image Blending
16
Outlines

• Introduction
• Background
• What is the problem? Why we need image blending?
• Related work
• Gradient domain image blending
• Optimal seam finding
• Transition smoothing
• Implementation
• Global image blending
• Sequential image blending
• Applications and result analysis
• Environments
• Applications
• Conclusions

17
Optimal Seam Finding

• Graph cut optimization
• Overlap the source images
• Find optimal seams
• Cut the images along the seams
• Create labeling for all pixels
• Create a composite image

18
Optimal Seam Finding

• Graph cut optimization
• Can find optimal seams globally in the whole
  composite image
• Does not care the stitching order

19
Optimal Seam Finding

• Graph cut optimization
• Source images with different colors and luminance
• The seams and differences between source images
  are still visible
• More processing is needed for hiding the seams
  and smoothing color transition

20
Outlines

• Introduction
• Background
• What is the problem? Why we need image blending?
• Related work
• Gradient domain image blending
• Optimal seam finding
• Transition smoothing
• Implementation
• Global image blending
• Sequential image blending
• Applications and result analysis
• Environments
• Applications
• Conclusions

21
Transition Smoothing in the Gradient Domain

• Poisson blending
• Create a gradient vector field, compute a
  divergence vector field, construct a Poisson
  equation, solve the Poisson equation with
  boundary conditions, and obtain the final
  panoramic image

22
Transition Smoothing in the Gradient Domain

• Process of Poisson blending

23
Outlines

• Introduction
• Background
• What is the problem? Why we need image blending?
• Related work
• Gradient domain image blending
• Optimal seam finding
• Transition smoothing
• Implementation
• Global image blending
• Sequential image blending
• Applications and result analysis
• Environments
• Applications
• Conclusions

24
Implementation---Global Stitching
25
Implementation---Global Stitching
26
Implementation---Global Stitching

• Disadvantages for mobile applications
• Need to keep all source images in memory for
  global optimization
• Not suitable for creating high-resolution
  panoramic images on mobile devices under limited
  resources

27
Outlines

• Introduction
• Background
• What is the problem? Why we need image blending?
• Related work
• Gradient domain image blending
• Graph cut optimization for optimal seam finding
• Poisson blending for transition smoothing
• Implementation
• Global image blending
• Sequential image blending
• Applications and result analysis
• Application environments
• Applications in outdoor, indoor, and low light
  scenes
• Conclusions

28
Implementation ---Sequential Stitching

• Sequential stitching procedure
• Stitching the source images to the final
  panoramic image sequentially
• Only keep the final panorama and the current
  source image in memory

Initial panorama
29
Implementation ---Sequential Stitching
30
Implementation ---Sequential Stitching
Implementation ---Sequential Stitching
31
Outlines

• Introduction
• Background
• What is the problem? Why we need image blending?
• Related work
• Gradient domain image blending
• Graph cut optimization for optimal seam finding
• Poisson blending for transition smoothing
• Implementation
• Global image blending
• Sequential image blending
• Applications and result analysis
• Application environments
• Applications in outdoor, indoor, and low light
  scenes
• Conclusions

32
Applications and Result Analysis

• Application environment
• Implemented in a mobile panorama imaging system
• Can be run on several mobile devices
• Here shows some applications and results
• Run the approach on Nokia N95-8G mobile phones
• Size of source images is 1024x768

ARM 11 332MHz processor 128MB RAM
33
Outlines

• Introduction
• Background
• What is the problem? Why we need image blending?
• Related work
• Gradient domain image blending
• Graph cut optimization for optimal seam finding
• Poisson blending for transition smoothing
• Implementation
• Global image blending
• Sequential image blending
• Applications and result analysis
• Application environments
• Applications in outdoor, indoor, and low light
  scenes
• Conclusions

34
Applications and Result Analysis

• Applications to panorama stitching for outdoor
  scenes

Optimal seam finding 16.4 seconds transition
smoothing 54.4 seconds.
35
Applications and Result Analysis

• Applications to panorama stitching for indoor
  scenes

10 1024x768 source images captured in an indoor
scene. Optimal seam finding takes 15.92 seconds.
Transition smoothing takes 80 seconds.
36
Applications and Result Analysis

• Applications to panorama stitching for low light
  scenes

7 1024x768 source images captured in a low light
scene. Optimal seam find takes 16 seconds.
Transition smoothing takes 52.2 seconds
37
Outlines

• Introduction
• Background
• What is the problem? Why we need image blending?
• Related work
• Gradient domain image blending
• Graph cut optimization for optimal seam finding
• Poisson blending for transition smoothing
• Implementation
• Global image blending
• Sequential image blending
• Applications and result analysis
• Application environments
• Applications in outdoor, indoor, and low light
  scenes
• Conclusions

38
Conclusions

• Gradient domain image blending
• Optimal seam finding.
• Transition smoothing.
• Implementation
• Global image blending
• Sequential image blending
• Applications and results
• Outdoor scenes
• Indoor scenes
• Low light scenes
• Advantages and disadvantages
• High blending quality
• Slow and more memory
• Future work
• Speed up the approach and reduce memory
  consumption

39
More Results (Nokia N95_8GB 332 MHz 128MB RAM)
Questions?