Light Field Completion

1
Light Field Completion

• Liron Yatziv
• Guillermo Sapiro
• Marc Levoy

http//www.ece.umn.edu/users/liron/lfc
2
Agenda

• General Background
• Goal
• Completion Method Overview
• Results

3
Light Field

• Radiance function on rays
• Can be represented with a 4D function
• Position in 3D space
• Direction

4
Completion

• Image completion
• Video completion
• Light field completion

5
Light Field Completion
6
Light Field Completion
7
Light field Representation1
t
v
L(u,v,s,t)
s
u
1 M. Levoy and Pat Hanrahan. Light Field
Rendering SIGGRAPH 1996
8
Light field Representation
t
v
s
u
9
4D Light Field Visualization1
Source Stanford Computer Graphics Laboratory
10
Light Field Completion - overview
Registration
input light field
registered light field
Median
Texture Synthesis
median image
output light field
11
Priority Based Texture Synthesis10

• Confidence
• Data term

front
?p patch centered at p
np unit vector orthogonal to the front
10 Criminisi, Perez and Toyama Object Removal
by Exemplar-Based Inpainting
12
Priority Based Texture Synthesis for a Light Field

• Extension of the priority based texture synthesis
  to 4D.
• Set the initial condition to be estimated data
  given from the median image.

C1
p
C0
C0
0ltClt1
13
Priority Based Texture Synthesis for a Light
Field (cont)

• Changes for handling partial neighborhoods
• New distance function that can handle partial
  neighborhoods.
• Averaging of best matching patches.

14
Computational Considerations

• The texture synthesis uses exhaustive search.
• Limit search to relevant areas.

15
Results
Removing the pillars
16
Results
Removing the fence
17
Results
18
References

• M. Levoy and Pat Hanrahan. Light field
  rendering, In Proc. SIGGRAPH 1996, pages 31-42,
  August 1996.
• S. Gortler, R. Grzeszczuk, R. Szeliski, and M.
  Cohen, The Lumigraph, In Proc. SIGGRAPH 1996,
  August 1996.
• A. Efros and T. Leung. Texture synthesis by
  non-parametric sampling. In Proc. ICCV, pp.
  10331038, Kerkyra, Greece, Sep 1999.
• A. Efros and W. Freeman. Image quilting for
  texture synthesis and transfer, In Proc.
  SIGGRAPH 01, Los Angeles California, August,
  2001.
• L. Wei and M. Levoy. Fast texture synthesis
  using tree-structured vector quantization, In
  Proc. SIGGRAPH 2000, pp. 479-488, July 2000.
• M. Bertalmio, G. Sapiro, V. Caselles, and C.
  Ballester. Image inpainting, In Proc. SIGGRAPH
  2000, New Orleans, July 2000.
• M. Irani and S. Peleg, Motion analysis for image
  enhancement Resolution, occlusion, and
  transparency , Journal of Visual Communication
  and Image Representation, Vol. 4, No. 4, pp.
  324-335, December 1993.
• A. Duci, A. Yezzi, S. Mitter, and S. Soatto,
  Region matching with missing parts, Proc. of
  the ECCV, LNCS, Springer Verlag, May 2002.
• K. Habuka and Y. Shinagawa, 4D light field
  interpolation using a mechanism to control
  optical flow, preprint, 2004.
• A. Criminisi, P. Perez, and K. Toyama. Object
  removal by exemplar-based inpainting, IEEE CVPR
  2003, Madison, Wisconsin, US, June 2003.
• I. Drori, D. Cohen-Or, and H. Yeshurun.
  Fragment-based image completion, SIGGRAPH 2003,
  San Diego, July 2003.
• M. Bertalmio, L. Vese, G. Sapiro, and S. Osher,
  Simultaneous structure and texture image
  inpainting, IEEE Trans. Image Processing 12, pp.
  882-889, 2003.
• T. Porter and T. Duff, Compositing digital
  images, Proc. SIGGRAPH 1984, pp. 253-259, 1984.

http//www.ece.umn.edu/users/liron/lfc