Texture%20Synthesis%20by%20Non-parametric%20Sampling%20/%20Image%20Quilting%20for%20Texture%20Synthesis%20 - PowerPoint PPT Presentation

Title: Texture%20Synthesis%20by%20Non-parametric%20Sampling%20/%20Image%20Quilting%20for%20Texture%20Synthesis%20


1
Texture Synthesis by Non-parametric Sampling /
Image Quilting for Texture Synthesis
TransferbyEfros and Leung / Efros and Freeman
ICCV 99 / SIGGRAPH 01

• Presentation by Gyozo Gidofalvi
• Computer Science and Engineering Department
• University of California, San Diego
• gyozo_at_cs.ucsd.edu
• November 15, 2001

2
Outline

• Background information on texture
• Growing texture pixel-by-pixel using a simple
  non-parametric method
• Image quilting a very simple texture synthesis
  algorithm
• Simple extension to image quilting for texture
  transfer
• Applications
• Summary

3
What is a texture?

• What features and statistics are characteristics
  of a texture pattern, so that texture pairs that
  share the same features and statistics cannot be
  told apart by pre-attentive human visual
  perception? ---- Julesz 1960s-1980s
• The concept of texture is intuitively obvious
  but has no precise definition something
  consisting of mutually related elements.
• On can describe texture by tone and structure
• Tone is based on pixel intensity properties
• Structure describes spatial relationships of
  primitives
• Texture can be described by the number and types
  of primitives and by their spatial relationships.

4
Texture Analysis Techniques

• Statistical methods gather information about
  textures by exploiting pixel first and second
  order statistics.
• Structural methods describe textures as
  composed of well defined texture primitives
  (texels), which are placed according to some
  syntactic rules.
• Model based methods construct a generative or
  stochastic model of textures, called random field
  models.
• Transform methods represent an image in a new
  form, in which the characteristics of the texture
  become more easily accessible. Some examples
  include Fourier-transforms and multi-resolution
  methods.

5
What is texture synthesis?
Given an input sample texture synthesize a
texture that is sufficiently different from the
given sample texture, yet appears perceptually to
be generated by the same underlying stochastic
process.
input image
SYNTHESIS
True (infinite) texture
generated image
6
Classification of texture

• Traditionally textures has been classified as
• regular repeated texels
• stochastic without explicit texels
• Most real world textures are mixtures of these
  basic types
• Challenge is to model the whole spectrum from
  regular to stochastic texture

regular
stochastic
both?
7
Some previous approaches

• multi-scale filter response histogram matching
  Heeger and Bergen,95
• sampling from conditional distribution over
  multiple scales DeBonet,97
• filter histograms with Gibbs sampling Zhu et
  al,98
• matching 1st and 2nd order properties of wavelet
  coefficients Simoncelli and Portilla,98

8
New method by Efros et al.

• goals
• preserve local structure
• model wide range of real textures
• ability to do constrained synthesis
• method
• inspired by N-gram language model of Shannon,
  texture is modelled as Markov Random Field (MRF)
  
• texture is grown one pixel at a time
• conditional pdf of a pixel given its neighbors
  synthesized thus far is estimated by searching
  the the sample image for similar neighborhoods

9
N-gram model of the English language

• Shannon Model language as a generalized Markov
  chain, where a set of n letters (words)
  completely determine the pdf of the next letter
  (word).
• Results (using alt.singles corpus) Mark V.
  Shaney
• "One morning I shot an elephant in my arms and
  kissed him.
• "I spent an interesting evening recently with a
  grain of salt
• Assuming Markov property, texture can be modeled
  as a MRF

10
Synthesizing one pixel
SAMPLE
p
finite sample image
Generated image
To synthesize a pixel p, search the sample image
for pixels with similar neighborhood to p,
construct a histogram for the distribution of
these pixels, finally sample this distribution to
obtain a value for p. Similarity is based on the
Gaussian-weighted sum squared difference, to
preserve local structure.
11
Growing texture on pixel at the time

• User defined window size indicates the randomness
  of the texture
• To grow from from scratch a 3x3 random seed from
  the sample is used
• Unless no close match is found pixels with most
  neighbors are synthesized first
• Importance of Gaussian-weighted similarity
  measure

12
Neighborhood window size / Randomness parameter
13
More Synthesis Results
Increasing window size
14
Results
aluminium wire
reptile skin
15
More results
French canvas
rafia weave
16
More results
wood
granite
17
More results
brick wall
white bread
18
Constrained synthesis
19
Visual comparison
Synthetic tilable texture
DeBonet, 97
Our approach
Simple tiling
20
Failure cases
Growing garbage
Verbatim copying
21
Homage to Shannon
22
Constrained text synthesis
23
What we have so far

• An algorithm that is
• simple
• models a wide range of real-world textures
• naturally well-suited for constrained texture
  synthesis
• but it
• very slow
• sometimes grows garbage
• How can it be improved?

24
Why only synthesize on pixel at the time?

• For most complex textures only a very few pixels
  actually have a choice of values
  wasted search effort
• Example Pattern of circles on the plane
• Once the algorithm starts synthesizing a
  particular circle, the values of the remaining
  pixels are completely determined.
• Unit of synthesis should be more than just a
  pixel patch
• Texture synthesis would be like jigsaw puzzle
• Questions
• What are the patches?
• How to put them together?

25
Chaos Mosaic Xu, Guo Shum, 00
input
idea
result

• Process 1) tile input image 2) pick random
  blocks and place them in random locations 3)
  Smooth edges

Used in Lapped Textures Praun et.al,00
26
Chaos Mosaic Xu, Guo Shum, 00
input
result
The approach works well on stochastic textures
but fails on structures.
27
Image Quilting
non-parametric sampling
Input image
28
block
Input texture
B1
B2
Random placement of blocks
29
Minimal error boundary
overlapping blocks
vertical boundary
30
Image Quilting algorithm

• Pick size of block and size of overlap
• Synthesize blocks in raster order
• Search input texture for block that satisfies
  overlap constraints (above and left)
• Easy to optimize using NN search Liang et.al.,
  01
• Paste new block into resulting texture
• use dynamic programming to compute minimal error
  boundary cut

31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
Comparison
Portilla Simoncelli
Xu, Guo Shum
input image
Wei Levoy
Image Quilting
39
Comparison
Portilla Simoncelli
Xu, Guo Shum
input image
Wei Levoy
Image Quilting
40
Homage to Shannon
Portilla Simoncelli
Xu, Guo Shum
input image
Wei Levoy
Image Quilting
41
Failures (Chernobyl Harvest)
42
Texture transfer

• Take the texture from one object and paint it
  onto another object
• This requires separating texture and shape
• Thats HARD, but we can cheat
• Assume we can capture shape by boundary and rough
  shading

Idea just add another constraint when sampling
similarity to underlying image at that
spot Correspondence can be based on image
intensity, blured image intensity, local image
orientation angles, etc There is a tradeoff
between the legitimacy of synthesized texture and
the correctness of the correspondence mapping.
43
parmesan


rice


44
Source texture
Target image
45


46
Applications of texture synthesis and transfer

• Occlusion fill-in
• for 3D reconstruction
• region-based image and video compression
• a small sample of textured region is stored
• Texturing non-developable objects
• growing texture directly on surface
• Motion synthesis
• Synthesizing and transferring music and
  environmental sounds?
• Rendering object in a different style without
  explicit 3D information

47
Summary

• We have seen
• A non-parametric method that grows texture
  pixel-by-pixel based on neighborhood statistics
  of pixels gathered from a sample texture
• Conceptually simple
• Spatial locality principle lends itself to
  constraint synthesis
• An image based method that where images are
  synthesized by stitching together patches of
  existing images
• A simple method based on similarity obtained by
  correspondence for texture transfer and rendering
  texture to objects that does not need explicit 3D
  information

48
Acknowledgements
Alexei Efros and his fellow researchers for
sharing their slides.
49
References and Related Literature
A. A. Efros and W. T. Freeman. Image Quilting for
Texture Synthesis and Transfer, SIGGRAPH 01. A.
A. Efros and T. K. Leung. Texture Synthesis by
Non-parametric Sampling. In ICCV 99. S. Livens.
Image Analysis for Material Characterization,
1998. http//www.ruca.ua.ac.be/visielab/livens/phd
1.ps.gz C. E. Shannon, A mathematical theory of
computation. Bell Sys. Tech. Journal, 27,
1948. M. Ashikhmin. Synthesizing natural
textures. In Symposium on Interactive 3D
Graphics, 2001. J. S. De Bonet. Multiresolution
sampling procedure for analysis and synthesis of
texture images. In SIGGRAPH 97, pages 361-368,
1997. D. D. Garber. Computational Models for
Texture Analysis and Texture Synthesis. PhD
thesis, University of Southern California, Image
Processing Institute, 1981. P. Harrison. A
non-hierarchical procedure for re-synthesis of
complex textures. In WSCG 2001 Conference
Proceedings, pages 190-197, 2001. D. J. Heeger
and J. R. Bergen. Pyramid based texture
analysis/synthesis. In SIGGRAPH 95, pages
229-238, 1995. A Hertzmann, C. E. Jacobs, D.
Oliver, B. Curless, and D. H. Salesin. Image
analogies. In SIGGRAPH 01, 2001. L. Liang , C.
Liu, Y, Xu, B. Guo, and H.-Y. Shum. Real-time
texture synthesis by patch-based sampling.
Technical Report MSR-TR-2001-40, Microsoft
Research, March 2001. Y. XU, B. Guo, and H.-Y.
Shum. Chaos Mosaic Fast and memory efficient
texture synthesis. Technical Report
MSR-TR-2000-32, Microsoft Research, April 2000.