Texture Synthesis

1
TextureSynthesis

• Aude Oliva
• CVC Seminar
• Fall 2008

2
Papers

• Heeger Bergen 1995
• Portilla Simoncelli, 2001

3
What is a texture?

• Definition 1 A spatially homogeneous pattern
  that typically contains repeated structures,
  often with some random variations.
• Definition 2 a set of elements at periodic or
  random locations within an image
• Definition 3 a visual pattern which has a
  stationary distribution of features the
  texture elements (e.g. textels) is repeated
  everywhere in the image.

4
What is a texture?

• Two categories of textures
• 1) determinist or regular textures determined
  by a set of primitives and a placement rule (e.g.
  a tile floor). Those are determined by repeated
  elements or groups of elements.
• 2) stochastic textures does not have easily
  identifiable primitives (e.g. granite, sand).
• Natural textures are characterized by both
  properties

5
Why is Texture important ?

• All surfaces have texture
• In real world, texture imposes the structure and
  give form to separate regions.
• Texture is implicated in the process of
  segregating the visual scene into discrete
  regions (e.g. separating the sidewalk from the
  grass, a low level analysis) and make basic
  decision about the nature of the regions (high
  level analysis)
• Affordance when walking across uneven outdoor
  terrain identify surface that affords good
  footing (the tilt of some surface indicates that
  some surfaces should not be stepped on)

Camouflage property An animal (e.g. snake) with
a texture similar to those of surrounding object
to provide camouflage. Camouflage is the process
of effectively imitating texture (cf. texture
synthesis).
6
Why is Texture important ? Texture Gradient

• Texture gradient (Gibson, 1950) In addition to
  providing information about depth, texture
  gradient also inform observers about the
  orientation of a surface in depth and about its
  curvature.
• Complex surface shape can be realistically
  depicted by textural variations

• Natural texture gradients many natural surfaces
  are defined by texture elements of about the same
  size and shape so that surface depth and
  orientation can be recovered from systematic
  changes in their projected sizes and shapes
  (Gibson, 1950)
• Artificial texture gradients. Artificial surfaces
  of arbitrarily complex shapes can be rendered by
  using identical texture elements in computer
  graphics displays (Marr, 1982)

7
Texture gradient
8
Texture synthesis

• Camouflage property An animal (e.g. snake) with
  a texture similar to those of surrounding object
  to provide camouflage. Camouflage is the process
  of effectively imitating texture (cf. texture
  synthesis).
• How can we imitate a texture?

9
Representing textures

• What filters?
• experience suggests spots and oriented bars at a
  variety of different scales
• What statistics?
• within reason, the more the merrier.
• At least, mean and standard deviation
• better, various conditional histograms.

• Textures are made up of quite stylized
  subelements, repeated in meaningful ways
• Representation
• find the sub-elements (textons, textels, etc),
  and represent their statistics
• But what are the sub-elements, and how do we find
  them?
• Find sub-elements by applying filters, looking at
  the magnitude of the response

10
Questions about Textures

• (1) How can we represent the information
  contained in a texture ?
• (2) How can we synthesis, imitate, replicate, a
  texture so that it looks like the original ?
  (Notion of camouflage)
• (3) Which texture information can be used by
  human observers (and a model) to classify
  together similar textures ?

11
Texture Synthesis Approach

• Philosophy To capture samples of the real world
  as images and use them to synthesize novel views
  rather than recreating the entire physical world
  from scratch.
• Image-based texture synthesis algorithms such
  an algorithm should be able to take a sample of
  texture and generate an unlimited amount of image
  data which, while not exactly like the original,
  will be perceived by humans to be the same
  texture.
• The origin D. J. Heeger and J. R. Bergen (1995).
  Pyramid Based Texture Analysis/Synthesis.,
  Computer Graphics Proceedings. p. 229-238.
• The approach is based on a model of human
  texture perception, and has potential to be a
  practically useful tool for graphics applications

12
Which textures are we going to talk about?
Stationary Stochastic
13
When are two textures similar?
All these images are different instances of the
same texture We can differentiate between them,
but they seem generated by the same process
14
The trivial texture synthesis algorithm
?
?
15
Texture synthesis and representation
?
Set of equivalent textures
?
?
Space of all images
Set of equivalent textures generated by exactly
the same physical process
16
Texture synthesis and representation
?
Set of equivalent textures
?
Set of perceptually equivalent textures
?
Space of all images
Set of equivalent textures generated by exactly
the same physical process
Set of perceptually equivalent textures well,
they just look the same to me
17
The main idea it works by kind of projecting a
random image into the set of equivalent textures
Set of equivalent textures
Set of perceptually equivalent textures
Space of all images
18
Overview of the algorithm
Two main tools 1- steerable pyramid 2- matching
histograms
19
Steerable Pyramid
Freeman Adelson, 91 Freeman Simoncelli, 95
20
1-The steerable pyramid
Decomposition
subband


Simoncelli, Freeman, Adelson
21
1-The steerable pyramid
Decomposition
Reconstruction
Low-pass residual
22
1-The steerable pyramid
Representation in pixel space
Alternative representation
Low-pass residual
23
1-The steerable pyramid
But why do I want to represent images like this?
24
1-The steerable pyramid
Argument used by Hegger Bergen Statistical
measures in the subband representation seem to
provide a distance between textures that
correlates with human perception better than
pixel-based representations.
25
1-The steerable pyramid

• In general seems a good idea to have a
  representation that
• Preserves all image information (we can go back
  to the image)
• Provides more independent channels of information
  than pixel values (we can mess with each band
  independently)
• But all this is just indirectly related to the
  texture synthesis task. But let assume is good
  enough

26
1-The steerable pyramid
27
1-The steerable pyramid
Steerable pyramid
Input texture
28
Overview of the algorithm
Two main tools 1- steerable pyramid 2- matching
histograms
29
2-Matching histograms
There are many ways of changing the histograms of
two signals.
30
2-Matching histograms
75 of pixels have an intensity value smaller
than 0.5
9 of pixels have an intensity value within the
range0.37, 0.41
5 of pixels have an intensity value within the
range0.37, 0.41
31
2-Matching histograms
Z(x,y)
We look for a transformation of the image Y Y
f (Y) Such that Hist(Y) Hist(f(Z))
?
Y(x,y)
32
2-Matching histograms
The function f is just a look up table it says,
change all the pixels of value Y into a value
f(Y).
Y f (Y)
Y(x,y)
Y 0.8
Y 0.5
New intensity
Original intensity
33
2-Matching histograms
Y f (Y)
34
Another example Matching histograms
10 of pixels are black and 90 are white
?
?
5 of pixels have an intensity value within the
range0.37, 0.41
35
Another example Matching histograms
The function f is just a look up table it says,
change all the pixels of value Y into a value
f(Y).
Y f (Y)
Y(x,y)
Y 0.8
Y 1
Original intensity
New intensity
36
Another example Matching histograms
Y f (Y)
In this example, f is a step function.
37
Matching histograms of a subband
38
Matching histograms of a subband
Y f (Y)
39
Texture analysis
(histogram)
Wavelet decomposition (steerable pyr)
Input texture
(histogram)
(Steerable pyr Freeman Adelson, 91)
The texture is represented as a collection of
marginal histograms.
40
Texture synthesis
Heeger and Bergen, 1995
Wavelet decomposition
(histogram)
Input texture
(histogram)
iterate
41
Why does it work? (sort of)
42
Why does it work? (sort of)
The black and white blocks appear by
thresholding (f) a blobby image
Iteration 0
43
Why does it work? (sort of)
The black and white blocks appear by
thresholding (f) a blobby image
44
Why does it work? (sort of)
After 6 iterations
Histograms match ok
red target histogram, blue current iteration
45
Color textures
R
G
Three textures
B
46
Color textures
R
G
B
47
Color textures
R
This does not work
G
B
48
Color textures
Problem we create new colors not present in the
original image. Why? Color channels are not
independent.
B
G
R
49
PCA and decorrelation
G
B
R
In the original image, R and G are correlated,
but, after synthesis,
G
G
R
R
50
PCA and decorrelation
The texture synthesis algorithm assumes that the
channels are independent. What we want to do is
some rotation
G
U2
Rotation
U1
R
See that in this rotated space, if I specify one
coordinate the other remains unconstrained.
51
PCA and decorrelation
G
correlation(R,G)
1.0000 0.9303 0.6034 0.9303 0.9438
0.6620 0.6034 0.6620 0.5569
C
R
PCA finds the principal directions of variation
of the data. It gives a decomposition of the
covariance matrix as
0.6347 0.6072 0.4779 0.6306 -0.0496
-0.7745 0.4466 -0.7930 0.4144
C D D
D
By transforming the original data (RGB) using D
we get

R G B
U1 U2 U3
D
3 x Npixels
3 x Npixels
3 x 3
The new components (U1,U2,U3) are decorrelated.
52
Color textures
R
G
Rotation Matrix (3x3)
D
B
These three textures Look less similar (lower
dependency)
These three textures look similar (high
dependency)
53
Color textures
R
Inverse Rotation Matrix
G
D
B
54
Color textures
55
Color channels
56
Color channels
57
Color channels
58
Examples from the paper
Heeger and Bergen, 1995
59
Examples from the paper
60
Examples not from the paper
Input texture
Synthetic texture
But, does it really work even when it seems to
work?
61
But, does it really work???How to measure how
well the representation constraints the set of
equivalent textures?
All the textures in this set have the same
parameters.
?
?
?
?
?
62
How to identify the set of equivalent textures?
?
This does not reveal how poor the representation
actually is.
63
We need a space that is more perceptual
In a perceptual space all these noise images are
very close. But in pixel space, they are very
far away.
How big is this set in a pixels space?
64
We need a space that is more perceptual
In a perceptual space all these noise images are
very close. But in pixel space, they are very
far away.
How big is this set in a perceptual space?
How big is this set in a pixels space?
65
How to identify the set of equivalent textures?
These trajectories are more perceptually salient
This set is huge
66
How to identify the set of equivalent textures?
67
Portilla Simoncelli
Heeger Bergen
Portilla Simoncelli
68
Portilla Simoncelli Texture Analysis

• Representation and synthesis of visual textures
• Texture Representation
• Crudely stated, our goal is to establish a
  minimal set of statistical measurements such that
  two textures are identical in appearance if and
  only if they agree on these measurements. We
  first decompose an example texture image using a
  multi-scale oriented linear basis (specifically,
  a steerable pyramid).
• Texture Synthesis
• Starting with an image of Gaussian white noise,
  we alternate between 1) constructing the
  steerable pyramid and forcing the sample
  statistics of each subband to match those of a
  reference texture image, 2) reconstructing an
  image from the pyramid, and forcing the sample
  statistics of the resulting pixels to match those
  of the reference image.

http//www.cns.nyu.edu/lcv/texture/
69
(No Transcript)
70
Texture analysis and synthesis
Original
Marginal Histograms (Heeger-Bergen)
Higher order statistics
Portilla Simoncelli, 00
71
Four statistics
72
(1) Marginal Statistics
73
(No Transcript)
74
(2) Coefficient correlation
It captures periodic or globally oriented
structure (within a neibhorhood size, e.g. 9
pixels). The local correlation of each subband.
It characterizes the salient spatial frequencies
and the regularity of the texture, as represented
by periodic or globally oriented structure
75
(3) Magnitude correlation
Capture structure (edges, bars, corners) and
second-order textures. cross-correlation of
each subband magnitudes with those of other
orientations at the same scale, and
cross-correlation of each subband magnitude with
all orientations at a coarser scale.
In the square texture large magnitude are at the
same location. Large magnitude of the two
diagonal orientations are anti-correlated in the
herringbone texture.
76
(3) Magnitude correlation
77
(3) Magnitude correlation
78
(4) Cross-scale phase statistics
Cross-scale phase statistics Distinguishes edges
from lines. Help represented gradients/lighting
effects. A local representation of the phase
(position), in order to represent edges and
lines. Important to represent 3dimensional aspect
and shadows, and more generally gradients due to
lighting effects.
79
Types of Texture classes

• Artificial-Non periodic
• (jrotpluses.jpg)

• Artificial-Periodic
• (checkerboard.jpg)

• Photographic/structured
• (food0008.jpg, windows.jpg)

• Photographic/pseudo-periodic
• (D76.jpg)

• Photographic/random
• (metal0004.jog)

• Inhomogeneous (non-textures)
• (face.jpg)

http//www.cns.nyu.edu/lcv/texture/
80
(No Transcript)