Texture Optimization for Example-based Synthesis

1
Texture Optimization for Example-based Synthesis

• Vivek Kwatra Irfan EssaAaron BobickNipun Kwatra

2
Image vs. Texture
3
Example-based Texture Synthesis
Input Example
4
Related Work

• Local region-growing method
• -Pixel-based
• -Patch-based
• Global optimization-based method
• Heeger and Bergen sig95,Pyramid-based texture
  synthesis
• Paget and Longstaff IEEE Tran 98,Texture
  synthesis via a noncausal nonparametric
  multiscale markov random field
• ..
• Physical Simulation et al

5
Example-based Texture Synthesis

1. Dynamic texturing (animation/video)
2. Greater control external knobs, e.g., flow,
   shape, illumination
3. Texture quality evaluation and refinement

6
Motivation

InputTexture
InputFlow Field
7
Limitations of previous work

• Pixel-based
• can not maintain global structure of the
  texture
• inconsistencies
• Patch-based
• have no control over individual pixel values
• Global methods
• pixel-to-pixel interactions
• insufficient to capture large scale
  structures
• complex formulations difficult to optimize

8
Texture Energy

• Pixel-by-pixel comparison of source and target
  not possible
• Compare texture elements
• Local pixel neighborhoods
• Want each target neighborhood to be similar to
  some source neighborhood

9
Texture Energy
Z(source)
X(target frame)
10
Energy for Single Neighborhood
Z(source)
p (pixel)
X(target frame)
11
Energy for Single Neighborhood
Z(source)
Xp (neighborhood)
X(target frame)
12
Energy for Single Neighborhood
Z(source)
Xp (neighborhood)
X(target frame)
13
Energy for Single Neighborhood
(nearest neighbor) Zp
Z(source)
Xp (neighborhood)
X(target frame)
14
Energy for Single Neighborhood
(nearest neighbor) Zp
Z
Xp (neighborhood)
Texture Energy(single neighborhood)
X
15
Texture Energy for Entire Image
Z
X
16
Texture Energy for Entire Image
Z
X
? individual neighborhood energy
17
Texture Synthesis

• 1.Initialize texture X

2.Find closest input neighborhood ZP
corresponding to each output neighborhood XP
3.Update X to be texture that minimizes the energy
4.Repeat iteratively until set ZP stops
changing
18
Texture Synthesis
19
Gradient-based Energy
D is the differentiation operator and µ is a
relative weighting coefficient
20
Muti-level Synthesis

• Multi-resolution
• Multi-scale/multiple neighborhood sizes
• Reasons
• a good initialization of the texture
• avoid undesirable local minima
• avoid inconsistencies
• capture large scale structures

21
Texture Energy Graph

• Random Initialization
• Multiple
• Resolution Levels
• Neighborhood Sizes
• Progressively refined output

22
Controllable Synthesis
23
Flow-guided Texture Animation

• Flow consistency
• Perceived motion similar to flow
• Texture similarity
• Shape, size, orientation of texture elements
  similar to source

Source Texture
Texture Similarity
FlowConsistency
Flowing Target
Target Flow
24
Naive Approach

• Ignore Texture Similarity
• Warp each frame via flow field

Source Texture
Warp
TargetFrames
X1
Xn
X0
Texture Similarity
FlowConsistency
Flowing Target
Target Flow
25
Naive Approach

• Texture structure not maintained

Source Texture
Texture Similarity
FlowConsistency
Flowing Target
Target Flow
26
Instead of

• Warp

TargetFrames
27
Instead of

• Warp

TargetFrames
28
Correct Operation

• Warp Correct

TargetFrames
29
Correct Operation

• Link to original goals

TargetFrames
X
X
Correct
Warp
W
30
Correct Operation

• Link to original goals

TargetFrames
X
X
Correct
Flow Consistency
Warp
W
31
Correct Operation

• Link to original goals

Source
Texture Similarity
TargetFrames
X
X
Correct
Flow Consistency
Warp
W
32
Optimization of Energy

• Energy Flow Energy Texture Energy

Source
Texture Similarity
TargetFrames
X
X
Correct
Flow Consistency
Warp
W
33
Flow Energy
2D flow field
frame i
frame i1
pixel (p,i)
pixel (q,i1)
34
Results Texture Synthesis
35
(No Transcript)
36
Results Texture Synthesis
Input
Wei-LevoyWei00
Image QuiltingEfros01
Graph-cutsKwatra03
Texture OptimizationKwatra05
37
Time Cost

• Static texture synthesis
• 5-10 iterations per resolution level
• Flowing texture synthesis
• 20-60 seconds per frame

Resolution One iteration Total time
64x64 2.5 sec 20 sec
128x128 10 sec 2 min
256x256 25 sec 8.5 min
38
Limitations

• Optimization finds local minima
• Blurry / misaligned texture elements
• Texture scale should be compatible with flow field

39

• Thanks a lot!

40
Pixel-based Methods

• Compare local causal neighbourhoods
• Efros and Leung (ICCV 99) Wei and
  Levoy (Siggraph 2000) Ashikhmin (I3D 2001)

Input
Output
41
Patch-based Methods

• Copy patches of pixels rather than single pixels
• Chaos Mosaic, Xu et al, 1997
• Patch-Based Sampling, Liang et al(ACM 2001)
• Image Quilting, Efros and Williams(Siggraph 2001)

42
Back