The patch transform and its applications to image editing*

1
The patch transformand its applications to image
editing

• Taeg Sang Cho1, Moshe Butman2, Shai Avidan3 and
  William Freeman1,3
• CSAIL-MIT1, Bar-Ilan Univ.2, Adobe Inc.3

To appear at CVPR 2008
2
The patch transform
3
The patch transform
4
The patch transform
Easy!
5
The inverse patch transform?
Easy!
Jigsaw puzzle is an NP-complete problem
6
Outline

• The inverse patch transform
• The exclusivity term in BP
• Image editing applications
• Subject reorganization
• Image retargetting
• Photomontage
• Future work and conclusions

7
The inverse patch transform
8
Seam-based compatibility
9
Seam-based compatibility
D-T compatibility
L-R compatibility
10
The inverse patch transform
11
The inverse patch transform
12
A low resolution image as the local evidence
Assume that we have a low resolution version of
the original image
13
The inverse patch transform
14
The exclusivity factor node
The factor node will steer the BP to use patches
seldom more than once
15
The inverse patch transform
16
The image model
Exclusivity
Pairwise Compatibility
Local evidence
Patch prior(assumed uniform for now)
17
The exclusivity term
18
A toy example two image nodes
2
1
P(x2) 0.3, 0.7
P(x1) 0.55, 0.45
m2f 0.3, 0.7
m1f 0.55, 0.45
19
A toy example two image nodes
2
1
P(x2) 0.3, 0.7
P(x1) 0.55, 0.45
m2f 0.3, 0.7
m1f 0.55, 0.45
mf2 0.45, 0.55
mf1 0.7, 0.3
20
A toy example three image nodes
P(x2) 0.15, 0.35, 0.5
P(x3) 0.65, 0.25, 0.1
3
m3f 0.65, 0.1, 0.25
m2f 0.15, 0.35, 0.5
mf1 0.2975, 0.5850, 0.3750
(1-0.15)x(1-0.65)
(1-0.35)x(1-0.1)
(1-0.5)x(1-0.25)
21
The exclusivity term
22
The exclusivity term
MN states
23
The exclusivity term
MN states
24
The exclusivity term
MN states
25
Applications Subject reorganization
Input image
26
Applications Subject reorganization
User input
27
Applications Subject reorganization
Output with corresponding seams
28
Applications Subject reorganization
Output image after Poisson blending
29
Applications Subject reorganization
No local evidence No exclusivity term
Yes local evidence No exclusivity term
No local evidence Yes exclusivity term
Yes local evidence Yes exclusivity term
30
Applications Object removal
Original image
User Input
31
Applications Object removal
Output image
Output imagewith overlaid seams
32
Applications Image retargetting
User Input
0.8 x 0.8
Output image
Output imagewith overlaid seams
33
Applications Texture control
Output image
Output imagewith overlaid seams
34
Applications Photomontage
Input image
Input image
35
Applications Photomontage
Output imagewith overlaid seams
Output image
36
Limitations
Artifacts due to large patch size
37
Conclusion

• The patch transform representation of images for
  easy manipulation
• The exclusivity term for controlling
  state-distribution in the MRF
• A number of image editing applications on a
  unified pipeline.

Acknowledgments ONR-MURI grant N00014-06-1-0734,
Shell Research, Samsung scholarship foundation.