Texturing a Photographed Surface

1
(No Transcript)
2
Texturing a Photographed Surface
3
Why is this Hard?

• Why not use shape-from-shading to extract surface
  and surface texture synthesis to apply texture
• Integration of recovered surface normals leads to
  an inconsistent surface description, blurring
  details and accumulating error

Q A
?
0
4
Outline

• OverviewHow to texture a photograph in six
  easy steps
• Details
• Normal Recovery Tweaking
• Segmentation
• Surface Orientation Deformation
• Intercluster Orientation Deformation
• Displacement Mapping
• Feature Matching
• Applications
• Texture Replacement
• Normal Transfer

5
How to Texture a Photo

• Outline region of photo

6
How to Texture a Photo

• Outline region of photo
• Recover normals

7
How to Texture a Photo

• Outline region of photo
• Recover normals
• Cluster pixels by normal

8
How to Texture a Photo

• Outline region of photo
• Recover normals
• Cluster pixels by normal
• Distort u,v by normal

9
How to Texture a Photo

• Outline region of photo
• Recover normals
• Cluster pixels by normal
• Distort u,v by normal

10
How to Texture a Photo

• Outline region of photo
• Recover normals
• Cluster pixels by normal
• Distort u,v by normal
• Texture each cluster

11
How to Texture a Photo

• Outline region of photo
• Recover normals
• Cluster pixels by normal
• Distort u,v by normal
• Texture each cluster
• Merge clusters via Graphcut

12
Textureshop Approach

• Overcomes limitations of shape-from-shading by
  operating on clusters instead of entire image
• Uses texture boundaries (via graphcut) to obscure
  cluster boundaries, surface inconsistencies
• Avoids surface representation altogether, using
  texture distortion to imply surface orientation

13
Other Approaches

• Could scan object and perform surface texture
  synthesis Wei Levoy S01, Turk S01
• Could create a layered-depth image from photoOh
  et al., S01 and texture that
• Or use a texture replacementmethod, such as
• Tsin et al, CVPR01based on stochasticannealing
  , which helps
• Liu et al., S04 to fit a gridto an extracted
  lightingdeformation field

From Y. Liu, W.-C. Lin J.H. Hays.Near-Regular
Texture Analysis andManipulation. Proc.
SIGGRAPH 2004
14
RecoveringNormals
s
S
N
c
G
?I

• From Horns Height gradient from shading,
  IJCV90
• Assume Lambertian Ixy A kDNxy?S
• Dimmest pixel gives ambient Imin A
• Brightest pixel faces light Imax A kD
• Intensity relative to contrast gives cosine of
  incidence
• cxy cos qi Nxy?S (Ixy-Imin)/(Imax-Imin)
• sxy sin qi (1-cxy2)½
• Image gradient ?Ixy (?Ixy/?x, ?Ixy/?y, 0)
• Project ?Ixy perp. to S Gxy ?Ixy (?Ixy?S)
  S
• Normal N cxy S sxy Gxy/Gxy

15
Normal Tweaking
orig. normals

• Problems w/recovered normals
• Lighting multiple sources, interreflection
• Material non-Lambertian, textured
• Interactive Fixes
• Manipulate spline-based BRDF (initialized to
  Lambertian)
• Smooth, enhance, invert normals
• Rotate normals if 2nd light source

enhanced
smoothed
inverted
16
Segmentation

• Need small round clusters of similar normals
• E(P1,P2) k1(1-N1?N2)½ k2C1-C2
  k3(P1P2)
• For each cluster Pi
• Pi pixels
• Mean normal Ni SNxy/SNxy
• Centroid Ci
• Constants
• k1, k2, k3 187,20,1
• Agglomerate
• Initialize Pi 1
• Merge when E(Pi,Pj) lt threshold

17
Deform Clusters byRecovered Normals

• Need to assign U (u,v) coords to each pixel
  (x,y) in cluster according to normal
• U(x,y)?(u,v)?? 2-D distortion map
• Set U(x,y) (0,0) at patch centroid
• Propagate texcoords through cluster in
  width-first floodfill order

18
(No Transcript)
19
(No Transcript)
20
Intercluster Orientation

• Paint vector field on image to align directional
  texture features

21
DisplacementMapping

• Apply shape-from-shading to recover normals NT of
  texture source image
• Recover height field of texture source
• ?2h(x,y) ??NT(x,y)
• Synthesize texture color height normal
• Each texture sample translated
• along photos recovered normal N(x,y)
• by the texture height h(u,v)
• foreshortened by the texture normal NT(u,v)
• Upsample to avoid holes

22
Feature Matching

• Define per-pixel offsets (u,v) in cluster overlap
  to align features of neighboring clusters
• Define an objective function that matches pixel
  colors where they overlap with neighboring
  clusters
• ? k1SP1(x,y) P2(xu, yv) k2S??(u,v)
• Minimize ? with conjugate gradients

23
Results
24
Surface Replacement
25
(No Transcript)
26
(No Transcript)
27
Normal Transfer

• Emboss an image (e.g. tree) with normals from
  second image (e.g. Mona Lisa)
• Combine normals brightness of both images via
  Poisson image editing
• Retexture using combined normals using targets
  texture (e.g. bark) for synthesis

28
Image of buildingwithout eagle
29
(No Transcript)
30
Texture SynthesisPerformance
FM Feature Matching Disp Displacement Mapping
24s, 55s w/Disp 140s w/Disp,FM 466x550
90s w/FM449x593
12s w/o FM(shown w/FM) 480x640
18s w/o FM 790x586
63s w/FM 1063x565
Synthesis in 10-30s FM and Disp add 30-90s
31
Conclusion

• Textureshop offers a new tool that conveniently
  and robustly allows a user to texture an object
  in a photograph.
• Textureshop is limited to smooth diffuse surfaces
  illuminated by simple lighting conditions, but
  the method has worked well for us on faces and
  T-shirts, as well as sculptures and architectural
  embellishments.
• Textureshop works well enough to serve as a tool
  for concept visualization in architecture, art,
  fashion, design, visual effects and personal
  digital content creation.

32
Thanks

• Funded in part by the National Science Foundation
  under ITR 0121288
• Jesse Hall, Patrick Lacz and Tony Kaap for videos
  proofreading