Extraction of Three-dimensional Information from Images Application to Computer Graphics

1
Extraction of Three-dimensional Information from
ImagesApplication to Computer Graphics

• Sylvain Paris

2
Introduction

• Data to render an image
• Shape
• Appearance
• Lighting

3
User-driven Approach
3D data
? Several days for an experienced user
4
Our Proposal Image-based
3D data
5
Expected Gain

• Data from real images (photographs)
• Duplication of the original object
• Shorter user time, edition as a post-process

6
Our Strategy

• General case is too broad
• Focus on a few useful cases
• Surface Reconstruction
• Image sequence, moving viewpoint
• Patchwork Reconstruction
• Face Relighting
• Single image and 3D model
• Capture of Hair Geometry
• Image sequence, moving light

7
Part One Surface Reconstruction

• Motivations
• Definitions and Concepts
• Previous Work
• Graph-cut Optimization
• Reconstruction Algorithm
• Results and Conclusions

8
Motivations

• Virtual environment are often empty
• 3D models from images
• From short image sequences
• User selects the interesting 3D region
• Automatic 3D reconstruction

Leyvand 03
9
Short image sequence Static geometry (time
freeze or still scene)
3D surface
Textured surface
Time freeze
10
Part One Surface Reconstruction

• Motivations
• Definitions and Concepts
• Previous Work
• Graph-cut Optimization
• Reconstruction Algorithm
• Results and Conclusions

11
Definition Consistency

• Does a 3D entity correspond to the input images?

?
12
Ill-posed Problem

• Several 3D scenes forthe same images.

2
1
1
2
13
Definition Functional

• Mathematical formula F ?? f (surface) d?
• Rates the quality of a surface
• Mixes consistency and a priori knowledge
• Goal Find the best surface
• Optimization problem
• Characterizes the problem

14
A priori Knowledge

• Classical claim Objects are smooth
• True for most of common objects
• Math Surfaces are continuous

15
Part One Surface Reconstruction

• Motivations
• Definitions and Concepts
• Previous Work
• Graph-cut Optimization
• Reconstruction Algorithm
• Results and Conclusions

16
Previous Work

• Carving
• Ease of use, arbitrary topology
• Not robust, ill-posed
• Level sets
• Arbitrary topology, geometric functional
• Over-smoothed results, convergence
• Graph cut
• Accurate segmentation, convergence
• Flat results, image functional

Seitz 99, Kutulakos 99, Broadhurst 01
Faugeras 98, Lhuillier 03
Roy 98, Ishikawa 00, Kolmogorov 02
17
Part One Surface Reconstruction

• Motivations
• Definitions and Concepts
• Previous Work
• Graph-cut Optimization
• Reconstruction Algorithm
• Results and Conclusions

18
Design of the Functional

• Surface parameterized in 3D world by z f (x,y)
• Functional Sum of two terms
• Consistency termIs the surface consistent with
  the input images ?
• Smoothing termIs the surface smooth ?

19
Consistency TermIs the surface consistent with
the input images ?
20
Smoothing termIs the surface smooth ?
Constraint Piecewise C1 surface
21
From Functional to Graph

• Optimization ? Graph-flow problem
• Global solution in polynomial time

22
Part One Surface Reconstruction

• Motivations
• Definitions and Concepts
• Previous Work
• Graph-cut Optimization
• Reconstruction Algorithm
• Results and Conclusions

23
Scenario

• Separating plane
• Matte surfaces
• Grouped cameras
• Challenging for depth precision

24
Algorithm

• Space discretization
• For each object (front to back)
• Consistency computation
• Selection of the consistent voxels
• Detection of potential discontinuities
• Graph-cut optimization
• Aliasing removal

25
Algorithm

• Space discretization
• For each object (front to back)
• Consistency computation
• Selection of the consistent voxels
• Detection of potential discontinuities
• Graph-cut optimization
• Aliasing removal

26
Algorithm

• Space discretization
• For each object (front to back)
• Consistency computation
• Selection of the consistent voxels
• Detection of potential discontinuities
• Graph-cut optimization
• Aliasing removal

27
Algorithm

• Space discretization
• For each object (front to back)
• Consistency computation
• Selection of the consistent voxels
• Detection of potential discontinuities
• Graph-cut optimization
• Aliasing removal

28
Algorithm

• Space discretization
• For each object (front to back)
• Consistency computation
• Selection of the consistent voxels
• Detection of potential discontinuities
• Graph-cut optimization
• Aliasing removal

29
Part One Surface Reconstruction

• Motivations
• Definitions and Concepts
• Previous Work
• Graph-cut Optimization
• Reconstruction Algorithm
• Results and Conclusions

30
40 images 692 x 461 1.5 meter wide
31
Reconstructed surface
Textured surface

• Discretization
• 140 x 150 x 330
• 7.106 voxels
• 35.106 nodes
• 166.106 edges

• Exceeds capacity of available graph-flow codes
  (specific code)
• 15min on a Xeon 2.4 GHz (700MB memory)
• To appear in Int. Journal of Computer Vision

32
Comparison
With Kolmogorov 02
Our result
These results have been discussed with Vladimir
Kolmogorov.
33
Limitations

• Hard to use Too many settings
• Time and memory consuming
• Already borderline on standard object
• Constrained scenario
• Dependent on the coordinate system

?
?
?
34
Conclusions onSurface Reconstruction

• Surface reconstruction method
• Shape from motion (parallax)
• Robust and precise (global minimum)
• Fundamental contributions
• Potential discontinuities from images
• Makes possible a convex smoothing term
• Eliminates the blocky effect
• Boundary characterization

35
Part Two Patchwork Reconstruction

• Motivations and Ideas
• Results and Conclusions

36
Main Idea

• Assumption Reconstruction is a local problem.
• No influence between distant regions
• Almost true Visibility is global
• Strategy Piece-by-piece reconstruction
• Surface seen as a patchwork
• Patches merged into
• a distance field

37
Advantages

• Low and constant memory footprint
• Lower time complexity
• Per-patch parameterization
• Multi-resolution

38
Part Two Patchwork Reconstruction

• Motivations and Ideas
• Results and Conclusions

39
25 images 800x600all around the object
Original images
Reconstructed model
European Conf. of Computer Vision 2004
40
25 images 800x600 all around the object
Original images
Reconstructed model
41
Conclusions onPatchwork Reconstruction

• Promising results
• More flexible surface representation
• Scalable (constant memory footprint)
• Accurate details (can still be improved)
• Edges (without blocky effect)

42
Part Three Face Relighting
Pacific Graphics 2003
43
Part Four Capture of Hair Geometry

• Motivations and Objectives
• Previous Work
• Overview of the Capture Process
• Details of the Algorithm
• Results and Conclusions

44
Motivations

• Hairstyle is important for digital characters
• Movies, games

• Duplicating real hairstyle is hard
• User-driven process
• Creation from scratch
• Edition at fine level

45
Goal

• Creation from images
• Automatic process
• Duplication of a real hairstyle
• 3D strands
• Appropriate structure
• Further manipulation (edition, animation)
• Only static geometry

46
Part Four Capture of Hair Geometry

• Motivations and Objectives
• Previous Work
• Overview of the Capture Process
• Details of the Algorithm
• Results and Conclusions

47
Previous Work

• Surface appearance
• Correct duplication
• Surface only, non-editable data
• Procedural filling
• 3D strands, volume duplication
• Style not duplicated
• 100 image-based
• 3D strands, style duplication
• Partial geometry (holes)

Matusik 02
Kong 97
Grabli 02
48
Part Four Capture of Hair Geometry

• Motivations and Objectives
• Previous Work
• Overview of the Capture Process
• Details of the Algorithm
• Results and Conclusions

49
Overview

• Dense and reliable 2D data
• Robust image analysis
• From 2D to 3D
• Reflection variation analysis
• Light moves, camera is fixed.
• Several light sweeps for all hair orientations
• Complete hairstyle
• Above process from several viewpoints

50
Setup Input
51
Input Summary

• We use
• 4 viewpoints
• 2 sweeps per viewpoint
• 50 to 100 images per sweep
• Camera and light positions known
• Hair region known (binary mask, 500x500)

52
Part Four Capture of Hair Geometry

• Motivations and Objectives
• Previous Work
• Overview of the Capture Process
• Details of the Algorithm
• Results and Conclusions

53
Main Steps

• Image analysis
• 2D orientation
• Highlight analysis
• 3D orientation
• Segment chaining
• 3D strands

54
Main Steps

• Image analysis
• 2D orientation
• Highlight analysis
• 3D orientation
• Segment chaining
• 3D strands

All camerastogether
55
Measure of 2D Orientation

• Goal 2D orientation of each pixel
• Hair images are complex
• Classical methods
• always fail on some pixels
• Our strategy Several methods, several images
• Select the method PER PIXEL
• Select the image PER PIXEL

56
Our Approach
Try several options ? Use the best

• Based on oriented filters

response
? argmax K? ? I
?
90
0
180
Most reliable ? most discriminant Lowest variance
57
Implementation

• 1. Pre-processing Filter images
• 2. For each pixel, test
• Filter profiles
• Filter parameters
• Light positions
• Pick option with lowest variance
• 3. Post-processing Smooth orientations
  (bilateral filter)

2
4
8
8
58
2D Results
8 filter profiles 3 filter parameters 9 light
positions
Our result
Sobel Grabli02
59
Main steps

• Image analysis
• 2D orientation
• Highlight analysis
• 3D orientation
• Segment chaining
• 3D strands

All camerastogether
60
Mirror ReflectionComputing Segment Normal
a
a
3 accuracy Marschner03
For each pixel Light position withhighest
intensity
61
Orientation from 2 planes
(3D position determined later)
62
Main Steps

• Image analysis
• 2D orientation
• Highlight analysis
• 3D orientation
• Segment chaining
• 3D strands

Camerasone by one
All camerastogether
63
Growing the Strands

• Start on an approx.of the head
• Chain 1 by 1
• Blend contributing views
• Stop
• Limit length (user)
• Out of volume (visual hull)

64
Part Four Capture of Hair Geometry

• Motivations and Objectives
• Previous Work
• Overview of the Capture Process
• Details of the Algorithm
• Results and Conclusions

65
Results
Siggraph 2004
66
Result Summary

• Similar reflection patterns
• Duplication of hairstyle
• Curly, wavy and tangled
• Blond, auburn and black
• Middle length, long and short

67
Limitations

• Structural point
• Image-based approach only visible part
• Occlusions not handled (curls)
• Limitations from the setup
• Head poor approximation
• Setup makes the subject move

68
Conclusions onCapture of Hair Geometry

• General contributions
• Dense 2D orientation (filter selection)
• 3D from highlights on hair
• System
• Proof of concept
• Sufficient to validate the approach
• Capture of a whole head of hair
• Different hair types

69
General Conclusions
70
Contributions

• Various information sources
• Camera motion, light motion
• Parallax, shading, highlights
• Appropriate representation
• 3D functional, meaningful skin parameters, 3D
  strands
• Limited a priori knowledge
• Prior is linked to the user, not to the data
• Piecewise-C1 surface, multi-filter scheme
• Useful applications in Computer Graphics

71
Limitations

• Only the visible information
• High resources (CPU time memory)
• Needed for quality and dense extraction
• Non trivial setting (except face relighting)

72
Future Work

• Extensions
• Small and large scale surface reconstruction
• Combining face and hair
• Hair motion capture
• Targeted data structures
• Theoretical study of the variance selection

73
Conclusions

• Using real images to extract information
• Hot topic
• Challenging problems
• More and more people interested

• Useful and usable applications
• Numerous applications
• appeared recently
• More to come

74
Thank you Questions?
75
Study of the smoothing term

• Linear term
• ? Continuous surface Discontinuous surface

? Need to control the discontinuities.
76
Effect of theVarious Treatments
Linear term
Convex approx.
No postprocessing
Full pipeline
77
11 images 640 x 480 1.5 meter wide
78
Reconstructed surface
Textured surface
Precision ? under 1/10th pixel
79
23 images 800 x 600 2 meters wide
80
Reconstructed surface
Textured surface
81
25 images 800x600 all around the object
Original images
Reconstructed model
82
3D mesh
83
Retrieving the Parameters
Pos

• Light Direct analysis
• Angular position Color
• Skin Analysis from synthesis
• Coarse-to-fine gradient descent
• 200 starting points ? lt 2.5
• Specific computation for shininess

SD
A
84
Detail Texture

• 3D mesh with the skin modelunder lighting
  conditions of the photo

Face rendered with skin
85
Reference image

• Four radial sines
• discontinuities
• Wavelength 2 pixels
• aliasing

86
Results on reference image
Our result(mean error 2.3)
Variance
87
Comparison with Sobel
Our result(mean error 2.3)
Sobel filter(mean error 17)
88
Visual hull

• 90 between the viewpoints
• Sharp edges

89
Reliable regions
? Front facing view
? Grazing view
90
Frequency selection
Band-filtering(difference of Gaussians)
Input image
91
Bilateral filtering

• Accounting for the adjacent pixels
• Spatial distance
• Filter reliability (variance)
• Appearance similarity (color)
• Weighted mean (Gaussian weights)

92
Comparison
Withoutvariance selection
Reference
93
Comparison
Withoutbilateral filtering
Reference