Machine Learning for Computer graphics

1
Machine Learning for Computer graphics
Bayesian

• Aaron Hertzmann
• University of Toronto

2
Computers are really fast

• If you can create it, you can render it

3
How do you create it?
Digital Michaelangelo Project
Steven Schkolne
4
Two problems

• How do we get the data into the computer?
• How do we manipulate it?

5
Data fitting
6
Key questions

• How do you fit a model to data?
• How do you choose weights and thresholds?
• How do you incorporate prior knowledge?
• How do you merge multiple sources of information?
• How do you model uncertainty?

Bayesian reasoning provides a solution
7
Talk outline

• Bayesian reasoning
• Facial modeling example
• Non-rigid modeling from video

8
What is reasoning?

• How do people reason?
• How should computers do it?

9
Aristotelian Logic

• If A is true, then B is true
• A is true
• Therefore, B is true

A My car was stolen B My car isnt where I left
it
10
Real-world is uncertain

• Problems with pure logic
• Dont have perfect information
• Dont really know the model
• Model is non-deterministic

11
Beliefs

• Let B(X) belief in X,
• B(X) belief in not X
• An ordering of beliefs exists
• B(X) f(B(X))
• B(X) g(B(XY),B(Y))

12
Cox axioms

• R.T. Cox, Probability, frequency, and reasonable
  expectation, American J. Physics, 14(1)1-13,
  1946

13

• Probability theory is nothing more than common
  sense reduced to calculation.
• - Pierre-Simon Laplace, 1814

14
Bayesian vs. Frequentist

• Frequentist (Orthodox)
• Probability percentage of events in infinite
  trials
• Medicine, biology Frequentist
• Astronomy, geology, EE, computer vision largely
  Bayesian

15
Learning in a nutshell

• Create a mathematical model
• Get data
• Solve for unknowns

16
Face modeling

• Blanz, Vetter, A Morphable Model for the
  Synthesis of 3D Faces, SIGGRAPH 99

17
Generative model

• Faces come from a Gaussian

Learning
18
Bayes Rule
Often
19
Learning a Gaussian
20
Maximization trick

• Maximize
• lt-gt minimize

21
Fitting a face to an image

• Generative model

22
Fitting a face to an image

• Maximize

minimize
23
Why does it work?
p(xmodel)
24
General features

• Models uncertainty
• Applies to any generative model
• Merge multiple sources of information
• Learn all the parameters

25
Caveats

• Still need to understand the model
• Not necessarily tractable
• Potentially more involved than ad hoc methods

26
Applications in graphics

• Shape and motion capture
• Learning styles and generating new data

27
Advanced topics

• Marginalize out some data
• Keep all uncertainty

28
Learning Non-Rigid 3D Shape from 2D Motion

• Joint work with Lorenzo Torresani and Chris
  Bregler (Stanford, NYU)

29
Camera geometry

• Orthographic projection

30
Non-rigid reconstruction

• Input 2D point tracks
• Output 3D nonrigid motion

Totally ambiguous!
31
Shape reconstruction

• Least-squares version

minimize
32
Bayesian formulation
maximize
33
How does it work?
Maximize
Minimize
(actual system is slightly more sophisticated)
34
Results
Input
Least-squares, no reg.
View 2
Least-squares, reg.
Gaussian
LDS
35
Conclusions

• Bayesian methods provide unified framework
• Build a model, and reason about it
• The future of data-driven graphics

36
Background

• C. Bregler, A. Hertzmann, H. Biermann, CVPR 2000,
  Recovering Non-Rigid 3D Shape from Image
  Streams
• Non-rigid structure from motion

37
Background

• L. Torresani et al., CVPR 2001, Tracking and
  Modeling Non-Rigid Objects with Rank Constraints
• M. Brand, CVPR 2001, Morphable 3D models from
  video, Flexible flow
• Loop
• Track based on non-rigid SFM
• Update non-rigid SFM based on tracking

38
Shape regularization

39
Our approach

• Learn Gaussian deformation PDF
• Simultaneously with reconstruction
• Similar to Cootes-Taylor, Vetter-Blanz, etc.
• Similar to factor analysis, PPCA, TCA

40
Factor analysis model
Like factor analysis Use EM Ghahramani-Hinton,
Roweis, Tipping and Bishop
41
Temporal dynamics
42
Results
43
Results