Geometric Algorithms in 3D Real-Time Rendering and Facial Expression Modeling (Part II)

1
Geometric Algorithms in 3D Real-Time Rendering
and Facial Expression Modeling (Part II)

• Prof. Marina L. Gavrilova
• University of Calgary

2
Where we are

• Brief Overview of the Area
• Geometric Approaches for mesh reconstruction and
  image morphing
• Real-Time Optimally Adapting Mesh Algorithms
• Non Photo-realistic Facial Expression Modeling
  and Morphing Algorithms
• Swarm Real-time Visualization System

3
Part II

• NPR Face Rendering

4
Terminology

• DT Delaunay triangulation
• VD Voronoi Diagram
• FT Feature Transform
• DT Distance Transform

5
Preliminaries

• Transformation from a photograph into an artistic
  work

Li and Huang 03
6
Preliminaries

• Transformation from an Excel into Artistic Work ?

7
NPR vs. PR philosophy

• PHOTOREALISTIC
• GOALS imitate as realistically as possible
  facial expression and details
• Methodology
• complex geometry fitting,
• generating high density meshes,
• recovering camera positions,
• utilizing data interpolation,
• simulating light conditions,
• applying texturing and other techniques
• Outcome
• Very costly processing, highly realistic
  visualization

• NON-PHOTOREALISTIC
• GOALS imitate emotions as closely as possible,
  while at the same time allowing the model its own
  identity and artistic representation
• Methodology
• Utilize artistic drawings
• Feature line extraction
• Expression morphing
• Model partitioning onto regions
• Outcome
• Expression and emotion preservation, efficient
  storage, fast processing, artistic visualization

8
Animation Trivia

• Face models used in Pixars Toy Story had
  approximately how many control points each?
• 100
• 1000
• 10000
• None they were made by hand out of clay!

9
Animation Trivia

• TOY STORY" used how many SUN workstations to
  render images?
• 1
• 100
• 1000
• None they were all IBM workstations!

10
Animation Trivia

• Pixar have received how many Academy Awards
  from the Academy of Motion Picture Arts and
  Sciences, including an Oscar for Best Short
  Animated Film for 'Tin Toy' and a Technical
  Achievement Award for Pixar's RenderMan
  software?
• 2
• 6
• 12
• One for every animated character created!

11
NPR vs. PR

• PHOTOREALISTIC - Final Fantasy (from Final
  Fantasy website)

NON-PHOTOREALISTIC - Shrek 10 to 1 preference
Crawford (Shrek movie web site)
12
Oscar for ACM Fellow

• ACM Bulletin ServiceToday's Topic ACM Fellow Ed
  Catmull Receives Oscar for Contributionsto
  Computer GraphicsFebruary 25, 2009

• ACM Fellow Ed Catmull was presented with an Oscar
  statuette for the Gordon E. Sawyer Award by the
  Board of Governors of the Academy of Motion
  Picture Arts and Sciences.
• Conferred at the Scientific and Technical Awards
  Presentations earlier this month, the award is in
  recognition of his lifetime of technical
  contributions and leadership in the field of
  computer graphics for the motion picture
  industry.

13
Objectives of our Research

• To develop efficient and realistic portrait
  generation method from 2D photographs
• To develop fast and easy 2D morphing algorithm
• To create appropriate 3D mesh representation for
  model morphing

14
Developed Methodology

• Proposed to use NPR modeling by example for
  portrait generation
• Applied binarization method to produce black and
  white images from photograph
• Developed segmentation technique for detailed
  feature extraction
• Used combined edge eextraction strategy
• Used application-dependent style for rendering
• Developed Euclidean distance transform based
  algorithm for morphing

15
Facial Modeling Research

• Focus on Human Faces

Lin and Hong 02
Li and Kobatake 97
16
Facial Modeling Research

• Other examples

Lin and Hong 02
Gooch 04
Hong 04
17
Motivation

• Shortcoming of some previous works
• Facial creases, wrinkles and freckles are hard to
  represent
• Detail feature lines of different expressions and
  which lines?
• Efficient morphing

18
Contribution

• Task Generate a smooth line-drawing animation
  from two input photographs
• Input/Output two human face images as input and
  output of a NPR facial animation
• Methodology
• Create NPR style portrait from a frontal facial
  photograph to generate a digital artwork.
• Use segmentation and tracking methods to map
  expressive lines representing expressive
  wrinkles and creases into the image.
• Utilize morphing algorithm based on Euclidean
  distance transform that does not require control
  points or control lines, and finds pixel
  correspondence automatically.
• Use the developed system as a prototype for 3D
  morphing based on controllable mesh deformations
  utilizing Delaunay triangulation.

19
System FlowChart
20
Methodology Use artists rendering as a template
Six Primary Facial Expressions drawn by
artistsanger, disgust, fear, happiness, sadness
and surprise
21
Methodology Portrait generator

• Producing black-and-white images using
  binarization

22
Methodology Details Extraction
Enlarge certain areas
Edge Detection
Delete normal contours
Detail feature lines tracking
Save feature Lines information
23
Methodology region subdivision

• Five interested sub-regions are used to extract
  details

24
Methodology edge detection

• Edge Detection Process Sobel, Prewitt, Roberts,
  LoG, Canny operators (from (b) to (f)), g
  combination of e and f

25
Methodology details extraction

• Close-up Details Extraction feature

26
Methodology feature line tracking

• Feature Lines Tracking

27
Methodology feature line tracking

• Tracking Strategy

28
Methodology feature line tracking

• Data Structure to store the feature lines

29
Methodology Distance transform for Morphing
Image
Template
30
Distance Transform
Given an n x m binary image I of white and black
pixels, the distance transform of I is a map that
assigns to each pixel the distance to the nearest
black pixel (a feature).
31
Feature Transform
The feature transform of I is a map that assigns
to each pixel the feature that is nearest to it.
32
A Fast Algorithm for Computing the Euclidean
Distance Transform

• Works in Euclidean Metric
• Optimal linear in the number of pixels
• Proceed in two passes over the image, top down
  and bottom up
• For each row
• Maintain a polygonal chain - a set of pixels in
  which the nearest feature may lies
• Updates the chain for next row by pruning - it
  becomes the minimal set of pixels
• Prune Endpoints and Internal Vertices

33
Chain Definition

• One Chain for the row
• Contains at most one pixel per column
• Contains the lowest pixel in the column
• After pruning, it contains only those points that
  will be nearest features to some points in the
  row
• Dynamically updated from row to row

34
Example of a a Chain
35
Pruning Endpoints

• Identifying superfluous endpoints

36
Pruning Internal Points

• Point q is removed

37
Pruning Internal Points

• Point q remains

38
Results on NPR Rendering
39
Experimental results for distance transform
algorithm
Algm 1 with pruning was tested Saturation levels
5, 15, 30 Results Linear Running Time,
Regardless of Saturation ( of black pixels in
image)
40
Morphing Direction
41
Results of utilizing Distance Transform for
Morphing
Starting Frame
Ending Frame
42
Results of feature lines extraction
43

• Line Simplification and Rendering Results

44

• Line Simplification and Rendering

45
Animation
46
3D Facial Model Synthesis
47
Brief Preliminaries

• Over the last 30 years, thousands of papers were
  devoted to 3D face animation and modeling.
• Different classification and registration
  techniques have been developed, including those
  based on 2D feature matching, multiple point of
  view registration (using calibrated cameras),
  nearest-neighbor feature comparison,
  segmentation-based feature extraction, ICP
  (Iterative Closest Point-based Matching), TPS
  (Thin Plates Spline) algorithms and others.

48
3D Facial System Classification(by Akarun,
Gokberk, Salah)
49
3D Facial System Classification(by Akarun,
Gokberk, Salah) cont.
50
3D Facial System Classification(by Akarun,
Gokberk, Salah) cont.
51
3D Facial System Classification(by Akarun,
Gokberk, Salah) cont.
52
Voronoi Diagram for 3D face modeling

• Voronoi Diagram is a partitioning of the space
  into convex regions such that any point within
  the region is closer to the site that originates
  this region than to any oethr site in the space.
• Delaunay Triangulation is the dual structure that
  connects those sites whose Voronoi Regions share
  an edge (in 2D) or a face (in 3D).

53
Methodology

• Utilize geometric concept Voronoi diagram in
  application to FFD (free-form deformation) model.
  
• Develop a method for 3D facial model synthesis
  that combines the traditional free-form
  deformation (FFD) model with techniques of data
  interpolation based on Voronoi diagrams.
• With 18 feature points extracted from 2D facial
  images in two orthogonal views, Dirichlet Free
  Form Deformation (DFFD) is utilized for modifying
  a generic 3D face to produce the individual face.
  
• The main advantages of this approach over former
  extensions of FFD is in removing the constraints
  on control lattice and control points location.
• By assigning different weights to those control
  points, we make the method more adaptable to the
  facial deformation. The reconstructed 3D faces
  can be used to generate different facial
  animations.

54
Methodology

• Build DFFD model
• Compute Sibson coordinates using space
  partitioning introduced by Voronoi diagram
• Compute deformation in 3D using weighted DFFD
  relationship based on computed Sibson coordinates
• Adapt approach for face synthesis using Delaunay
  triangulation

55
Illustration

• 28 zones (below).

• Control points (above).

56
Illustration

• FaceGen generic mesh and texturing result

57
Illustration

• Original face (left) and generated face (right)

58
Extensions

• The developed methodology is currently being
  utilized for 3D face morphing and expression
  synthesis.

59
Acknowledgements NSERCGranting Agency,
CanadaGEOIDENetwork of Excellence,
CanadaCanadian Foundation for Innovation All
students of SPARC LAB and BT Lab, University of
Calgary, in particularly Russel Apu and Luo Yuan
All collaborators and of course the audience