Example-Based Composite Sketching of Human Portraits

1
Example-Based Composite Sketching of Human
Portraits

• Hong Chen1,2, Ziqiang Liu1,2, Yingqing Xu1,
  Chuck Rose3,
• Heung-Yeung Shum1, David Salesin4,5
• NPAR 2004
• 1 Microsoft Research , Asia
• 2 University of California, Los Angeles
• 3 Microsoft Corporation
• 4 University of Washington
• 5 Microsoft Research

2
Result
3
Outline

• Introduction
• Related work
• System framework
• Composing a face
• Composing hair
• Examples
• Conclusions and future work

4
Introduction

• An interactive system for generating human
  portrait sketches
• Input
• A human face image
• Output
• A sketch that exhibits the drawing style of a set
  of training examples provided by an artist
• Style of Japanese cartooning

5
Introduction (Cont.)

• Propose a composite sketching approach
• Decompose the data into components that are
  structurally related to each other.
• As the eyes or mouth
• Propose a system which combines two separate but
  similar subsystems.
• The face subsystem
• The Hair subsystem

6
Introduction (Cont.)

• After these have been independently processed,
  they are carefully recomposed to obtain the final
  result.

7
Related Work

• NPR and digital arts
• Used to depict facial images with an artistic
  style
• Digital facial engraving Ostromoukhov, SIGGRAPH
  99 and Caricature generation Brennan, 85
• Emulate traditional artist tools to assist users
  in drawing pictures with a certain style
• Rare attempt to generate digital paintings by
  learning from artists

8
System Framework

• Goal
• Create a system that could leverage the artists
  skill with a high degree of automation.
• A training set
• The gamut of east Asian female faces
• Divide the portrait system
• The face subsystem
• The hair subsystem

9
System Framework (Cont.)

• The face subsystem
• Segment into sub-problems for each of the natural
  facial features.
• i.e. eyes, mouth
• Global and Local models
• The hair subsystem
• Be handled carefully with a structural model and
  a detailed model.
• These sub-problems are tackled independently

10
Composing a Face

• Objective
• Construct a model to take an input image I and
  generate a sketch I that matches the style of
  the training examples.
• Training Set
• Model

11
Composing a Face (Cont.)

• Split into two layers
• Global
• capture how the artist places each face element
  in the sketch image.
• Local
• mimic how the artist draws each independent
  element locally.

12
Composing a Face (Cont.)
13
Drawing the Facial Component with the Local Model

• A human face is decomposed semantically into 6
  local components.
• Left right eyebrows, left right eyes, a nose,
  and a mouth

14
Drawing the Facial Component with the Local Model
(Cont.)

• Extract the accurate shape and associated texture
  information using an Active Shape Model.
• Determine to which prototype the component
  belongs
• Build a different classifier for each type of
  component to cluster the input components into
  the appropriate prototype.
• KNN interpolation

15
K Nearest Neighbor

• The goal is to find a class label for the unknown
  example xu

- Euclidean distance - K5
16
Composing the Face Using the Global Model

• The global model
• Use to arrange each face element on a canvas.
• For drawing facial caricatures
• Relationship of elements to others of their own
  kind.
• Relationship of elements to their surrounding and
  adjacent elements.

17
Composing the Face Using the Global Model (Cont.)

• For the representation of Ig

18
Composing the Face Using the Global Model (Cont.)

• By not tying these relations to fixed value
• The model can adjust the size of the feature as
  the overall size of the head is changed.
• Generate Ig
• Use Active Shape Model
• 87 control points
• Determine the placement of the face elements on
  the cartoon canvas.

19
The Placement of the Face Elements on the Cartoon
Canvas

• Each element needs five parameters
• (tx, ty), (sx, sy),?

20
Composing Hair

• Hair cannot be handled in the same way as the
  face
• Hair has many styles.
• Is not structured in the same regular way that
  faces are
• A single unit
• Render using long stroke
• There is no clear correspondence between regions
  of two different hairstyles.

21
Hair System Flow
22
Hair System Flow (Cont.)

• Structural components
• Coarsely segment the hair into five segments
• Each indicates important global information about
  the hair.
• The detail model
• Add uniqueness and expression to a portrait.

23
Hair Composite Model

• The global hair structure or impression is more
  important than the detail.

24
Extracting the Image Features for the Hair

• Determine the image features of the hair
• Match against the database
• An estimated alpha mask
• Hair strand orientation fields

25
Fitting Structural Components

• For an input image, finding the best training
  data.
• Classify input image into the correct style.
• Find the best training example.

26
Fitting Structural Components (Cont.)

• Deform hair components to a standard shape using
  a multi-level freeform deformation warping
  algorithm.
• Hair orientation vector
• G gx1,gy1, gx2,gy2 , , gxn,gyn
• Alpha value
• a a1,a2 ,,an
• E(H1,H2) G1-G2wa1-a2

27
Fitting Structural Components (Cont.)

• Shape a set of corresponding key points
• S x1,y1,,xm,ym
• Find the best matched training example
• Minimize the distance combining the appearance
  and the shape distance

28
Fitting Detail Components

• Different kinds of detail require slightly
  different approaches.
• Boundary and bangs

29
Fitting Detail Components (Cont.)

• Boundary details
• The alpha value and orientations for these
  patterns are quite different.

30
Fitting Detail Components (Cont.)

• Bang detail components
• Used to detect the bang
• Segment out the bang regions in the alpha mask
• The orientation field can be inspected.
• Determine the length of the bang line

31
Synthesizing the Hair Sketch

• The stroke in the training samples are all
  divided into two classes.
• Boundary strokes and streamline strokes
• Link points
• The points in the strokes crossing the boundary
  of a structural component
• Face contour exaggeration
• Adjust the each part of the inner hair boundary
  according to the corresponding face contour

32
Composing the Structural Components

• Warp to the target coordinates.
• Match link points those in the same class.
• Adjust to the average position and link the
  corresponding strokes.
• Remove unmatched streamline strokes.

33
Add the Detail Strokes

• Detail components are connected to strokes
  generated by the component match.
• Warp to the target coordinates from the global
  phase

34
Examples

• (b) Result of local model.
• (c) Result of local model plus global model.
• (d) Result without local model and global model.

35
Examples (Cont.)

• Compare the effect of adding detail strokes.
• (b) Result of composing structural components.
• (c) Composing with detail components.

36
Examples (Cont.)

• Combine the face and the hair
• Neck, shoulder, and clothing are chosen from a
  set of templates supplied by the artist.

37
Examples (Cont.)
38
Conclusions

• Adapting a global/local hybrid was an effective
  approach for generating face portrait sketches.
• Application
• Create a virtual person for cartoon style online
  games or chat environments
• Create sketched portraits for places where a
  sketch is preferred over a photograph

39
Future Work

• Limitations and future work
• Add to the training set
• Encompass faces of many racial backgrounds
• Render male images
• A third subsystem to handle
• Aging, injury, spectacles, and jewelry
• A face in profile