Integrating Speech Synthesis and Emotional Variations with Dynamic 3D Facial Expressions using Simul

1
Integrating Speech Synthesis and Emotional
Variations with Dynamic 3D Facial Expressions
using Simulation of Anatomical Muscles

• by
• Youssef Hijazi
• Committee Members
• Dr. Arvind Bansal (Chair)
• Dr. Cheng-Chang Lu
• Dr. Paul Wang

2
Motivation

• Facial expressions form the major modality in
  human communication.
• Although it takes little effort for humans to
  recognize facial expressions, it is rather
  difficult to develop an automated system to do
  such task.
• The major issue is the lack of detailed
  mathematical integration of constrained muscle
  movement, constrained skull movement, and skin
  deformations.
• Application
• -Medicine Facial tissue surgical simulation
• -Teleconferencing Transmitting receiving
  facial images
• -Game industry Realistic games
• -Media and art Movies A bugs life, The
  Mummy

3
Previous Work

• image manipulation
• Image morphing
• -It restricts by the number of input
  images
• video sequences
• -It is difficult to ensure smooth
  transitions between these V.sequences
• geometry manipulation
• geometric interpolations
• -Combinations of independent face
  motions are difficult to produce
• parameterizations
• - rarely produces human expressions when
  a conflict between parameters occurs
• muscle based modeling
• - It has improved realism, but they do
  not capture interaction of all the muscles,,
  Skelton movement, and skin deformations.

4
Facial Action Coding System

• FACS is a set of all possible basic actions
  performable on the human face. An action unit
  (AU) is a single action or movement for the face

5
This Research

• A new XML based language FEML (Facial Expression
  Markup Language) has been developed that
  integrates the emotion with the speech. that is
  suitable for tele-presence
• The 3D facial simulating a detailed anatomical
  model that integrates the constrained muscle
  movement does reconstruction, bone movement, and
  skin movement.
• the whole speech file and the image of the
  announcer is downloaded from the server end, and
  displayed at the client end.
• Facial expression synthesis has been modeled as a
  weighted combination of the contraction of a set
  of typical facial muscles.
• The idea is to build multi-layer structure
  involving trusses, springs, and 3D meshes to
  simulate the dynamic behavior of different types
  of muscles and soft tissue needed to show the
  emotions and visemes.
• Language used MS Visual C 7.0 using
  Text-tospeech conversion

6
Contributions

• A new multilayered approach based upon the
  integration of finite elements and trusses has
  been developed to model 3D facial expressions.
• The use of 2D trusses to model sheet muscles
  improves realism as muscles contract in all the
  direction when the force is applied.
• An XML based markup language has been developed
  for transmitting facial expressions and speech
  over the Internet.
• The system supports smooth transition from one
  emotion to another emotion, and also supports
  blending of two emotions.
• The system supports limited variations in speech
  tone and length with emotions.
• Multiple faces can be superimposed automatically.

7
Multimedia Transmission over the Internet

• Multimedia transmission requires representing an
  animation as a sequence of frames where each
  frame contains multiple objects
• buffering is needed at the server end to avoid
  disk access delay
• delay and the client end to reduce jitters caused
  by bandwidth variations in the transmission line
  and queuing delays
• Jitter caused either by loss of frame, or the
  delay in rendering of frames, or the lack of
  synchronization

8
3D Representation of Object

• Modeling The modeling is described as shaping
  individual objects that are later used in the
  scene.
• Scene layout setup Scene setup involves
  arranging virtual objects, lights, cameras and
  other entities on a scene which will later be
  used to produce a still image or an animation
• meshes The process of transforming
  representations of objects
• Rendering It is the final process of creating
  the actual image or animation from the prepared
  scene

9
Text-to-Speech Generation

• The user inputs text that is automatically
  translated into the corresponding phonetic
  symbols
• After receiving the sequence of phonemes, viseme
  table is looked up to find out the mouth
  deformation
• Animation is produced by interpolating between
  the visemes
• The two processes of phoneme to sound conversion
  and face visualization are launched
  simultaneously to synchronize the face movement
  with the sound

10
Facial Muscles and their Coordination
11
Muscle types

• Linear/parallel muscle It has a linear shape. It
  has a common emergence point (fixed point) in
  bone and another end pulls the facial tissue and
  skin in a linear and angular direction.
• Sphincter muscle These muscles have ellipsoid
  shape, Both the origin and the insertion point
  attaches to the skin, and contracts around a
  virtual center
• Sheet muscle There are multiple fibers with no
  localized fixed point. Sheet muscles can be
  rectangular or triangular. The fiber bundle
  stretches or contracts non-uniformly along two
  axis

12
Elastic Bodies and Tensors

• Stress is defined as force per unit area. It has
  the same units as pressure.
• Strain is defined as the amount of deformation an
  object experiences compared to its original size
  and shape.
• Elastic material deforms under stress but returns
  to its original size and shape when the stress is
  released.
• A truss is a slender member that can support an
  axial load. The axial members connected by pin
  joints that restrict the translation at the
  constrained node but allow free rotation in any
  direction

13
Finite elements method

• The finite element method is a numerical
  techniques to solve complex set of differential
  equations involving complicated geometries where
  it is generally not possible to obtain an
  analytical mathematical solution.
• ?F K ? U.
• The stiffness matrix K is a function of the
  geometry of the member.
• The displacements are calculated using the
  relationship above

14
XML ? EXtensible Markup Language

• XML is a markup language used for complex
  structures using user defined tags in a flat
  textual form.
• The Document Object Model is a platform- and
  language-neutral interface that allows programs
  and scripts to dynamically access and update the
  content, structure and style of documents.