Synthetic Actors

1
Synthetic Actors
2
Synthetic Actors

• Think Gollum in Lord of the Rings, but without
  Andy Serkis (sorry Andy!).
• Particular area of interest is in live stage
  performances working within the constraints of
  a script, but adapting the performance as
  necessary.
• Live theatre is developed via rehearsals - so
  this is how the synthetic actors will develop as
  well.

3
Synthetic Actors

• Think Gollum in Lord of the Rings, but without
  Andy Serkis (sorry Andy!).
• Particular area of interest is in live stage
  performances working within the constraints of
  a script, but adapting the performance as
  necessary.
• Live theatre is developed via rehearsals - so
  this is how the synthetic actors will develop as
  well.

4
Existing Components

• The current experimental system is made up of a
  number of (predominately open source and
  cross-platform) components 'glued' together
• Speech generation Microsoft Speech SDK /
  Festival / Festival Light
• Speech recognition Microsoft Speech SDK /
  Sphinx
• Graphics output Unreal Tournament with the
  Gamebots mod
• Vision system primarily routines from Intel's
  OpenCV library
• Tripod system Lego Mindstorms SDK and Baum's
  NQC
• Interval engine provides timing information

5
(No Transcript)
6
Improvements to system

• Performance related particular models,
  movements, vision processing, and so on.
• Unreal Tournament 2003 rather than Unreal
  Tournament
• Drop Microsoft Speech SDK, add affect to speech
• Tools to work within a rehearsal (understanding
  the director, for example)
• Core ('brain') of the system

7
Research

• The bulk of my research will be carried out over
  the next 12 months.
• Essentially, a stage performance that includes
  one or more synthetic actors will be produced
  over a seven month period (with a performance for
  the public at the end).
• At weekly rehearsals, the synthetic actor will
  try and successfully (in the view of the
  director) improvise the performance (staying
  within the script) so that the performance is
  improved. Various changes will be made to the
  synthetic actor system during the periods between
  rehearsals, and these will then be evaluated.

8
More information

• If you would like some more substantial
  information about what I'm doing, why I'm doing
  it, and what my plan is, please see
• http//www.massey.ac.nz/tameyer/research/computer
  theatre
• This includes papers at varying levels of
  technicality, from newspaper article to research
  proposal.

9
Demonstration

• Any questions at this point?
• Before I conclude, I will demonstrate some of the
  existing system's capabilities remember that
  the focus is on the (yet to be created) core, and
  not these parts i.e. they are meant to be
  sufficient, not excellent.

10
(No Transcript)
11
Image Processing

• Captures video via USB2 from two standard
  webcams can handle about 150 frames per
  second if no processing is done.
• Hands over this information to a series of
  processing functions currently these include
  motion detection and face detection. This drops
  the framerate to about 5 frames per second!

12
Motion Detection

• From Intels Open Computer Vision library
• Uses motion history images basically an image
  that contains a history of recent frames.
• Generates location(s) of motion, and direction(s).

13
MHI Example
14
Face Detection

• From Intels Open Computer Vision Library
• Uses a cascade of boosted haar classifiers in
  other words a series of weak classifiers to
  determine whether a face is present.
• Can be used to detect other objects, with
  training.

15
Face Detection Examples


16
(No Transcript)