Interactive Media Group iMG

1
Interactive Media GroupiMG
Ali Arya Carleton School of Information
Technology Department of Systems and Computer
Engineering
2
CSIT

• Carleton School of Information Technology
• Areas of research
• Networking
• Multimedia
• Independent undergrad program
• Graduate program through Dept. of Systems and
  Computer Engineering

3
iMG _at_ SCE

• Interactive Media Group (iMG)
• Part of Department of Systems and Computer
  Engineering (SCE)
• Motivations new multimedia systems and
  interactive technologies
• Computer games
• Interactive virtual environments
• Interactive web applications
• New human-computer interfaces
• New media and digital art

4
Areas of Research

• Multimedia Systems and Applications
• Human-Computer Interaction
• Computer Graphics and Animation
• Virtual Worlds and Virtual Characters
• Virtual and Digital Art

5
Some Research Projects

• iFACE, facial animation engine
• SPICE, interactive virtual dance
• Interactive Story Spaces
• Digital Libraries
• Medical Planning
• Game and Simulation
• Automotive Displays and Multimodal Interaction

6
img.csit.carleton.ca
7
Animated Images
8
Milestones

• Disney studios
• Cel animation
• Computer generated animation in early 60s
• Mr. Computer Image ABC with Scanimate
• 1970s
• Voyager 2 by JPL (James Blinn)
• Star Wars by ILM (George Lucas)

9
Pong by Atari (1972)

• The story begins.
• The first commercially successful video game
• Table tennis

10
Games to Remember
11
Commodore 64 - 1982
12
Computer Games

• computer game
• a game composed of a computer-controlled virtual
  universe that players may interact with in order
  to achieve a goal (or set of goals)
• video game
• a computer game where a video display is the
  primary feedback device

13
Gaming Platforms

• PC
• Windows
• Mac
• Console
• Commodore 64 and Atari ?
• X-Box from Microsoft
• PlayStation from Sony
• Game Boy from Nintendo
• Arcade

14
Milestones

• 1980s
• Disney TRON combined live action with 3D computer
  animation
• Atari and video games
• Visual effects, Star Wars
• Genesis Effect in Star Trek II by ILM
• Pixar RenderMan
• Little Mermaid (89) the last traditional film

15
Geris Game by RenderMan
16
Milestones

• Early 1990s
• Terminator II (91) by ILM
• 1st to include morphing and human simulation
• Jurassic Park (93) by ILM
• Babylon 5 (93)
• 1st 3D animation, made by Amiga and Mac
• Cocacola Polar Bear (93) commercial
• Disney CAPS, Lion King

17
Coca-cola Polar Bear
18
Doom by id Software (1993)

• Pioneer in 3D graphics
• Next generation of Wolfenstein 3D
• Re-usable engine
• Violence, satanic imagery, and other
  controversies

19
Milestones

• Late 1990s
• Toy Story (95)
• 1st computer generated feature film
• DreamWorks vs. Disney/Pixar
• Antz, Prince of Egypt
• Bugs Life, Mulan, Toy Story 2
• Fantasia 2000 for IMAX
• Armagedon (98)

20
Feature Films
21
Game Genres

• Role-playing games (RPG)
• Third-person shooter (TPS)
• First-person shooter (FPS)
• Sports and racing
• Simulation (e.g. flight)
• Strategy

22
Flight Simulators
23
Racing Game
24
FPS Games
25
The Sims by Maxis (2000)

• Simulated characters, simulated life
• SimCity (1989)
• God-game!

26
Multimedia Webpage
27
Milestones
21st Century
28
Interactive Webpage
29
Social Agents

• Virtual characters
• Interactive
• Social
• Realistic or stylistic
• Storyteller
• iFACE Demo
• Arya and DiPaola, 2005
• A little bit of history

30
8th Art

• Ricciotto Canudo (1879-1923)
• Seventh Art (Cinema)
• Three arts of space - painting, architecture and
  dance
• Three arts of time - music, theatre, and
  literature
• Multimedia, the 8th art!
• One art of space photography
• Two arts of time - music and literature
• One art of time and space and light - cinema

31
Interactive Multimedia Presentation

• Live-action vs. animation
• Sequential vs. interactive
• Merging of different domain into one medium
• Interactivity
• Face-based communication
• Verbal and non-verbal
• Social user experience

32
Characters
33
Making Faces ?

• Virtual Characters Substitute for real human
  figures in a social interface
• Virtual Environments
• Training and Customer Service
• Games
• Special Effects
• Video Phones

34
A Social User Interface
35
Another Example of SUI
36
Social Communication

• Creating a social exchange
• Natural aspects of human communication
• Verbal cues
• Gestures
• Facial expressions
• Communication is a major functionality of
  computer-based systems
• Explicitly or implicitly

37
Social Conversational Agent

• Flexibility and availability of virtual
  characters vs. live-action
• Avatar as a visual agent
• Interactivity and understandability
• As opposed to traditional UI objects
• Front end
• Realistic or stylistic character
• Back end
• Intelligence, rules of interaction, scenario

38
Face Multimedia Object(Arya and DiPaola, 2004)

• Object-based multimedia standards
• MPEG-4
• Media types
• Audio
• Video
• Face-centric applications
• Face media type

39
FMO Advantages

• Encapsulation of face functionality
• Coordination of face-related functions
• Ease of application development
• Through dedicated API and scripts
• Independence of characters from other objects and
  media type
• Autonomous behavior
• 3D and 2D with the same interface
• Hierarchical Modularity

40
Computer Facial Animation

• Fredric Parke, early 1970s, first computer model
  for face
• Some applications
• Computer games
• Social conversational agents
• Visual communication
• Video phones, etc
• Visual effects in movies

41
Geris Game by RenderMan
42
Final Fantasy ?
43
Modeling/Animation Methods

• 2D
• Morphing and image transforms
• No 3D computation, usually more realistic, but
  might need a database of images
• 3D
• Direct parameterized
• Physically-based (skin, muscle, bone)

44
Morphing
Mapping of each pixel is a function of the
mappings of some Control Points. Needs manual
setting of controls.
45
Talking Head (Ezzat et al.)
Optical Flow The mapping of each pixel of source
image is calculated based on the similar point
found in target images. Works for one set of
images (one person, one view) only.
46
Feature-based Image Transformation (ShowFace,
Arya et al., 2002)
T (Fs ,M) M Ft - Fs
Learning
Apply T Warp image
Runtime
47
3D Faces

• Models
• Physically-based
• Parameterized
• Performance-based
• Data sources
• 3D Digitizer
• Laser scanner
• Photogrammetry

48
3D Digitized Model
49
Scanned Range Data
50
Scanned Colour Data
51
Adapted Mesh
52
3D From 2D
53
3D Face Models (Virtual Human Director, Lee et
al.)
54
FACS

• Facial Action Coding System (FACS)
• 64 basic facial Action Units (AUs)
• Inner Brow Raiser
• Mouth Stretch
• Head Turn Left
• Originally not for computer graphics and
  animation, but used extensively

55
MPEG-4

• Motion Picture Experts Group
• Standards
• MPEG-1 (standard audio/video compression)
• MP3 (MPEG-1 Layer-3)
• MPEG-2 (streaming)
• MPEG-4 (object-based multimedia)
• MPEG-7 (content description)

56
iFACE System (Arya and DiPaola, 2004)

• Interactive Face Animation Comprehensive
  Environment
• Face engine
• Modular multi-layer framework
• Authoring and runtime components
• PC version
• .NET framework
• C
• DirectX
• Console version? Mobile version?

57
iFACE Architecture
Application Layer (GUI App, Web Page, etc)
Wrapper Layer (Form Controls, ActiveX, etc)
Stream Layer (Frame Reader/Writer, Mixer, etc)
Data Layer (Geometry, Behaviour, Media)
58
iFACE Parameter Spaces
Events
Knowledge
Geometry
Mood
Personality
59
iFACE Geometry
60
Different Geometry Types

• Layered interfaces allow working with different
  data types in a similar way.

61
Face Regions
62
Believability vs. Realism

• Disney animation makes audience really believe
  in characters. There is a special ingredient in
  our type of animation that produces drawing that
  appear to think and make decisions and act of
  their own volition it is what creates the
  illusion of life.
• F. Thomas and O. Johnston

63
Believability

• Believability. That is what we were striving
  for.
• Chuck Jones (Warner Bros.)
• From the earliest days, it has been the
  portrayal of emotions that has given the Disney
  characters the illusion of life.
• Thomas and Johnston (Disney)

64
iFACE Behaviour (Arya DiPaola, 2005)

• Knowledge
• Scenario (character-independent actions)
• Rules of interaction
• Personality
• Long-term and character-dependent
• Mood
• Short-term emotional states

65
Emotions

• Charles Darwin
• Universality of emotions
• Expression of the Emotions in Men and Animals,
  1872
• Paul Ekman and W.V. Friesen
• Analysis of facial expressions in different
  cultures
• Facial Action Coding System (FACS), 1978

66
FACS Sample Action Units
67
Universal Emotions

• Sadness and agony
• Joy
• Anger
• Surprise
• Fear
• Disgust
• Contempt

68
Facial Expressionssee Paul Ekman, Emotions
Revealed, 2003
69
Homers Expressions
Clueless Annoyed Playful Pleased Stupified Mad
about Boy
70
Facial Action for Expressions

• Joy
• eye-lid tightened
• cheeks raised
• brows slightly lowered
• wrinkles around eyes (especially corners)
• Sadness
• upper eye-lid lowered
• lower eye-lid raised
• inner brows raised
• lip cornered lowered
• lower lip raised slightly
• cheeks raised

71
Russells Mood Model
72
Expression Units

• Research on Expression Units (EUs)
• Collective set of all small facial actions
  involved in expressions
• Associate expression units to parameters
  (Russells dimensions)
• Create different expressions directly from
  parameters and EUs

73
Expression Units Example
Exp-1, average, Exp-2, blend using EUs
74
Mood-dependency

• Moods affect the way facial actions are performed.

75
Do We Really Depend on the Kindness of
Strangers?
76
Static Visual Cues

• Standard Moods
• Joy
• Sadness
• Anger
• Fear
• Disgust
• Surprise
• Custom-defined Expressions
• Eye, brow, mouth and head rest position
• Speaking out of a corner of mouth
• Gaze (looking into camera)
• Gender
• Age
• General Appearance
• Round face, Full lips, Eye separation, Nose
  shape, Brow thickness, etc
• Baby face vs. mature
• Other attraction-related features

77
Dynamic Visual Cues

• 3D Head Movements
• Frequency
• Duration
• Direction (yaw, pitch, roll)
• Nodding (especially in emphasis for speech)
• Laughing
• Raising eyebrows
• Frowning
• Symmetric vs. one-sided
• Frequency
• Duration
• Gaze shift
• Blinking
• Frequency
• Duration
• Frequency and duration of expressions listed in
  Static Visual Cues

78
Wigginss Circumplex
79
iFACE Personality

• Facial actions associated with Wiggins
  dimensions (visual cues)
• Trigger facial actions
• Speech energy (Impulse and Emphasis)
• Randomly selected from the associated set
• Random/periodic
• Blinking and expressions
• Speech content
• Research

80
Test Animations

• Asian Female
• White Female
• Asian Male
• White Male

81
Cues Emotions
82
Cues Slow Actions
83
Cues Fast Actions
84
Sample Mixed-Actions

• Exhibitionist
• High Dominance, High Affiliation
• Shy
• Low Dominance, Low Affiliation
• Competitor
• High Dominance, Low Affiliation
• Helper
• Low Dominance, High Affiliation

85
Mixed-Actions Details

• HD-HA
• Joy, Fast Turn, Fast Tilt, Fast Double-Brow Raise
• LD-LA
• Contempt, Slow Nod, Fast Avert Gaze, Down Head
  Rest
• HD-LA
• Anger, Pride, Fast Single-Brow Raise
• LD-HA
• Surprise, Fast Blink, Slow Double-Brow Raise,
  Right Head Rest

86
Mixed-Action Results
87
iFACE Software Architecture
88
Sample iFACE Applications

• MusicFace
• Music-driven facial choreograpy
• COMPS
• Collaborative online multimedia problem-solving
  simulation
• Problem-based learning for medical students

89
MusicFace (DiPaola Arya, 2004)

• Affective communication remapping
• Extracting affective info from music and creating
  facial actions based on it (Demo)

90
COMPS (DiPaola, Arya et al. 2005, Demo)
91
More iFACE Applications

• Storytelling Masks
• Evolving Faces

92
Recent Projects

• Painterly animation
• Create facial (and non-facial) animation similar
  to the style of a painter
• Non-photorealistic rendering (sample)
• Interactive stories
• Story space elements with relations
• Virtual communities
• Rules of behaviour
• Characters with personality

93
Motion Capture
94
Native Dance
95
Digital Pow-wow

• Recording the motion data
• Defining style parameters
• Establishing computational style models
• Developing a designer software
• Creating a web service for making and displaying
  virtual shows