100 Day Review

1
SceneMaker Intelligent Multimodal
Visualisation of Natural Language Scripts

• 100 Day Review
• Eva Hanser
• Supervisors

Prof. PAUL MC KEVITT Dr. TOM LUNNEY Dr. JOAN
CONDELL
School of Computing Intelligent Systems,
Faculty of Computing Engineering, University of
Ulster
2
Presentation Outline

• Project Aims Objectives
• Literature Review
• Project Proposal
• Conclusion

3
Project Aims Objectives

• Extraction of Visual and Emotional Information
  from Play/Film Scripts
• Structuring for Visualisation Purposes
• Synchronised Presentation in all Modalities
  (Vision, Speech, Illumination, Camera Work)
• Compelling, Life-like Animations

4
Literature Review

• 1. Detecting Emotions and Personality
• 2. Modeling Affective Behaviour of Virtual
  Actors
• 3. Multimodal Visualisation of 3D Scenes

5
Literature Review1. Detecting Emotions and
Personality

• Which Emotions exist? Which are needed for
  realistic animation?
• How to categorise all Emotions and Personality?

6
Literature Review1. Detecting Emotions and
Personality

• Psychological Theories of Emotion
• 6 Basic, Universal Emotions (Ekman and
  Rosenberg, 1997)
• Happiness, Sadness, Anger, Disgust, Fear,
  Surprise
• Pleasure-Arousal-Dominance (PAD)
  ModelIntensity Value Scale
• OCC Model (Ortony et al. 1988) 22 Emotion
  Categories arranged by Goals, Likelihood,
  Familiarity
• Personality ModelOCEAN (De Raad, 2000)
  Openness, Conscientiousness, Extroversion,
  Agreeableness, Neuroticism
• Social Roles
• Story Character Roles

7
Literature Review 1. Detecting Emotions and
Personality

• Fuzzy Rule Based System (Su et al., 2007)
• Annotation of Scene Scripts
• XML-based Scripting Languages
• BEAT (Cassell et al., 2001), MPML (Breitfuss et
  al., 2007)
• Tags to Control Body Language Utterances
• ALMA (Gebhard, 2005), SCREAM (Prendinger and
  Ishiyuka, 2002)
• Emotion Expression Mapped to Body Language

8
Literature Review 2. Modeling Affective Virtual
Actors

• Gandalf, Rea (Vilhjálmsson and Thórisson, 2007),
  Max (Kopp et al., 2008), Greta (Pelachaud, 2005)
  Face-to-Face ConversationLanguage
  Understanding and GenerationGaze Responding to
  GesturesSpeaker and Listener FeedbackFace and
  Hand Expressions

Casino Virtuell
Greta
Image Source Pelachaud, 2005
Image Source Gebhard and Schröder, 2008
9
Literature Review 3. Visualisation of 3D Scenes

• Sound and Music Effects
• Cinematic Principles (Camera, Lighting, Staging)
• EML Expression Mark-up Language (De Melo and
  Paiva, 2006) Multimodal Expressions for
  Virtual Human and Environment

10
Literature Review 3. Visualisation of 3D Scenes

• CONFUCIUS (Ma, 2006) Multimodal 3D Animation
  of NL stories

• WordsEye (Coyne and Sproat, 2001) Static 3D
  scenesRestricted Language Input

Image Source Coyne and Sproat, 2001
Image Source Ma, 2006
11
Literature Review 3. Visualisation of 3D Scenes

• Virtual Theatre Interface(Virtual Theatre,
  2004) Different Audience ViewsStage Lighting
  and Positioning

• Madame Bovary(Cavazza et al., 2007) Scene
  Projection into RoomInteraction with Human Actor

Image Source Cavazza et al., 2007
Image Source Virtual Theatre, 2004
12
Project Proposal
1. THE PROJECT IDEA 2. SCENEMAKER
ARCHITECTURE 3. SOFTWARE ANALYSIS 4. RELATION
TO OTHER WORK 5. PROJECT SCHEDULE
13
Project ProposalThe Project Idea

• Affective Actors
• Emotionally Influenced Body Language
• Environmental Expression
• Emotionally Influenced Multimedia Music,
  Sound, Illumination, Timing and Camera Work
• Software Prototype SceneMaker
• Generates 3D Scenes from Screenplays
• Web Based User Interface for Manual Manipulation
• Accessible on Mobile Devices
• Testing with Sample Screenplays of Different
  Genres

14
Project ProposalSceneMaker Architecture
15
Project ProposalSoftware Analysis

• Constructionist Design Methodology (CDM)
• Constructionist AI Methodology (CAIM)
  (Thórisson, 2007)
• Psyclone (Thórisson et al., 2004) , OpenAIR
  (Thórisson, 2007)
• Unified Messaging Format between
• Various Input and Output Modules/Modalities
• CONFUCIUS (Ma, 2006)
• Natural Language Processing
• Animated 3D Scenes with H-Amin 3D Models
• CSLU Text-To-Speech Synthesis (Sutton et al.,
  1998)

16
Project ProposalRelation to Other Work

• Emotion Visualisation from NL Text
• Emotional Influence on Various Modalities
• Mobile User Interface

17
Relation to Other Work
18
Project ProposalProject Schedule
19
Conclusion

• Multimodal System, SceneMaker, for Automatic
  Visualisation of Affective Expressions in Story
  Scripts
• Assisting Directors, Drama Students, Writers and
  Animators to Test Ideas
• User Interface for Manual Adjustments
• Existing Systems Partially Solve Aspects of NLP,
  ECAs, Emotion Modeling and Multimodal
  Storytelling
• Test Basis Screenplays of Different Genres

20
Thank you for your attention.

• Questions or Comments?

www.infm.ulst.ac.uk/eva