A Common Ground for Virtual Humans:

1
A Common Ground for Virtual Humans

• Using an Ontology in a Natural Language Oriented
  Virtual Human Architecture

Arno Hartholt (ICT), Thomas Russ (ISI), David
Traum (ICT), Eduard Hovy (ISI), Susan Robinson
(ICT) 05/30/2008
2
Overview

• Virtual Humans Project
• Challenge
• Virtual Humans Architecture
• Ontology
• Conclusion
• Future Work

3
Virtual Humans Project
4
Virtual Humans Project
Natural Language
Natural Language
Natural Language
Natural Language
Reasoning and
Knowledge
Reasoning and
Knowledge
Reasoning and
Knowledge
Reasoning and
Knowledge
Dialogue
Dialogue
Dialogue
Dialogue
Emotion
Representation
Emotion
Representation
Emotion
Representation
Emotion
Representation
Conversational
Conversational
Natural Language
Natural Language
Conversational
Conversational
Natural Language
Natural Language
Minor
Minor
Understanding
Understanding
Minor
Minor
Understanding
Understanding
Characters
Characters
Characters
Characters
Virtual
Virtual
Virtual
Virtual
Natural Language
Natural Language
Human
Human
Speech
Speech
Natural Language
Natural Language
Human
Human
Speech
Speech
Generation
Generation
Processing
Processing
Generation
Generation
Architecture
Architecture
Processing
Processing
Architecture
Architecture
Non
-
Verbal
Non
-
Verbal
Behavior
Behavior
Non
-
Verbal
Non
-
Verbal
Behavior
Behavior
Behavior
Behavior
Animation
Animation
Behavior
Behavior
Animation
Animation
Sensing
Sensing
Non
-
Verbal
Non
-
Verbal
Non
-
Verbal
Non
-
Verbal
Sensing
Sensing
Non
-
Verbal
Non
-
Verbal
Non
-
Verbal
Non
-
Verbal
Behavior
Behavior
Behavior
Behavior
Behavior
Behavior
Behavior
Behavior
Understanding
Understanding
Generation
Generation
Understanding
Understanding
Generation
Generation
5
Challenge Knowledge Representation

• Module-specific representation
• How to communicate between modules?
• How to make sure translations are correct?
• Uniform representation
• Common understanding
• Reuse
• Impoverished or rich?

NLU
Speech
Task Modeling
Dialogue
NLG
Gesture
6
Our Process

• Multi-phase project life cycle
• First, choose individual representations,
  suitable for the state of the art in that area
• Next, bring languages closer together

7
Solution - Old and New Architecture
Old architecture
New architecture
8
Knowledge Representation

• Static knowledge
• Offline authoring
• Social / psychological behavior
• Domain knowledge
• Dynamic knowledge
• Runtime
• Current state / events
• Communication through message protocol

9
Virtual Humans Architecture
Cognition
Speech-act statement Polarity negative Object
market Attribute resource Value
medical-supplies
Mind
Knowledge Management
Intelligent Cognitive Agent
Dialog and Discourse Management
Emotion Model
Task Planner
Domain Specific Knowledge
Natural Language Understanding
Vision Understanding
Body and Affective State Management
Natural Language Generation
Speech-act statement Event move Agent
captain Theme clinic
Domain Independent Knowledge
We have no medical supplies
Body
World State Protocol
Non-Verbal Behavior Generator
Speech Generation
Body Planning
Speech Recognition
Vision Recognition
Smartbody Procedural Animation Planner
I want to move the clinic
Visual Game Engine
Real Environment
Virtual Human
Human Trainee
Environment
10
Focus

• Task model and NLU
• Domain dependent knowledge
• Strong interaction
• Big authoring effort

11
Task Model Tasks States
12
Task Model - Plans
13
Task Model - Whole
14
Task Model - Task

• defTask doctor-moves
• agent doctor
• theme clinic
• event move
• source here
• destination there
• instrument locals
• pre have-transport clinic-here
• add clinic-there
• del clinic-here

15
Task Model - State

• defState clinic-here location clinic
  here ltpgt \
• belief true \
• initialize here \
• probability 0.8 \
• concern doctor 20
• sim-object none

16
Natural Language Understanding (NLU)

• ltSgt i want to move the clinic lt/Sgt
• ltSgt.meta.id mcsw
• ltSgt.mood declarative
• ltSgt.sem.speechact.type statement
• ltSgt.sem.modal.desire want
• ltSgt.sem.type event
• ltSgt.sem.event move
• ltSgt.sem.theme clinic
• ltSgt.sem.source here
• ltSgt.sem.destination there

17
Natural Language Understanding (NLU)

• Framebank sets of semantic frame / utterance
  tuples
• NLU trains on framebank
• Either building frames or retrieving frames

18
Frames Ontology
Core Concepts
Exporters
Module 1
Module 2
uses
uses
Module-specific knowledge
Module-specific knowledge
19
Frames Ontology Core Concepts

• event move
• agent captain-kirk
• theme clinic
• source market
• destination downtown
• object clinic
• attribute location
• value downtown

Task
State
20
Frames Ontology Task Model
Task

• event move
• agent captain-kirk
• theme clinic
• source market
• destination downtown
• pre clinic-location-market
• del clinic-location-market
• add clinic-location-downtown
• object clinic
• attribute location
• value downtown
• belief false
• concern doctor-perez 10

State
21
Frames Ontology Natural Language

• mood declarative
• sem.speechact.type statement
• sem.modality.deontic must
• sem.polarity positive
• sem.type event
• sem.event move
• sem.agent captain-kirk
• sem.theme clinic
• sem.source market
• sem.destination downtown

22
OWL

• Hierarchy
• Assertions on all levels
• Automatic classification

23
OWL Ontology

• General world knowledge
• (generic actions, objects)
• Linguistic structures
• Generic dialogue items

Scenario- independent Ontology

• Scenario actor classes
• Scenario action templates
• (generic preconds, effects...)
• Propositions for scenario

• Specific actors, locations, etc.
• Sentences for scenario

• Actor positions, attitudes, etc.
• Simulation initialization

24
Ontology Action Templates
25
Creating a new move action
26
Creating a new move utterance
27
Conclusion

• Multiple phase life cycle allows quick
  prototyping
• Ontology pros
• Module synchronization
• Formal specification
• Reuse of knowledge
• Reference to data, rather than copy
• Common user interface
• Ontology cons
• Extra learning curve
• Some changes are more difficult
• User interface not ideal

28
Future Work

• Extend natural language capabilities (NLG,
  Lexicon)
• Integrating other modules
• Completing consistency and integrity testing
  routines
• Exploring using existing resources
• We want to enable non-Computer Science people to
  author new scenariosso were investigating the
  optimal point in the trade-off between
• Programming for Non-Experts
• Reduced, simple, scripting languages
• BUT limited in functionality
• Powertools for Experts
• High functionality
• BUT need considerable expertise

29
Questions

• Thank you