Visual Semantics and Ontology of Eventive Verbs

1
Visual Semantics and Ontology ofEventive Verbs

• Minhua Eunice Ma and Paul Mc Kevitt
• School of Computing and Intelligent Systems
• Faculty of Engineering
• University of Ulster, Magee
• Derry/Londonderry, N. Ireland

2
Outline

• Background CONFUCIUS
• Previous verb taxonomies
• Visual semantics verb classes
• CONFUCIUS ontology of verbs
• Current status of implementation
• Relation to other work
• Conclusion future work

3
Architecture of CONFUCIUS
Natural language sentences
Surface transformer
Media allocator
Prefabricated objects (knowledge base)
LCS lexicon WordNet
Natural Language Processing
Text To Speech
Sound effects
Language knowledge
3D authoring tools, existing 3D models
character models
semantic representations
mapping
Animation engine
visual knowledge
Visual knowledge (3D graphic library)
Synchronising fusion
3D world with audio in VRML
4
NLP in CONFUCIUS
Connexor FDG parser
FEATURES
Disambiguation
WordNet LCS database
Post-lexical temporal relations
Lexical temporal relations
5
Previous verb taxonomies

• Grammatical categorisation valency
• Thematic roles (Fillmore, 1968 Jackendoff, 1990
  Halliday, 1985 Dowty, 1991)
• Aspectual classes (Vendler, 1967 Stede, 1996)
• Semantic verb classes in WordNet (Fellbaum,
  1998)
• Levins (1993) verb classes
• Dimension of causation
  (Asher Lascarides, 1995)

6
Grammatical categorisation valency

• Subcategorisation description of verb categories
  in LDOCE (Longman Dictionary of Contemporary
  English)
• D ditransitive
• I intransitive
• L linking verb with complement
• T1 transitive verb with NP object
• T3 transitive verb with infinitival clause as
  object

7
Grammatical categorisation valency

• Subcategorisation description of verb categories
  in LDOCE (Longman Dictionary of Contemporary
  English)
• Syntactic valency
• Obligatory valency fillers (complements)
• e.g. subject, object
• Optional valency fillers (adjuncts)
• e.g. temporal, locational adjuncts
• Semantic valency (Leech, 1981)

8
Thematic roles

• Other names theta-role, case role, deep
  grammatical function, transitivity role, valency
  role, case frame
• Extend syntactic analysis into semantic domain to
  capture roles of participants
• surface case (nominative, accusative)
• surface function (subject, object)
• Classifying verbs based on thematic roles (Dixon,
  1991)

Thematic roles (e.g. agent, patient/theme,
instrument, source, goal, place)
9
Aspectual (temporal) classification

• Vendlers (1967) verb classes
• activities run, swim, sleep, cry
• statives love, hate, know
• achievements arrive, win, find, die
• accomplishments build (a house), write (a book)
• Stedes (1996) MOOSE ontology
• Formal ontologies DOLCE, SUMO, and CYC assume
  traditional aspectual (temporal) classification
  for events

10
Aspectual (temporal) classification

• Vendlers (1967) verb classes
• Stedes (1996) ontology of MOOSE
• Formal ontologies DOLCE, SUMO, and CYC assume
  traditional aspectual (temporal) classification
  for events

11
Aspectual (temporal) classification

• Vendlers (1967) verb classes
• Stedes (1996) ontology of MOOSE
• Formal ontologies DOLCE, SUMO, and CYC assume
  traditional aspectual (temporal) classification
  for events

12
Semantic verb classes in WordNet

• Taxonomic approach based on pure lexical
  semantics
• Reveal semantic organisation of lexicon in terms
  of lexical semantic relations
• Top nodes of WordNets verb file

13
Levins (1993) verb classes

• Theoretic ground -- semantic/syntactic
  correlations verbs with similar meaning
  (identical LCSs in terms of specific meaning
  components) show same syntactic behaviors

Verbs of inherently directed motion arrive,
come, enter Leave verbs leave, abandon,
desert Manner of motion verbs roll, run, sneak,
waddle Verbs of motion using a vehicle bike,
drive, fly Chase verbs chase, follow,
track Accompany verbs accompany, escort, guide
Waltz verbs clog, polka, waltz
Verbs of motion
14
Dimension of causation

• Asher and Lascarides (1995) dimension of
  causation-change
• causation and change are specified along four
  dimensions locative, formal, matter, intentional

locative
cause
formal
matter
change
intentional
loc-change obj of put
intent-cause sub of amuse
fml-cause sub of build
mtr-cause sub of paint
fml-change obj of build
mtr-change obj of paint
intent-change obj of amuse
loc-cause sub of put
15
Visual semantics verb classes

• Visual factors concerning verb categorisation
• Visual valency
• Somatotopic factors in visualisation
• Level-of-detail of visual information
• Verbs belonging to same class in
  the classification
• Visual synonyms
• Substitutable in same set of animation keyframes
• Visualisation of action verbs is effective
  evaluation of the classification

16
Visual valency

• Capacity of verb to take specific number and type
  of visual arguments in language visualisation
  (3D animation)
• valency filler -- visual role
• 2 types of visual roles requiring different
  processes in visualisation
• human (biped articulated animate entity)
• object (inanimate entity)
• Visual valency overlaps with syntactic semantic
  valency
• Visual modality requires more obligatory roles
  than surface grammar or semantics

17
Somatotopic effectors of action verbs

• Theoretical ground execution/perception/visualisa
  tion of action verbs produced by same somatotopic
  effector activate same parts of cortex
• Distinguish facial expression (e.g. lip movement)
  body posture (arm/leg/torso) in our ontological
  system
• Further divisions like distinction between
  upper/lower arm, hands, even fingers are
  possible

torso bow
18
Level-Of-Detail (LOD)basic-level verbs
troponyms
19
CONFUCIUS verb taxonomy
2.2.1. Action verbs 2.2.1.1. One visual
valency (the role is a human, (partial) movement)
2.2.1.1.1. Biped kinematics arm actions
(wave, scratch), leg actions (walk, jump, kick),
torso actions (bow), combined actions (climb)
2.2.1.1.2. Facial expressions lip movement,
e.g. laugh, fear, say, sing, order 2.2.1.2.
Two visual valency (at least one role is human)
2.2.1.2.1. One human and one object (vt. or
vi.instrument) e.g.
throw, push, kick, open, eat, drink, bake,
trolley 2.2.1.2.2. Two humans, e.g.
fight, chase, guide 2.2.1.3. Visual valency
3 (at least one role is human)
2.2.1.3.1. Two humans and one object (inc.
ditransitive verbs), e.g. give, show
2.2.1.3.2. One human and 2 objects (vt. object
implicit instr./goal/theme)
e.g. cut, write, butter, pocket, dig, cook
2.2.1.4. Verbs without distinct visualisation
when out of context verbs of trying, helping,
letting, creating/destroying 2.2.1.5. High
level behaviours (routine events), political and
social activities e.g. interview, eat out (go
to restaurant), go shopping
20
Text-to-Animation of single sentences

• Collision detection example (contact verbs hit,
  collide, scratch, touch), no human role involved
• The car collided with a wall.
• using ParallelGraphics VRML extension--object-to-
  object collision
• non-speech sound effects
• H-Anim examples action verbs
• 3 visual valency verbs
• John gave Nancy a loaf of bread.
• John put a cup of coffee on the table.
• H-Anim Site node
• locative tags of object (on_table tag for table
  object)
• 2 visual valency verbs
• John pushed the door.
• John ate the bread.
• Nancy sat on the chair.
• For more demos, please visit http//www.infm.ulst.
  ac.uk/eunice/3D_anim.html

21
Relation to other work

• Categorise verbs from visual semantics
  perspective
• Language visualisation in CONFUCIUS provides
  independent criteria for identifying classes of
  verbs sharing certain aspects of meaning,
  i.e. semantic/visual
  correlations
• Relation to Levins verb classes

2.2.1.3.2, visual valency3 2.1.2, visual
valency2
Carol cut the whole wheat bread. Whole wheat
bread cuts easily.
Verbs of cutting
1 to N
2.2.1.3.1 visual valency3
Verbs of sending carrying Verbs of change of
possession
Nancy brought the book to John. Nancy gave the
book to John.
N to 1
22
Conclusion future work

• Categorise verbs from visual semantic perspective
• Provides independent criteria for identifying
  classes of verbs based on semantic/visual
  correlations
• Visual semantic analysis of eventive verbs
  revealed striking influences in taxonomic verb
  tree
• Various criteria ranging from visual valency,
  somatotopic effectors, to LOD are proposed

• Evaluation issues using automatic animation
  generation psychological experiments
• Discourse level interpretation