Embodiment%20and%20Computation:%20Convergent%20Constraints%20on%20Language%20Use

1
Embodiment and ComputationConvergent
Constraints on Language Use

• Nancy Chang
• nchang_at_icsi.berkeley.edu
• UC Berkeley / International Computer Science
  Institute

2
What does language do?
A sentence can evoke an imagined scene and
resulting inferences

• Harry walked to the cafe.

• Harry walked into the cafe.

• Goal of action at cafe
• Source away from cafe
• cafe point-like location

• Goal of action inside cafe
• Source outside cafe
• cafe containing location

3
Embodied inferences

• The scientist walked into the wall.

The hobo drifted into the house.
The smoke drifted into the house.
4
Metaphorical inference

• France fell into recession. Germany pulled it
  out.
• The economy is moving along at the pace of a
  Clinton jog.
• The Indian Government is stumbling in
  implementing its liberalization plan.

5
Embodied knowledge needed

• What things can serve as containers?
• rooms but not walls (usually)
• How do different entities interact?
• how people and gases interact with houses.
• How are different actions/states related?
• stumbling / walking, falling / containment
• How can actions vary?
• rate, direction, degree of force, etc.

that is, more than predicate-argument
structure! WALK(x), FALL(y), HIT(x,y), etc.
6
Embodiment in language

• Perceptual and motor systems play a central role
  in language production and comprehension
• Theoretical proposals
• Linguistics Lakoff, Langacker, Talmy
• Neuroscience Damasio, Edelman
• Cognitive psychology Barsalou, Gibbs, Glenberg,
  MacWhinney
• Computer science Steels, Feldman

7
Goal computationally precise theories of language
Theory of Language Structure
Theory of Language Use
Theory of Language Acquisition
8
Theory of Language Structure
Theory of Language Use
Theory of Language Acquisition
9
(No Transcript)
10
Simulation hypothesis

• We understand utterances by mentally simulating
  their content.
• Simulation exploits some of the same neural
  structures activated during performance,
  perception, imagining, memory
• Linguistic structure parametrizes the simulation.
• Language gives us enough information to simulate

11
Language understanding as simulative inference
12

• Embodiment and Simulation

What is an idea? It is an image that paints
itself in my brain. Voltaire
13
Neural evidence Mirror neurons

• Gallese et al. (1996) found mirror neurons in
  the monkey motor cortex, activated when
• an action was carried out
• the same action (or a similar one) was seen.
• Mirror neurons found in humans (Porro et al.
  1996)
• Mirror neurons activated when someone
• imagines an action being carried out (Wheeler et
  al. 2000)
• watches an action being carried out (with and
  without object) (Buccino et al. 2000)

14
The Motor System is somatotopically organized
15
The Mirror System

• The mirror system, like the motor system, is
  somatotopically organized.

Buccino et al., 2001
humans watching videos of actions without
objects humans watching same actions with
objects
16
Mirror neurons for language?

• Mirror neurons for specific effectors activated
  during passive listening
• Sentences describing mouth/leg/hand motions
  activates corresponding part of pre-motor cortex
• (Tettamanti et al., forthcoming)
• Verbs associated with particular effectors
  activates corresponding areas of motor
  cortex (Pulvermuller et al. 2001, Hauk et al.
  2004)

17
Psycholinguistic evidence

• Embodied language impairs action/perception
• Sentences with visual components to their meaning
  can interfere with performance of visual tasks
  (Richardson et al. 2003)
• Sentences describing motion can interfere with
  performance of incompatible motor actions
  (Glenberg and Kashak 2002)
• Sentences describing incompatible visual imagery
  impedes decision task (Zwaan et al. 2002)
• Simulation effects from fictive motion sentences
• Fictive motion sentences describing paths that
  require longer time, span a greater distance, or
  involve more obstacles impede decision task
  (Matlock 2000, Matlock et al. 2003)

18
Suggestive evidence

• Mirror system raises possibility of integrated,
  multi-modal representation of actions, along with
  objects and locations
• Global economy exploit existing sensory-motor
  systems for language understanding

19
Computational efficacy

• Embodied representations the norm in robotics!
• Computational representations for lexical
  semantics have been developed for
• Spatial relations (Regier 1996)
• Actions (Bailey 1997, Narayanan 1997)
• Objects / attributes (Roy 1998)
• Metaphor understanding system based on simulation
  (Narayanan 1997)

20
(No Transcript)
21
Embodied lexical semantics
22
Learning System
dynamic relations (e.g. into)
structured connectionistnetwork (based on
visual system)
23
Metaphor system architecture
Target domain
Metaphor maps
Source domain
(Narayanan 1997)
24
Metaphor understanding system

• Indian Government stumbling in implementing
  liberalization plan

25
Missing link grammar!

• Metaphor understanding system demonstrates that
  embodied inferences for difficult case are
  feasible.
• BUT system has no grammar!
• How do we bridge the gap?
• Need a grammatical theory/formalism that can
  served as an interface between linguistic units
  and embodied, dynamic, encyclopedic,
  context-based information (i.e., that can support
  simulation).

26
Language understanding as simulative inference
Linguisticknowledge
27

• 2. Embodied Construction Grammar

It is not enough to say that the mind is
embodied one must say how. Damasio
28
What passes as grammar?

• Syntactic investigation of a given language has
  as its goal the construction of a grammar that
  can be viewed as a device of some sort for
  producing the sentences of the language under
  analysis. (Chomsky 1957)
• Inadequate notion of grammar
• Meaning-free syntax separate from meaning,
  function and processing unanalyzable symbolic
  units
• Inflexible strict word order, strictly
  hierarchical, strictly compositional

29
Whos up to the task?

• Most theories of language are not explicitly and
  systematically tied to action and perception
• Promising exceptions
• Cognitive Grammar / cognitive linguistics
• Construction Grammar
• Typically criticized for being informal / vague
• We borrow liberally from both and formalize.

30
Cognitive Linguistics

• Language is an integral part of cognition which
  reflects the interaction of cultural,
  psychological, communicative, and functional
  considerations, and which can only be understood
  in the context of a realistic view of
  conceptualization and mental processing.

International Cognitive Linguistics Association
website (http//www.cognitivelinguistics.org/aims.
shtml)
31
Key borrowed ideas

• Conceptual structures are embodied.
• Meaning is conceptualization (part of larger
  cognitive system).
• Concepts are grounded in human experience as
  physical, psychological and social beings in the
  world.
• Basic symbolic unit at all levels is a
  form-meaning pair, or construction.
• Syntax is not independent of semantics.
• Phrasal/clausal constructions can contribute
  meaning independently of constituents.

(Lakoff 1987, 1985 Langacker 1991, 1987)
(Fillmore 1988, Kay Fillmore 1999, Lakoff 1987,
Goldberg 1995)
32
Traditional levels of analysis
Pragmatics
Semantics
Syntax
Morphology
Phonology
Phonetics
33
Form-meaning mappings for language
Linguistic knowledge consists of form-meaning
mappings

• Form
• phonological cues
• word order
• intonation
• inflection

• Meaning
• event structure
• sensorimotor control
• attention/perspective
• social goals...

34
Toward a computational account

• Embodied representations that can be simulated
• Previous computational models
• Grammatical formalism for linking the forms of
  language with embodied representations
• Cognitive linguistics
• Construction Grammar
• Detailed descriptions of the processes involved
  in language analysis / simulation

35
Construction Grammar
A construction is a form-meaning pair whose
properties may not be strictly predictable from
other constructions. (Construction Grammar,
Goldberg 1995)
Form
Meaning
block
walk
to
36
Constructions as maps between relations
Complex constructions are mappings between
relations in form and relations in meaning.
Form
Meaning
Mover Motion before(Mover, Motion)
MotionEvent mover(Motion, Mover)
is Action ing before(is,
Action) suffix(Action, ing)
ProgressiveAction aspect(Action, ongoing)
DirectedMotionEvent direction(Motion,
Direction) mover(Motion, Mover)
Mover Motion Direction before(Motion,
Direction) before(Mover, Motion)
37
More on Construction Grammar

• (Goldberg 1995)
• Clause-level patterns correspond to basic events
• transitive Agent Action Patient
• ditransitive (dative) Giver Action Recipient
  Gift
• Economical no explosion of senses
• He pushed the ball.
• He pushed her the ball.
• Novel uses handled more robustly
• Mary pushed the tissue off the table.
• ?Mary sneezed the tissue off the table.
• Mary slept the tissue off the table.

38
Embodied Construction Grammar(Bergen and Chang
2002)

• Embodied representations
• active perceptual and motor schemas (image
  schemas, x-schemas, frames, etc.)
• situational and discourse context
• Construction Grammar
• Linguistic units relate form and meaning.
• Both constituency and (lexical) dependencies
  allowed.
• Constraint-based
• based on feature structure unification (as in
  HPSG)
• Diverse factors can flexibly interact.

39
English
Bowerman Pederson
40
Dutch
Bowerman Pederson
41
Chinese
Bowerman Pederson
42
Harry walked into the cafe.
43
ECG Structures

• Schemas
• image schemas, force-dynamic schemas, executing
  schemas, frames
• Constructions
• lexical, grammatical, morphological, gestural
• Maps
• metaphor, metonymy, mental space maps
• Spaces
• discourse, hypothetical, counterfactual

44
Image schemas

• Trajector / Landmark (asymmetric)
• The bike is near the house
• ? The house is near the bike
• Boundary / Bounded Region
• a bounded region has a closed boundary
• Topological Relations
• Separation, Contact, Overlap, Inclusion, Surround
• Orientation
• Vertical (up/down), Horizontal (left/right,
  front/back)
• Absolute (E, S, W, N)

LM
TR
45
Embodied schemas
schema name
schema Source-Path-Goal roles source path g
oal trajector
schema Container roles interior exterior po
rtal boundary
role name
Boundary
Interior
Trajector
Portal
Source
Goal
Path
Exterior
These are abstractions over sensorimotor
experiences.
46
Embodied constructions
ECG Notation
Form
Meaning
construction HARRY form /hEriy/ meaning
Harry
Harry
construction CAFE form /khaefej/ meaning
Cafe
cafe
Constructions have form and meaning poles that
are subject to type constraints.
47
Representing constructions TO
construction TO form selff.phon ?
/thuw/ meaning evokes Trajector-Landmark
as tl Source-Path-Goal as spg
constraints tl.trajector spg.trajector tl
.landmark spg.goal
local alias
identification constraint
The meaning pole may evoke schemas (e.g., image
schemas) with a local alias. The meaning pole may
include constraints on the schemas (e.g.,
identification constraints ).
48
The INTO construction
construction INTO form selff.phon ?
/Inthuw/ meaning evokes Trajector-Landmark
as tl Source-Path-Goal as spg
Container as cont constraints tl.traj
ector spg.trajector tl.landmark
cont cont.interior spg.goal cont.exterio
r spg.source
TO vs. INTO INTO adds a Container schema and
appropriate bindings.
49
Constructions with constituentsThe
SPATIAL-PHRASE construction
construction SPATIAL-PHRASE constructional cons
tituents sp Trajector-Landmark lm
Thing form spf before lmf meaning spm.landma
rk lmm
local alias
order constraint
identification constraint

• Constructions may also specify constructional
  constituents and impose form and meaning
  constraints on them
• order constraints
• identification constraints

50
An argument structure construction
construction DIRECTED-MOTION subcase of
Pred-Expr constructional constituents a
Ref-Exp m Pred-Exp p Spatial-Phrase form
af before mf mf before
pf meaning evokes Directed-Motion as
dm selfm.scene dm dm.agent am dm.motion
mm dm.path pm
schema Directed-Motion roles agent
Entity motion Motion path SPG
51
The CAUSED-MOTION construction
construction CAUSED-MOTION subcase of
Pred-Expr constructional constituents agent
Entity action Action patient
Entity path SPG form agentf before
actionf actionf before patientf actionf
before pathf meaning evokes Caused-Motion as
cm selfm.scene cm cm.agent
agentm cm.action actionm cm.patient
patientm cm.path pathm
52
Language Understanding Process

• An utterance is perceived
• This activates the form pole of some
  constructions
• The Analysis process assembles the constructions,
  using construal where necessary, and binds
  together their forms and their meanings
• The product is a constructional analysis
• This yields a semspec -- parameterized schemas
  linked together in specified ways
• The semspec is input into the Simulation process,
  where the understander imagines the content
• Resulting inferences are propagated through the
  conceptual system.

53
Simulation-based language understanding
54
Simulation specification

• A simulation specification consists of
• schemas evoked by constructions
• bindings between schemas

55
Language Understanding Process
56
Constructional analysis
57
Semantic Specification
58
Basic Feature Structure
A new rule for I
The corresponding fstruct
Pronoun ? I
number ?SG person ? 1st
-The top part of the rule is the old CFG rule.
-This data structure is attached to
the nonterminal during parsing so that the parser
can use the information.
-The next two lines set the agreement features.
-The feature is on the lhs of the colon And the
value is rhs of the colon.
-The ? denotes assignment to the feature listed
on the lhs.
59
Feature Structure Unification

• To check the compatibility of two fstructs
• Two feature structures are compatible if they
  have the same value for every feature they have
  in common (or if one or both leave the value
  unspecified).
• This process of checking compatibility is called
  unification.
• Unification
• Is a recursive process that takes two feature
  structures and either returns the combined
  feature structure if they are compatible or it
  returns failure.
• Base case Two values unify if they are the same
  string.
• Recursive Case Two feature structures unify if
  for each feature they have in common, those
  values unify.
• The resulting feature structure just adds the
  features they dont have in common to the
  resulting structure.

60
Language Analysis and Embodied Construction
Grammar

• John Bryant
• jbryant_at_icsi.berkeley.edu

61
Getting From the Utterance to the SemSpec

• Need a grammar formalism
• Embodied Construction Grammar (Bergen Chang
  2002)
• Need new models for language analysis
• Traditional methods too limited
• Traditional methods also dont get enough
  leverage out of the semantics.

62
Embodied Construction Grammar

• Semantic Freedom
• Designed to be symbiotic with cognitive
  approaches to meaning
• More expressive semantic operators than
  traditional grammar formalisms
• Form Freedom
• Free word order, over-lapping constituency
• Precise enough to be implemented

63
Traditional Parsing Methods Fall Short

• PSG parsers too strict
• Constructions not allowed to leave constituent
  order unspecified
• Traditional way of dealing with incomplete
  analyses is ad-hoc
• Making sense of incomplete analyses is important
  when an application must deal with ill-formed
  input
• Traditional unification grammar cant handle
  ECGs deep semantic operators.

64
Our Analyzer

• Replaces the FSMs used in traditional chunking
  (Abney 96) with much more powerful machines
  capable of backtracking called construction
  recognizers
• Arranges these recognizers into levels just like
  in Abneys work
• But uses a chart to deal with ambiguity

65
Our Analyzer (contd)

• Uses specialized feature structures to deal with
  ECGs novel semantic operators
• Supports a heuristic evaluation metric for
  finding the right analysis
• Puts partial analyses together when no complete
  analyses are available
• The analyzer was designed under the assumption
  that the grammar wont cover every meaningful
  utterance encountered by the system.

66
The Levels

• The analyzer puts the recognizer on the level
  assigned by the grammar writer.
• Assigned level should be greater than or equal to
  the levels of the constructions constituents.
• The analyzer runs all the recognizers on level 1,
  then level 2, etc. until no more levels.
• Recognizers on the same level can be mutually
  recursive.

67
Recognizers

• Each Construction is turned into a recognizer
• Recognizer active representation
• seeks form elements/constituents when initiated
• Unites grammar and process - grammar isnt just a
  static piece of knowledge in this model.
• Checks both form and semantic constraints
• Contains an internal representation of both the
  semantics and the form
• A graph data structure used to represent the form
  and a feature structure representation for the
  meaning.

68
Recognizer Example
Mary kicked the ball into the net.
This is the initial Constituent Graph for
caused-motion.
Patient
Action
Agent
Path
69
Recognizer Example
Construct Caused-Motion
Constituent Agent
Constituent Action
Constituent Patient
Constituent Path
The initial constructional tree for the instance
of Caused-Motion that we are trying to create.
70
Recognizer Example
71
Recognizer Example
processed
Mary kicked the ball into the net.
A node filled with gray is removed.
Patient
Action
Agent
Path
72
Recognizer Example
Construct Caused-Motion
Constituent Action
Constituent Patient
Constituent Path
RefExp Mary
Mary kicked the ball into the net.
73
Recognizer Example
74
Recognizer Example
processed
Mary kicked the ball into the net.
Patient
Action
Agent
Path
75
Recognizer Example
Construct Caused-Motion
Verb kicked
Constituent Patient
Constituent Path
RefExp Mary
Mary kicked the ball into the net.
76
Recognizer Example
77
Recognizer Example
processed
Mary kicked the ball into the net.
According to the Constituent Graph, The next
constituent can either be the Patient or the Path.
Patient
Action
Agent
Path
78
Recognizer Example
processed
Mary kicked the ball into the net.
Patient
Action
Agent
Path
79
Recognizer Example
Construct Caused-Motion
Verb kicked
RefExp Det Noun
Constituent Path
RefExp Mary
Noun
Det
Mary kicked the ball into the net.
80
Recognizer Example
81
Recognizer Example
processed
Mary kicked the ball into the net.
Patient
Action
Agent
Path
82
Recognizer Example
Construct Caused-Motion
Verb kicked
RefExp Det Noun
Spatial-Pred Prep RefExp
RefExp Mary
RefExp
Noun
Det
Noun
Det
Prep
Mary kicked the ball into the net.
83
Recognizer Example
84
Resulting SemSpec
After analyzing the sentence, the following
identities are asserted in the resulting SemSpec
Scene Caused-Motion Agent Mary Action
Kick Patient Path.Trajector The Ball Path
Into the net Path.Goal The net
85
Progress

• The analyzer (as described so far) is already
  being put to use in Changs thesis work.
• The levels are well-suited to incremental
  learning.
• Syntactic robustness important for generating
  partial analyses with poor coverage
• It will also be used this semester for producing
  SemSpecs for Narayanans enactment engine.
• Put the deep semantics towards parameterizing
  x-schemas

86
Current Work

• Leveraging the semantics to make the analyzer
  more robust
• When the analyzer cannot find a single analysis
  that spans all the input, it can still make a lot
  of progress towards a sensible analysis.
• Find combinations of sub-analyses that span and
  share common semantics in the chart.
• Leveraging the semantics to find the best
  analysis
• ECG analyses evoke semantic frames.
• Prefer analyses that better fill frame elements.

87
Summary

• By expanding traditional notions of parsing and
  unification grammar, it is possible to make a
  robust ECG-based language analyzer.
• Further work is necessary to better ground
  partial analysis/semantic density.
• But they seem promising.

88

• Embodied Construction Grammar providesformal
  tools for linguistic description and analysis
  motivated largely by cognitive/functional
  concerns.

• A shared theory and formalism for different
  cognitive mechanisms
• Constructions, metaphor, mental spaces, etc.
• Precise specifications of structures/processes
  involved in language understanding
• Bridge to detailed simulative inference using
  embodied representations

89
Summary ECG

• Linguistic constructions are tied to a model of
  simulated action and perception
• Embedded in a theory of language processing
• Constrains theory to be usable
• Frees structures to be just structures, used in
  processing
• Precise, computationally usable formalism
• Practical computational applications, like MT and
  NLU
• Testing of functionality, e.g. language learning
• A shared theory and formalism for different
  cognitive mechanisms
• Constructions, metaphor, mental spaces, etc.

90
ECG applications

• Grammar
• Spatial relations/events (Bergen Chang 1999
  Bretones et al. In press)
• Verbal morphology (Gurevich 2003, Bergen ms.)
• Reference measure phrases (Dodge and Wright
  2002), construal resolution (Porzel Bryant
  2003), reflexive pronouns (Sanders 2003)
• Semantic representations / inference
• Aspectual inference (Narayanan 1997 Chang,
  Gildea Narayanan 1998)
• Perspective / frames (Chang, Narayanan Petruck
  2002)
• Metaphorical inference (Narayanan 1997, 1999)
• Simulation semantics (Narayanan 1997, 1999)
• Language acquisition
• Lexical acquisition (Regier 1996, Bailey 1997)
• Multi-word constructions (Chang 2004 Chang
  Maia 2001)

91

• 3. Simulation-based inference

92
Interpretation simulation

• Constructions can
• specify which schemas and entities are involved
  in an event, and how they are related
• profile particular stages of an event
• set parameters of an event

walker at goal
energy
goalhome
walkerHarry
Harry is walking home.
93
Simulation Semantics

• execution-based model of events/processes
• tractable, distributed, concurrent,
  context-sensitive
• X-schemas provide natural model of
• resource consumption/production
• goals, preconditions, effects
• hierarchical events (multiple granularities)

94
Simulation Semantics (2)

• Captures fine-grained distinctions needed for
  interpretation
• aspectual inferences Narayanan 1997, 1999
  Chang et al. 1998
• metaphoric inferences Narayanan 1997, 1999
• perspectival inferences Chang et al. 2002
• inductive bias for language learning Bailey
  1997, Chang 2000
• Captures essential features of neural computation
  Feldman Ballard 1982, Feldman 1989, Valiant
  1994
• active, context-sensitive knowledge
  representation
• same representational substrate for action,
  perception Boccino et al. 2001, NBL01, CNS02
• natural model of concurrent and distributed
  computation

95
Simulation Semantics

• Inspired by biological control theory, Simulation
  Semantics models events as executing-, or
  x-schemas.
• An x-schema is a Petri net a weighted graph
  consisting of places (circles) and transitions
  (rectangles) connected by directed input and
  output arcs.
• A state is defined by the placement of a token (a
  black dot or number) in a particular place.
• The real-time execution semantics of Petri nets
  models the production and consumption of
  resources
• A transition is enabled when its input places are
  marked such that it can fire by movement of
  tokens from input to output.
• Arcs include resource, enable and inhibitory
  arcs.
• Actions have hierarchical structure, permitting
  embeddings.

96
Language is embodiedit is learned and used by
people with bodies who inhabit a physical,
psychological and social world.