Title: NTL Converging Constraints
1NTL Converging Constraints
- Basic concepts and words derive their meaning
from embodied experience. - Abstract and theoretical concepts derive their
meaning from metaphorical maps to more basic
embodied concepts. - Structured Connectionist Models can capture both
of these processes nicely. - Grammar extends this by Constructions pairings
of form with embodied meaning.
2 Simulation-based language understanding
Utterance
Harry walked to the cafe.
Constructions
Analysis Process
General Knowledge
Simulation Specification
Schema Trajector Goal walk Harry cafe
Belief State
Simulation
3(No Transcript)
4Background Primate Motor Control
- Relevant requirements (Stromberg, Latash, Kandel,
Arbib, Jeannerod, Rizzolatti) - Should model coordinated, distributed,
parameterized control programs required for motor
action and perception. - Should be an active structure.
- Should be able to model concurrent actions and
interrupts. - Model
- The NTL project has developed a computational
model based on that satisfies these requirements
(x- schemas). - Details, papers, etc. can be obtained on the web
at http//www.icsi.berkeley.edu/NTL
5Active representations
- Representation based on stochastic Petri nets
captures dynamic, parameterized nature of actions - Many inferences about actions derive from what we
know about executing them - Generative model action, planning, recognition,
language.
Walking bound to a specific walker with a
direction or goal consumes resources (e.g.,
energy) may have termination condition(e.g.,
walker at goal) ongoing, iterative action
6Somatotopy of Action Observation
Foot Action
Hand Action
Mouth Action
Buccino et al. Eur J Neurosci 2001
7Active Motion Model
Evolving Responses of Competing Models over Time.
Nigel Goddard 1989
8Language Development in Children
- 0-3 mo prefers sounds in native language
- 3-6 mo imitation of vowel sounds only
- 6-8 mo babbling in consonant-vowel segments
- 8-10 mo word comprehension, starts to lose
sensitivity to consonants outside native language - 12-13 mo word production (naming)
- 16-20 mo word combinations, relational words
(verbs, adj.) - 24-36 mo grammaticization, inflectional
morphology - 3 years adulthood vocab. growth,
sentence-level grammar for discourse purposes
9food toys misc. people
sound emotion action prep.
demon. social
Words learned by most 2-year olds in a play
school (Bloom 1993)
10Learning Spatial Relation Words Terry Regier
- A model of children learning spatial relations.
- Assumes child hears one word label of scene.
- Program learns well enough to label novel scenes
correctly. - Extended to simple motion scenarios, like INTO.
- System works across languages.
- Mechanisms are neurally plausible.
11Learning System
dynamic relations (e.g. into)
structured connectionistnetwork (based on
visual system)
- Well look at the details next lecture
12Limitations
- Scale
- Uniqueness/Plausibility
- Grammar
- Abstract Concepts
- Inference
- Representation
- Biological Realism
13(No Transcript)
14Constrained Best Fit in Nature
inanimate animate
15Learning Verb MeaningsDavid Bailey
- A model of children learning their first verbs.
- Assumes parent labels childs actions.
- Child knows parameters of action, associates with
word - Program learns well enough to
- 1) Label novel actions correctly
- 2) Obey commands using new words (simulation)
- System works across languages
- Mechanisms are neurally plausible.
16(No Transcript)
17(No Transcript)
18Motor Control (X-schema) for SLIDE
19(No Transcript)
20 Parameters for the SLIDE X-schema
21Feature Structures for PUSH
22(No Transcript)
23System Overview
24Learning Two Senses of PUSH
Model merging based on Bayesian MDL
25Training ResultsDavid Bailey
- English
- 165 Training Examples, 18 verbs
- Learns optimal number of word senses (21)
- 32 Test examples 78 recognition, 81 action
- All mistakes were close lift yank, etc.
- Learned some particle CXN,e.g., pull up
- Farsi
- With identical settings, learned senses not in
English
26(No Transcript)
27(No Transcript)
28Constrained Best Fit in Nature
inanimate animate
29(No Transcript)
30Compositionality
- Traditional
- Context-free composition of logical forms
- Contemporary
- Constructional composition of conceptual
- frames
- Formal Cognitive Linguistics
31Embodied Construction Grammar(Bergen, Chang
Paskin 2000)
- Assumptions from Construction Grammar
- Constructions are form-meaning pairs(Lakoff
1987, Goldberg 1995) - Constructions vary in degree of specificity and
level of description (morphological, lexical,
phrasal, clausal) - Constructions evoke and bind semantic schemas
- Additional influences
- Cognitive Grammar (Langacker 1987)
- Frame Semantics (Fillmore 1982)
- Structured Connectionism (Feldman 1987)
32Traditional Levels of Analysis
Pragmatics
Semantics
Syntax
Morphology
Phonetics
33Harry walked into the cafe.
Pragmatics
Semantics
Utterance
Syntax
Morphology
Phonetics
34Language understanding analysis simulation
construction WALKED in context c constituents
form f of type wakt meaning walking construed
as Walk-Action semantic constraints walking.ti
me before c.speech-time walking.aspect
encapsulated designates walking
Harry walked into the cafe.
Utterance
Analysis Process
Constructions
General Knowledge
Simulation Specification
Belief State
Simulation
35Simulation specification
- A simulation specification consists of
- semantic schemas evoked by constructions
- bindings between schemas (labeled by the
constructions that enforce them)
36Conclusion
- Language acquisition and use is a hallmark of
being human - Language seems to rely on fine-grained aspects of
embodied (sensory-motor and social cognition)
primitives and brain-like computation (massively
parallel, distributed, spreading activation,
temporal binding). - Understanding requires imaginative simulation!
- We have built a pilot system that demonstrates
the use of motor control representations in
grounding the language of abstract actions and
policies. - Sensory-Motor imagination and simulation is
crucial in interpretation! - Ongoing Work.
- Formalize and use a compositional set of embodied
conceptual primitives and grammatical
constructions. - Perform both behavioral and fMRI imaging
experiments to test the predictions of the
simulation hypothesis. - Further refine and ground the model in details of
neural anatomy and functional architecture
(basal-thalamic-cortical loops).