NTL

1
NTL Converging Constraints

• Basic concepts and words derive their meaning
  from embodied experience.
• Abstract and theoretical concepts derive their
  meaning from metaphorical maps to more basic
  embodied concepts.
• Structured Connectionist Models can capture both
  of these processes nicely.
• Grammar extends this by Constructions pairings
  of form with embodied meaning.

2
Simulation-based language understanding
Utterance
Harry walked to the cafe.
Constructions
Analysis Process
General Knowledge
Simulation Specification
Schema Trajector Goal walk Harry cafe
Belief State
Simulation
3
(No Transcript)
4
Background Primate Motor Control

• Relevant requirements (Stromberg, Latash, Kandel,
  Arbib, Jeannerod, Rizzolatti)
• Should model coordinated, distributed,
  parameterized control programs required for motor
  action and perception.
• Should be an active structure.
• Should be able to model concurrent actions and
  interrupts.
• Model
• The NTL project has developed a computational
  model based on that satisfies these requirements
  (x- schemas).
• Details, papers, etc. can be obtained on the web
  at http//www.icsi.berkeley.edu/NTL

5
Active representations

• Representation based on stochastic Petri nets
  captures dynamic, parameterized nature of actions
• Many inferences about actions derive from what we
  know about executing them
• Generative model action, planning, recognition,
  language.

Walking bound to a specific walker with a
direction or goal consumes resources (e.g.,
energy) may have termination condition(e.g.,
walker at goal) ongoing, iterative action
6
Somatotopy of Action Observation
Foot Action
Hand Action
Mouth Action
Buccino et al. Eur J Neurosci 2001
7
Active Motion Model
Evolving Responses of Competing Models over Time.
Nigel Goddard 1989
8
Language Development in Children

• 0-3 mo prefers sounds in native language
• 3-6 mo imitation of vowel sounds only
• 6-8 mo babbling in consonant-vowel segments
• 8-10 mo word comprehension, starts to lose
  sensitivity to consonants outside native language
• 12-13 mo word production (naming)
• 16-20 mo word combinations, relational words
  (verbs, adj.)
• 24-36 mo grammaticization, inflectional
  morphology
• 3 years adulthood vocab. growth,
  sentence-level grammar for discourse purposes

9
food toys misc. people
sound emotion action prep.
demon. social
Words learned by most 2-year olds in a play
school (Bloom 1993)
10
Learning Spatial Relation Words Terry Regier

• A model of children learning spatial relations.
• Assumes child hears one word label of scene.
• Program learns well enough to label novel scenes
  correctly.
• Extended to simple motion scenarios, like INTO.
• System works across languages.
• Mechanisms are neurally plausible.

11
Learning System
dynamic relations (e.g. into)
structured connectionistnetwork (based on
visual system)

• Well look at the details next lecture

12
Limitations

• Scale
• Uniqueness/Plausibility
• Grammar
• Abstract Concepts
• Inference
• Representation
• Biological Realism

13
(No Transcript)
14
Constrained Best Fit in Nature
inanimate animate
physics lowest energy state
chemistry molecular minima
biology fitness, MEU neuroeconomics
vision threats, friends
language errors, NTL
15
Learning Verb MeaningsDavid Bailey

• A model of children learning their first verbs.
• Assumes parent labels childs actions.
• Child knows parameters of action, associates with
  word
• Program learns well enough to
• 1) Label novel actions correctly
• 2) Obey commands using new words (simulation)
• System works across languages
• Mechanisms are neurally plausible.

16
(No Transcript)
17
(No Transcript)
18
Motor Control (X-schema) for SLIDE
19
(No Transcript)
20
Parameters for the SLIDE X-schema
21
Feature Structures for PUSH
22
(No Transcript)
23
System Overview
24
Learning Two Senses of PUSH
Model merging based on Bayesian MDL
25
Training ResultsDavid Bailey

• English
• 165 Training Examples, 18 verbs
• Learns optimal number of word senses (21)
• 32 Test examples 78 recognition, 81 action
• All mistakes were close lift yank, etc.
• Learned some particle CXN,e.g., pull up
• Farsi
• With identical settings, learned senses not in
  English

26
(No Transcript)
27
(No Transcript)
28
Constrained Best Fit in Nature
inanimate animate
physics lowest energy state
chemistry molecular minima
biology fitness, MEU neuroeconomics
vision threats, friends
language errors, NTL
29
(No Transcript)
30
Compositionality

• Traditional
• Context-free composition of logical forms
• Contemporary
• Constructional composition of conceptual
  
• frames
• Formal Cognitive Linguistics

31
Embodied Construction Grammar(Bergen, Chang
Paskin 2000)

• Assumptions from Construction Grammar
• Constructions are form-meaning pairs(Lakoff
  1987, Goldberg 1995)
• Constructions vary in degree of specificity and
  level of description (morphological, lexical,
  phrasal, clausal)
• Constructions evoke and bind semantic schemas
• Additional influences
• Cognitive Grammar (Langacker 1987)
• Frame Semantics (Fillmore 1982)
• Structured Connectionism (Feldman 1987)

32
Traditional Levels of Analysis
Pragmatics
Semantics
Syntax
Morphology
Phonetics
33
Harry walked into the cafe.
Pragmatics
Semantics
Utterance
Syntax
Morphology
Phonetics
34
Language understanding analysis simulation
construction WALKED in context c constituents
form f of type wakt meaning walking construed
as Walk-Action semantic constraints walking.ti
me before c.speech-time walking.aspect
encapsulated designates walking
Harry walked into the cafe.
Utterance
Analysis Process
Constructions
General Knowledge
Simulation Specification
Belief State
Simulation
35
Simulation specification

• A simulation specification consists of
• semantic schemas evoked by constructions
• bindings between schemas (labeled by the
  constructions that enforce them)

36
Conclusion

• Language acquisition and use is a hallmark of
  being human
• Language seems to rely on fine-grained aspects of
  embodied (sensory-motor and social cognition)
  primitives and brain-like computation (massively
  parallel, distributed, spreading activation,
  temporal binding).
• Understanding requires imaginative simulation!
• We have built a pilot system that demonstrates
  the use of motor control representations in
  grounding the language of abstract actions and
  policies.
• Sensory-Motor imagination and simulation is
  crucial in interpretation!
• Ongoing Work.
• Formalize and use a compositional set of embodied
  conceptual primitives and grammatical
  constructions.
• Perform both behavioral and fMRI imaging
  experiments to test the predictions of the
  simulation hypothesis.
• Further refine and ground the model in details of
  neural anatomy and functional architecture
  (basal-thalamic-cortical loops).