The%20ICSI/Berkeley%20Neural%20Theory%20of%20Language%20Project

1
The ICSI/Berkeley Neural Theory of Language
Project
ECG
2
Moving from Spatial Relations to Verbs

• Open class vs. closed class
• How do we represent verbs (say of hand motion)
• Can we build models of verbs based on motor
  control primitives?
• If so, how can models overcome central
  limitations of Regiers system?
• Inference
• Abstract uses

3
Perceptual And Motor Schemas

• A perceptual schema embodies the process
  whereby the system determines whether a given
  domain of interaction is present in the
  environment.
• A schema assemblage combines an estimate of
  environmental state with a representation of
  goals and needs
• The internal state is also updated by knowledge
  of the state of execution of current plans made
  up of motor schemas which are akin to control
  systems but distinguished by the fact that they
  can be combined to form coordinated control
  programs

4
Coordination of Pattern Generators
5
Coordination

• PATTERN GENERATORS, separate neural networks that
  control each limb, can interact in different ways
  to produce various gaits.
• In ambling (top) the animal must move the fore
  and hind leg of one flank in parallel.
• Trotting (middle) requires movement of diagonal
  limbs (front right and back left, or front left
  and back right) in unison.
• Galloping (bottom) involves the forelegs, and
  then the hind legs, acting together

6
Preshaping While Reaching to Grasp
7
Hypothetical coordinated control program for
reaching and grasping
Perceptual Schemas
Motor Schemas
Dashed lines activation signals solid lines
transfer of data. (Adapted from Arbib 2004)
8
(No Transcript)
9
Internal Model and Efference Copy
10
(No Transcript)
11
Many areas code for motion parameters
12
(No Transcript)
13
Multiple, chronically implanted, intracranial
microelectrode arrays would be used to sample
the activity of large populations of single
cortical neurons simultaneously. The combined
activity of these neural ensembles would then be
transformed by a mathematical algorithm into
continuous three-dimensional arm-trajectory
signals that would be used to control the
movements of a robotic prosthetic arm. A closed
control loop would be established by providing
the subject with both visual and tactile feedback
signals generated by movement of the robotic arm.
14
A New Picture
Rizzolatti et al. 1998
15
The fronto-parietal networks
Rizzolatti et al. 1998
16
F5 Mirror Neurons
Gallese and Goldman, TICS 1998
17
Category Loosening in Mirror Neurons (60)
Observed A is Precision Grip B is Whole Hand
Prehension Action C precision grip D
Whole Hand Prehension
(Gallese et al. Brain 1996)
18
PF Mirror Neurons

• Neuron responds to execution (grasping) but to
  grasping and releasing in observation.
• Mirror neurons in parietal cortex.
• Difference in left hand and right hand.

(Gallese et al. 2002)
19
A (Full vision) B (Hidden) C (Mimicking) D
(HiddenMimicking)
Umiltà et al. Neuron 2001
20
F5 Audio-Visual Mirror Neurons
Kohler et al. Science (2002)
21
Summary of Fronto-Parietal Circuits

• Motor-Premotor/Parietal Circuits
• PMv (F5ab) AIP Circuit
• grasp neurons fire in relation to movements
  of hand prehension necessary to grasp object
• F4 (PMC) (behind arcuate) VIP Circuit
• transforming peri-personal space coordinates so
  can move toward objects
• PMv (F5c) PF Circuit F5c
• different mirror circuits for grasping, placing
  or manipulating object
• Together suggest cognitive representation of the
  grasp, active in action imitation and action
  recognition

22
Evidence in Humans for Mirror, General Purpose,
and Action-Location Neurons Mirror Fadiga et
al. 1995 Grafton et al. 1996 Rizzolatti et al.
1996 Cochin et al. 1998 Decety et al. 1997
Decety and Grèzes 1999 Hari et al. 1999
Iacoboni et al. 1999 Buccino et al.
2001. General Purpose Perani et al. 1995
Martin et al. 1996 Grafton et al. 1996 Chao and
Martin 2000. Action-Location Bremmer, et al.,
2001.
23
(No Transcript)
24
FARS (Fagg-Arbib-Rizzolatti-Sakata) Model
AIP extracts the set of affordances for an
attended object.These affordances highlight the
features of the object relevant to physical
interaction with it.
Itti CS564 - Brain Theory and Artificial
Intelligence. FARS Model
25
Hypothetical coordinated control program for
reaching and grasping
Perceptual Schemas
Motor Schemas
Dashed lines activation signals solid lines
transfer of data. (Adapted from Arbib 2004)
26

• MULTI-MODAL INTEGRATION
• The premotor and parietal areas, rather than
  having
• separate and independent functions, are neurally
  integrated
• not only to control action, but also to serve the
  function of
• constructing an integrated representation of
• Actions, together with
• objects acted on, and
• locations toward which actions are directed.
• In these circuits sensory inputs are transformed
  in order to
• accomplish not only motor but also cognitive
  tasks, such as
• space perception and action understanding.

27
Modeling Motor Schemas

• Relevant requirements (Stromberg, Latash, Kandel,
  Arbib, Jeannerod, Rizzolatti)
• Should model coordinated, distributed,
  parameterized control programs required for motor
  action and perception.
• Should be an active structure.
• Should be able to model concurrent actions and
  interrupts.
• Should model hierarchical control (higher level
  motor centers to muscle extensor/flexors.
• Computational model called x-schemas
  (http//www.icsi.berkeley.edu/NTL)

28
An Active Model of Events

• At the Computational level, actions and events
  are coded in active representations called
  x-schemas which are extensions to Stochastic
  Petri nets.
• x-schemas are fine-grained action and event
  representations that can be used for monitoring
  and control as well as for inference.

29
Model Review Stochastic Petri Nets
Basic Mechanism
1
1
Firing function -- conjunctive -- logistic
-- exponential family
30
Model Review
Firing Semantics
31
Model Review
Result of Firing
32
Active representations

• Many inferences about actions derive from what we
  know about executing them
• Representation based on stochastic Petri nets
  captures dynamic, parameterized nature of actions
• Generative model action, recognition, planning ,
  language

Walking bound to a specific walker with a
direction or goal consumes resources (e.g.,
energy) may have termination condition(e.g.,
walker at goal) ongoing, iterative action
33
Lecture Overview

• Moving beyond Spatial Prepositions to Verbs
• Motor Control Schemas Recap
• A Computational Model of Motor Control Schemas
• Basic Primitives
• Demo of the X-schema model
• Applications of Model
• Actions and Inference

34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
(No Transcript)
50
(No Transcript)
51
(No Transcript)
52
(No Transcript)
53
(No Transcript)
54
Preshaping While Reaching to Grasp
55
(No Transcript)
56
(No Transcript)
57
(No Transcript)
58
(No Transcript)
59
(No Transcript)
60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
(No Transcript)
65
(No Transcript)
66
(No Transcript)
67
(No Transcript)
68
(No Transcript)
69
(No Transcript)
70
(No Transcript)
71
The ICSI/Berkeley Neural Theory of Language
Project
ECG
72
Representing concepts using triangle nodes
triangle nodes when two of the neurons fire, the
third also fires
73
Feature Structures in Four Domains
Barrett Ham Container Push
deptCS Color pink Inside region Schema slide
sid001 Taste salty Outside region Posture palm
empGSI Bdy. curve Dir. away

Chang Pea Purchase Stroll
deptLing Color green Buyer person Schema walk
sid002 Taste sweet Seller person Speed slow
empGra Cost money Dir. ANY
Goods thing
74
Simulation hypothesis

• We understand utterances by mentally simulating
  their content.
• Simulation exploits some of the same neural
  structures activated during performance,
  perception, imagining, memory
• Linguistic structure parameterizes the
  simulation.
• Language gives us enough information to simulate

75
Simulation Semantics

• BASIC ASSUMPTION SAME REPRESENTATION FOR
  PLANNING AND SIMULATIVE INFERENCE
• Evidence for common mechanisms for recognition
  and action (mirror neurons) in the F5 area
  (Rizzolatti et al (1996), Gallese 96, Buccino
  2002, Tettamanti 2004) and from motor imagery
  (Jeannerod 1996)
• IMPLEMENTATION
• x-schemas affect each other by enabling,
  disabling or modifying execution trajectories.
  Whenever the CONTROLLER schema makes a transition
  it may set, get, or modify state leading to
  triggering or modification of other x-schemas.
  State is completely distributed (a graph marking)
  over the network.
• RESULT INTERPRETATION IS IMAGINATIVE SIMULATION!

76
Simulation-based language understanding
77
Simulation specification

• A simulation specification consists of
• schemas evoked by constructions
• bindings between schemas

78
Language Development in Children

• 0-3 mo prefers sounds in native language
• 3-6 mo imitation of vowel sounds only
• 6-8 mo babbling in consonant-vowel segments
• 8-10 mo word comprehension, starts to lose
  sensitivity to consonants outside native language
• 12-13 mo word production (naming)
• 16-20 mo word combinations, relational words
  (verbs, adj.)
• 24-36 mo grammaticization, inflectional
  morphology
• 3 years adulthood vocab. growth,
  sentence-level grammar for discourse purposes

79
food toys misc. people
sound emotion action prep.
demon. social
Words learned by most 2-year olds in a play
school (Bloom 1993)
80
Regier Model Limitations

• Scale
• Uniqueness/Plausibility
• Grammar
• Abstract Concepts
• Inference
• Representation
• Biological Realism

81
Learning Verb MeaningsDavid Bailey

• A model of children learning their first verbs.
• Assumes parent labels childs actions.
• Child knows parameters of action, associates with
  word
• Program learns well enough to
• 1) Label novel actions correctly
• 2) Obey commands using new words (simulation)
• System works across languages
• Mechanisms are neurally plausible.

82
(No Transcript)
83
Reasoning about Actions in Artificial
Intelligence (AI)

• The earliest work on actions in AI took a
  deductive approach
• designers hoped to represent all the system's
  world knowledge' explicitly as axioms, and use
  ordinary logic - the predicate calculus - to
  deduce the effects of actions
• Envisaging a certain situation S was modeled by
  having the system entertain a set of axioms
  describing the situation
• To this set of axioms the system would apply an
  action - by postulating the occurrence of some
  action A in situation S - and then deduce the
  effect of A in S, producing a description of the
  outcome situation S'

84
Grasping the action

• A set of pre-conditions in S
• free_top(y), free_hand(x), accessible(y)
• The grasp action (effect axiom)
• Result(Grasp(x,y, S), hold(x,y,S))
• A set of effects describing the new situation S
• Hold(x,y), not(free-hand(x))

85
Actions

• An action is described as an axiom linking
  preconditions (literals and terms true in the
  before situation) to effects (literals and terms
  true in the after situation).
• The action specification is called an effect axiom

86
Assumptions

• Necessary and sufficient conditions
• Object categories
• Event categories
• Monotonicity
• Axioms, once asserted, cannot be retracted
• Complex for actions since actions are about
  change
• Closed World Assumption

87
Problems with action concepts

• Frame problem
• Qualification problem
• Ramification problem

88
The Frame Problem

• Which things dont change in an action
• S1 blue(x), on_table(x), free_hand(y)
• Action grasp(y,x)
• S2 in_hand(x,y), hold(x,y), ?

89
Frame axioms are needed in logic

• Consider some typical frame axioms associated
  with the action-type
• move x onto y.
• If z ! x and I move x onto y, then if z was on w
  before, then z is on w after.
• If x is blue before, and I move x onto y, then x
  is blue after.

90
Active Representations dont need frame axioms

• X-schemas directly model change, so no need for
  frame axioms. Also, they deal with concurrency,
  so no need to treat one action at a time.
• Based on x-schema type models there are a new set
  of logics called resource logics which attempt to
  model the frame problem directly.

91
Logical approaches to the Frame Problem

• STRIPS assumptions
• Preconditions and effects
• Add and Delete Lists for actions
• Only one action at a time
• Non Monotonic Logics
• Circumscription (explicitly specify abnormal
  conditions)

92
Qualification problem

• How do I specify all the pre-conditions for an
  action?
• Problem arises out of the necessary and
  sufficient conditions for the pre-conditions

93
Approaches to the Qualification problem

• Traditional Models
• STRIPS assumption
• Modern AI Approach
• Probabilistic Models of Actions

94
Ramification Problem

• How do I specify all the effects
• Direct (if I move, I change my location) and
• Indirect (things that were accessible before I
  moved may not be anymore)
• Central issue is to propagate changes of an
  action to all the connected knowledge that might
  be impacted.
• How might the brain do this?
• Spreading Activation

95
Solutions to the Ramification Problem

• Traditional Solution
• One action at a time
• Closed World assumption
• Modern AI Solution
• Bayes Nets and Probabilistic Models

96
General Modern Solution

• Use Probabilistic Models to model the indirect
  effects of actions
• Graphical Models
• Stochastic Causal Models

97
But

• Actions and events (and concepts in general)
• Are context sensitive
• Admit detailed structure
• Events can have starts, middles, ends..
• Have prototype structure
• Have basic level categorization structure
• Can be executed and reasoned about in parallel
• Concurrency and synchronization

98
Somatotopy of Action Observation
Foot Action
Hand Action
Mouth Action
Buccino et al. Eur J Neurosci 2001
99
Pattern Generator Networks

• a), Four-cell network for bipedal locomotor CPG
  b), eight-cell network for quadrupeds c),
  4n-cell network for 2n-legged animals. Double
  lines indicate contralateral coupling single
  lines indicate ipsilateral coupling. Direction of
  ipsilateral coupling is indicated by arrows
  contralateral coupling is bidirectional.