Situation Models and Embodied Language Processes

1
Situation Models and Embodied Language Processes

• Franz Schmalhofer
• University of Osnabrück / Germany

• Memory and Situation Models
• Computational Modeling of Inferences
• What Memory and Language are for
• Neural Correlates
• Integration of Behavioral Experiments and Neural
  Correlates (ERP fMRI) by Formal Models

2
Theories of Knowledge

• Prior to the twentieth century knowledge was
  assumed to be perceptual
• Past several decades fields of cognition and
  perception have diverged.

• Perceptual approaches viewed as untenable for
  conceptual systems.
• Logic, statistics, programming languages have
  inspired amodal theories different from
  perceptional characteristics

3
Symbol Grounding

• Illustration for computational theories of
  language understanding
• The Chinese room (Searle, 1980)
• - getting Chinese input symbols
• - manipulation of symbols only according to their
• shapes ( no meanings/no understanding )
• returning Chinese symbols as output
• The symbol grounding problem (Harnad, 1990)
• ?cognition cannot be just symbol manipulation

4
Perceptual Symbol Systems (Barsalou, 1999)

• WRONG Perceptual systems pick up information
  from the environment and pass it on to separate
  systems that support the various cognitive
  functions. (i.e. language, memory, and thought)
• CORRECT Cognition is inherently perceptual,
  sharing systems with perception at both the
  cognitive and the neural levels.
• No divergence between cognition and perception

5
How we got the wrong ideas

• Behaviorist attacks on mentalism. (Watson)
• Similar attacks from ordinary language philosophy
  (Wittgenstein)
• Continuing attacks after the cognitive
  revolution.
• Development of logic, programming languages,
  statistical representation
• Contributions of amodal symbol systems
• Formalizable, runnable, applicable
• Highlight important computational properties of
  productivity, proposition, structure, etc.

6
Assumption Cognition is Grounded in Perception

• A common representational system underlies
  perception and cognition.
• From Aristoteles to Locke to Kant, theorists over
  the last 2000 years have viewed cognition as
  imagistic in nature.
• Image-based theories disappeared as behaviorists
  and language philosophers began to avoided to
  talk about mental states.

7
How is cognition grounded in perception?

• During perceptual experience, association areas
  in the brain capture bottom-up patterns of
  activation in sensory-motor areas and in a
  top-down manner, association areas partially
  reactivate sensory-motor areas to implement
  perceptual symbols.
• The storage and reactivation of perceptual
  symbols operates at the level of perceptual
  components--not at the level of holistic
  perceptual experiences.
• Use of selective attention, schematic
  representations of perceptual components are
  extracted from experience and stored in memory

8
Properties of amodal symbol system

• As amodal symbols are transduced from perceptual
  states, they enter into larger representational
  structures.
• In turn, these structures support all of the
  higher cognitive functions, including knowledge,
  memory, language and thought.
• Across the cognitive sciences, standard theories
  of knowledge adopt the assumptions, so called
  amodal symbol systems.
• Information in the physical world produces neural
  states in perceptual systems.
• A transduction process takes these states as
  input, produces descriptions of them in a
  completely new representation language.

9
4. Amodal symbol system

• Amodal symbol representations

• Perceptual states are transduced into a
  completely new representational system that
  describes these states amodally.
• The internal structure of these symbols is
  unrelated to the perceptual state that produced
  them.

10
Problems with amodal symbol system

• No account for the relation between cognition and
  perception.
• No empirical evidence that amodal systems exist.
  
• Transduction process that maps perceptual states
  into amodal symbols remains unclear.
• Converse of transduction problem no account how
  perceptual states map to amodal symbols.
• Too powerful amodal symbol systems are
  unconstrained, offer little insight into the
  phenomena.

11
Resurgence of perceptual symbol system

• Theorists develop perceptual views that are
  provocative, powerful.
• It provides a natural account of the relation
  between cognition and perception.
• An obvious account exists of how perceptual
  symbols are implemented in the brain.
• No need for a major leap in evolution
• Provide natural mechanisms for representing space
  and time
• Make clear a priori predictions that are
  falsifiable
• Growing empirical evidence for perceptual symbols

12
Basic assumption of perceptual views

• States arise in sensory-motor systems during
  contact with the physical world.
• traditionally viewed as conscious states
• but will be viewed here primarily as neural
  states
• these sensory-motor states are stored in memory
  to some extent (utilizing sensory-motor systems)
• stored perceptual states later support higher
  cognitive processes
• during memory, language, and thought
• may establish reference back into the physical
  world

13
Representation in Perceptual Symbol Systems

• Subsets of perceptual states in sensory-motor
  systems are extracted.
• The internal structure of these symbols is
  therefore
• modal and
• analogically related to the perceptual state that
  produced them.

14
(No Transcript)
15
Core assumptions about perceptual symbols

• Perceptual symbols are schematic.
• Perceptual symbols are multimodal
• Perceptual symbols enter into simulation
  competence.
• Perceptual symbols are productive.
• Perceptual symbols represent situation
  components.
• Perceptual symbols also represent abstract
  concepts

16
Experiments

• Spivey, M.J. et al. (2000) Eye movements during
  comprehension of spoken scene descriptions.
• Zwaan, R.A., Stanfield, R. Y. Yaxley, R. H.
  (2002) Do language comprehenders routinely
  represent the shapes of objects. Psychological
  Science, 13, 168-171.
• Glenberg A. M. Kaschak M. P. (2002) Grounding
  language in action. Psychonomic Bulletin
  Review, 9, 558-565

17
Zwaan (2004) The immersed experiencer
C construal T time S spatial region
(personal, action, vista) P perspective F
focal entity R relation B background entity f
feature
18
Embodied Language Comprehension

• Taylor and Tversky (1992) language is a
  surrogate for experience
• Goal of language comprehension creation of an
  embodied mental model
• In the brain words activate experiences with
  their referents (Pulvermüller)
• Perceptual simulation of a described object or
  situation construction of a situation model

19
Visual field

• Fovea
• Para-fovea
• Plateau
• Periphery
• Temporal monocular

20
From the Retina to V1
nasal retinal fibres
temporal retinal fibres
21
Visual Pathways
Vision for Action
Vision for Perception
Goodale Humphrey (1998)
22
Perceptual Experience and Perceptual States

• Perceptual experiences activate association areas
  in the brain which capture
• bottom-up patterns of activation in sensory-motor
  areas.
• top-down patterns, association areas partially
  reactivate sensory-motor areas to implement
  perceptual symbols.

• Perceptual States
• Arise in sensory-motor systems with two
  components
• Unconscious neural representation of physical
  input
• Optional conscious experience
• Related perceptual symbols combine to form
  simulators that allow the cognitive system to
  construct specific simulations of an entity or
  event in its absence

23
Perceptual symbols

• Are patterns that rise in hierarchical feature
  maps of sensory-motor systems during perception
  and action.
• May or may not be topographical captured by
  association areas (Damasio, 1989)
• From local, to poly-sensory and to frontal
• Tuned for specific combinations of features
• Activating an associative pattern reinstates
• not necessarily complete nor veridical but
  partial records of the neural states that
  underlie perception
• Is dynamic, not discrete
• not necessarily representative of specific
  individuals
• potentially indeterminate

24
(No Transcript)
25
Perceptual symbols are NOT

• like physical pictures
• entire perceptual states
• If they were, the componential character of
  conceptual system would not be feasible
• mental images or conscious experience
• states in neural systems

26
Simulators

• Perceptual symbols of the same category are
  integrated together in a single system
  (simulators).
• A simulator
• An organized system of perceptual symbols that
  can produce simulations of a category in the
  absence of physical exemplars.
• Typically contains perceptual symbols extracted
  from many members of category on many modalities.
  
• Composed of frames, the simulations that the
  frame produces.

27
Simulators and Simulations

• Specific runs of a simulator that reenact the
  multimodal experience of a category
• Utilize a small subset of the information in a
  simulator
• Infinite many simulations is possible.
• Simulations are always partial and sketchy, never
  complete.
• A simulator goes beyond a simple empirical
  collection of sense impressions.
• A huge set of simulations that include the range
  of experience associated with a category
• i.e. the representation of a type, not a token

28
(No Transcript)
29

• Simulations as
• vehicles.
• This figure is showing
• some partial frame
• for car, it illustrates
• how the frame has
• changed dynamically.

30
Categorization by construal

• When an entity is categorized, the best fitting
  simulator is found
• The simulator runs a simulation that provides a
  good fit to the entity
• Some examples
• Hearing a bark and simulating a dog
• Seeing a growling dog and simulating the
  experience of being attacked
• Smelling food and simulating what it is

31
Concepts and Offline Conceptualization

• Concepts and Simulators
• A concept is equivalent to a simulator.
• If we have an appropriate simulator of something,
  then it can be said that we understand the
  concept.
• Goal of human learning is to establish
  simulators.
• Offline conceptualizing during memory, language
  and thought
• Simulations provide inferences about likely
  properties of entities in their absence
• The simulations run in sensory-motor systems and
  they can be mapped later to perceived entities
• E.g. remembering ones parking spot, finding the
  referent of a linguistic description.

32
(No Transcript)
33
Summary I Perceptual Symbol Systems

• Perceptual states arise in the sensory-motor
  system.
• A subset of the state is extracted by selective
  attention and stored in long-term memory.
• This perceptual memory can function as a symbol
  entering into symbol manipulation.
• Collections of the perceptual symbols comprise
  our conceptual representations.
• The structure of a perceptual symbol corresponds
  (at least somewhat) to the perceptual state that
  produced it.
• Note this does not claim that a perceptual
  symbol corresponds to the physical world.

34
Summary II Perceptual Symbol Systems

• A very different approach to knowledge in the
  form of perceptual symbol systems.
• Perceptual states are not transduced into a
  completely new representational language, instead
  subsets of perceptual states are extracted to
  function symbolically and support the higher
  cognitive functions.
• Reference Barsalou, L. W. (1999). Perceptual
  Symbol Systems. Behavioral and Brain Sciences
  22 (507-569).