Situation Models and Embodied Language Processes

1
Situation Models and Embodied Language Processes

• Franz Schmalhofer
• University of Osnabrück / Germany

• Memory and Situation Models
• Computational Modeling of Inferences
• What Memory and Language are for
• Neural Correlates
• Integration of Behavioral Experiments and Neural
  Correlates (ERP fMRI) by Formal Models

2
Cognition and Knowledge

• Traditional
• Cognition Computation
• Representation by propositions
• Propositions are abstract relations

• Embodiment of Meaning
• Cognition is serving perception and actions
• Representation Patterns of possible bodily
  interactions with the world (lawfully related to
  the world)
• What an object, event, sentence means for you, is
  what you can do with the object, event, sentence.

3
Embodiment of Memory (Glenberg, 1997)

• Projectable properties information available
  through the senses
• Non-projectable properties information available
  through other sources (e.g. memory)
• Conceptualization Combination (mesh) of
  projectable and non-projectable properties
• ? Primary function of memory is to mesh the
  embodied conceptualizations of projectable
  properties of the environment with embodied
  experiences that provide non-projectable
  properties

4
Embodiment of Memory

• Evidence for embodiment...
• ...and affect
• Forced frowning or smiling influences affective
  judgments (Berkowitz Trocolli 1990, Berkowitz
  et. al 1993)
• ...and imagery
• Actually rotating facilitates orientation opposed
  to imagining rotating(Montello Presson 1993,
  Rieser etl al. 1994).
• ...and memory
• Retrieval of memorized spatial layouts depends on
  position on body axis (Bryant 1992).

5
Embodied Memory

• Combination of patterns of possible action
• meshing.
• Meshing of patterns of action derived from
• projectable properties of the environment
• non-projectable properties
• as (spatial-functional) constraint satisfaction.

6
Function of Memory

• Meshing projectable non-projectable properties
• Meshing is important for
• perception
• imagination
• Comprehension
• Projectable properties as well as non-projectable
  properties can be meshed with each-other.

7
Interaction of perceptions and memory

• Environment has to be primary
• Clamping projectable properties keeps the system
  reality-oriented
• Experiences stay individuated

8
Updating memory

• Experiences are shifts between conceptualizations
• Trajectories from one pattern of action to
  another
• Trajectories more often used become reinforced

9
Prediction and Planning

• Hypothetical conceptualizations by using
  trajectories from memory
• But
• Simulating action does not change the environment
• Clamped projectable properties provide wrong
  constraints for prediction
• Environment has to be suppressed (effortfull
  process)
• Suppression loosens tie to reality

10
Two modes of memory (like the distinction
between implicit/explicit)

• Automatic
• Meshing of projectable and nonprojectable
  properties
• Causes conceptual priming based on
  non-projectable properties, (therefore semantic)

• Effortful
• Suppression of projectable properties,
  conceptualization by trajectories from memory

11
Cognitive Meshing

• Imagine a ball
• Now Imagine that it has yellow and white stripes
• Now Imagine that it is deflated

• Mutual modification of mashed pattern not only
  the ball but also the stripes become deformed
  when the ball is deflated.
• Patterns of actions to the same spatio-functional
  constraints.

12
What memory and language are for

• The primary function of memory is to mesh
• the embodied conceptualizations of projectable
• properties of the environment with embodied
• experiences that provide nonprojectable
• propertiesThis meshed conceptualization,
• the meaning, is in the service of control of
• action in a three-dimensional environment
• (Glenberg 1997)
• Language is a surrogate for experience (Taylor
  and Tversky, 1992)

13
Summary Embodied Representations Symbol
Grounding

• Embodied Meaning action based coding of objects
  and situations
• Embodied representations are lawfully and
  analogically related to properties of the world
  (Harnard, 1990, 1993)
• Notion of mesh mutual modification of patterns
  of action (Glenberg, 1997)
• Meaning of a situation is a meshed pattern of
  possible actions embodied conceptualization.

14
Overview of the methods of cognitive
neuropsychology/science (I)

• Advances in science by technology
• Invention of the telescope in 1608 changed
  astronomers observational methods
• If well-formulated questions are not asked, even
  the most powerful tools will not provide sensible
  answers
• Cognitive Psychology / Computer Modeling
• Neuroanatomy
• Gross Neuroanatomy (general structures and
  connections)
• Fine Neuroanatomy (components of individual
  neurons)

15
Overview of the methods of cognitive
neuropsychology/science (II)

• Neurophysiology (experimental methods used with
  animals)
• Electrical stimulation
• Single-cell recording
• Lesions
• Genetic manipulations
• Neurology
• Structural imaging and neurological damage
• Causes of neurological disorders (vascular
  disorders,tumors, degenerous and infectious
  disorders, traumata, epilepsies)
• Functional neuro-surgery

16
Overview of the methods of cognitive
neuropsychology/science (III)

• Converging methods
• Cognitive deficits following brain damage
• Virtual lesions Transcranial magnetic
  stimulation (TMS)
• Functional imaging
• Electrical and magnetic signals in the brain
  (EEG, MEG)
• Metabolic signals
• Positron emission tomography (PET) regional
  cerebral blood flow
• fMRI blood oxygenation level dependent effect or
  BOLD effect

17
(No Transcript)
18
Brain functions (1810-1819)

• Do parts of the brain working independently
  enable the mind? (componential hypothesis)
• Franz Joseph Gall and J. G. Spurzheim
• 35 specific brain functions
• Language, color perception, hope, self-esteem
• With practice, areas grow, causing a bump in the
  overlying skull
• Anatomical personology
• phrenology

• Does the whole brain work in concert?(wholistic
  hypothesis)
• Pierre Flourens (1794-1867)
• All sensations, all perceptions and all volitions
  occupy the same seat in these cerebral organs.
• The faculty of sensation, percept and volition
  is then esssentially one faculty.
• Empirical evidence no matter where he leasoned a
  bird brain, the bird recovered

19
Language Areas

• Broca

• Wernicke

20
Neuroanatomist Korbinian Brodman (1909)

• Analyzed cellular organization of the cortex
• Tissue stains to visualize different brain
  regions
• To a large extent cytoarchitecturally described
  brain areas do indeed represent functionally
  distinct brain regions

21
Componential or wholistic? Again the question

• Camillo Golgi
• Developed stain that impregnated individual
  neurons
• Believed the whole brain to be a continuous mass
  of tissue that shares a common cytoplasm

• Cajal
• Used Golgi stains
• Identified the unitary nature of neurons
• Transmittion of information by electricity

22
How does the nervous system work (20-th century)

• Understand how single neurons behave and interact
• Knowing all the elements, can we figure out the
  system?
• Billions of neurons

• Brain-damaged humans show lack of typical
  symptoms
• Impossible to localize higher cognitive
  functions

Jackson Lesion might well affect other
structures in the brain because the lesion might
have damaged neurons connected to other regions
diaschisis damage of one part can create
problems for another. Gestaltist view The whole
is different from the sum of its parts
23
Summary

• Localists
• Wrong, in that they tried to map behaviors and
  perceptions into single locations in the cortex
• Any behavior is produced by many areas
• Complex functions
• Globalists
• A function can be achieved in numerous different
  ways (in this sense the globalists were right)
• But
• simple processes that are recruited to exercise
  an ability are localized

24
Event-Related Brain Potentials
25
What are event-related brain potentials (ERPs)

• Like EEG, but related to an event (a task, e.g.
  making a decision, reading a word, etc.)
• The ERP (a few µV is small in relation to the EEG
  (about 50µV)
• The international 10-20 system (Jasper 1958)
  allows for between-laboratory and
  between-experiment comparisons

26

• EEG profiles obtained during various states of
  consciousness
• After Penfield and Jasper (1954)

27
How ERPs are obtained from EEG-data
28
Schematic representation of ERP-Procedure
29
ERP-components
30
ERP-Components

• Usually labeled by polarity and latency, P300, P3
  (ordinal latency of the component)
• Scalp locations, e.g. frontal P300
• Psychological or experimental conditions
• Novelty P3
• Readiness potential
• Mismatch negativity, MMN
• Sensory or exogeneous
• Interaction subject response (task
  requirements) endogenous

31
From the brain to the scalp

• Distant manifestations of activations of
  populations of neurons (recorded on surface of
  skull)
• Requirements
• Neurons must act synchronously
• Electric fields must be oriented so that they
  cumulate
• Therefore only a subset of neural activity is
  visible
• Open field organizations (dentritic trees are
  ordered), neurons are organized in layers, most
  of cortex, parts of thalamus, cerebellum and
  others
• Presynaptic potentials (spikes) high frequency
• Postsynaptic potentials (slower), summation
  thereof

32
Forward and backward solutions inverse dipole
modeling
33
Conclusions from ERPs

• Just a sample of neuronal activity
• If you find the same effect in different
  experimental conditions
• If you do not find an effect

34
From the scalp to the brain Inferring the
sources of ERPs

• Observations are voltages differences between
  scalp electrodes and a reference electrode.
• Identify neural generators of ERPs
• indefinite number of unknown parameters
• No unique solution
• Head is not a homogeneous medium
• Difficult to compute
• Non-invasive and invasive techniques
• Dense electrode arrays and source monitoring
• Neurophysiological knowledge, other imaging
  techniques
• Invasive techniques
• Implanting electrodes, lesion studies with
  animals

35
The concept of components

• Voltage x time x location function.
• Segments of the ERP waveform to covary in
  response to a specific experimental condition
• Positive, negative
• Aspects of the ERP waveform
• In terms of neural structures that generate them
• But a peak may be the sum of several functionally
  and structurally distinct components

36
Quantification of ERP components

• Artifacts eyeballs, eyelids, muscles of the
  head.
• Signal to noise ratio
• ERPs are constant over trials
• Noise is random
• ERPs are independent of the background noise
• Peak measurements
• Covariation measures (e.g. covary with condition)
• Source-activity measures (algorithms for dipole,
  Loreta, Baillet Garnero, 1997) spatio-temporal
  dipole model distributed source models

37
Problems in Component Measurements

• Are components identical.
• Revisions of component classification
• Components overlap
• Principle component analysis (application
  statistics, linear algebra)
• Subtraction procedure, only amplitude, not
  latency vary across conditions

38
Experimental logic

• Discovery establishing functional significance
• Components antecedents (I.e. Experimental
  manipulations)
• Consequences of variation
• Speculations about the psychological or
  neuropsychological function it manifests

39
Psycho physiological inference

• Conditions are different
• Conditions differ at a particular time
• Conditions differ with respect to the latency of
  some process
• Conditions differ with respect to the degree to
  which some process occurs

40
Some ERP-findings

• Movement-related potentials
• Lateralized readiness potential (LRP)
• Contingent negative variation (CNV)
• Error-related negativity (ERN)
• Sensory components
• The early negatives (ERPs and locus of selective
  attention)
• The middle latency cognitive components (mismatch
  negativity of MMN)
• N200s (or N2)
• The late cognitive ERPs
• P300, elicited by deviant stimuli
• The frontal P3, elicited by novel stimuli,
  novelty P3 (no memory template is available

41
Some ERP-findings (continued)

• ERP effects associated with subsequent memory
• Distinctive word (van Restoff, character change,
  large P300s recalled ones show larger P300s as
  compared to not recalled ones.
• Same-different task (Sanquist et al., 1980)
  larger amplitude P300s were better recognized in
  subsequent recognition test.
• Two-process model of recognition (large P300 when
  explicit recollection as opposed to just know
• N 400 (language-related)
• More prolonged over the right rather than the
  left hemisphere
• N400 may be generated by the parahippocampal
  anterior fusiform gyrus
• A distinctively semantic process
• Inversely related to the subjects expectancy
  (cloze probability)
• Semantically related to sentence completion
  produce smaller N 400. The pizza was too hot to
  drink / cry.

42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
Brain Imaging
47
Brain Imaging

• Magnetic resonance imaging (MRI)
• Angelo Mosso (1846-1910) correlation blood flow
  neuronal activity
• Seymour Kety (1915-2000), Lou Sokoloff quantified
  relation (middle of the 20-th century)
• Position Emission tomography (PET)
• 1980s Michael Posner, Steve Peterson Study
  human cognition by PET Marcus Raichle
• Donders (1868) method of subtraction

48
Subtraction Method
49
Brain imaging (fMRI)

• Behavior of hydrogen atoms or protons in a
  magnetic field
• Paul Lauterbur MRI (Nobel-prize, 2003)
• Seiji Ogawa functional states of the brain
  (fMRI)
• Amount of oxygen carried by hemoglobin chances
  the degree to which hemoglobin disturbs a
  magnetic field
• Tracking blood flow
• BOLD-signal blood oxygen level dependent

50
(No Transcript)
51
(No Transcript)
52
(No Transcript)
53
(No Transcript)
54
Steps in analysis cortex-segmentation
55
Computational Modeling and fMRI
56

• A prominent cognitive architecture
• ACT-R
• (Anderson Lebiere, 1998)

57
(No Transcript)
58
(No Transcript)
59
(No Transcript)
60
How to map time predictions to the BOLD-signal
61
(No Transcript)
62
Experimental Predictions by the ACT-R model
63

• Comparison of model predictions and observations
  by a measure of proportionality

64
Summary

• Embodied memory representations
• New methods of cognitive science
• ERP
• fMRI
• Modeling