Situation Models and Embodied Language Processes

1
Situation Models and Embodied Language Processes

• Franz Schmalhofer
• University of Osnabrück / Germany

• Memory and Situation Models
• Computational Modeling of Inferences
• What Memory and Language are for
• Neural Correlates
• Integration of Behavioral Experiments and Neural
  Correlates (ERP fMRI) by Formal Models

2
Text comprehension

• Mary heard the ice-cream van coming.
• She remembered the pocket money.
• She rushed into the house.

3
Kintsch, Welsch, Schmalhofer Zimny (1990)
4

• How the strengths of the different memory
  representations can be empirically determined

Original/Explicit Mary heard the ice cream van
coming. Paraphrase Mary noticed the ice cream
van coming. InferenceMary picked up her pocket
money. False/Incorrect Mary was 50 years of age.
5
Experimental Indentification of the Different
Processing Levels
6
Propositional Representations
Many cognitive science theories assume that
knowledge and/or the meaning of sentences is
represented by propositions, semantic nets and
the like (e.g. Anderson, 1976 Kintsch, 1974
Collins Quillian, 1969 Schank 1975, Schank
Abelson, 1978)
Example The propositional representation of the
sentence George loves Sally LOVES(GEORGE,
SALLY)
Compare to Two word sentences of children during
language learning Protolanguage (Bickerton,
1981, 1995)
7
The difference between pictures and perceptions
Situation models are formed by perceptual
symbols
8
Comprehension

• Comprehension includes a large range of topics in
  cognitive psychology
• pattern recognition,
• knowledge representations,
• Working memory,
• Recognition and recall,
• learning, problem solving and decision making

Kintsch, W. (1998). Comprehension as a Paradigm
for Cognition
9
The Construction-Integration Model (Kintsch 1992)

• Comprehension a two phase process
• Construction
• Constructing mental units and interconnecting
  them in a network
• Integration
• Integration of constructed units via a context
  sensitive process

10
The Construction-Integration Model (Kintsch 1992)
11
Text Comprehension

• Up to the 1980s language comprehension was
  mostly viewed as the representation of the
  meaning of the text itself (focus on
  propositional representations)
• Now language is viewed as a set of processing
  instructions (Zwaan, 2004) on how to construct a
  mental representation of the described situation
  (mental model or situation model) (Johnson-Laird,
  1983 van Dijk Kintsch, 1983)

12
Situation Models as Event Indices (Zwaan and
Radvansky, 1998)

• The Event-indexing model of Zwaan Radvansky
  (1998) suggests that readers monitor five indexes
  (aspects) of the evolving situation model at the
  time when they read stories
• Protagonist
• Temporality
• Causality
• Spatiality
• Intentionality
• Or more generally space, time, causes, agents,
  intentions
• In other words everything that is relevant for
  planning actions and predicting future perceptions

13
How do people acquire knowledge from different
materials?
Properties of

• Text
• general, in a natural language
• relatively short
• sentence may describe a single attribute of a
  concept
• The function first requires one argument. The
  argument of the function first must be a list.
  The function first returns the first element of
  the argument.
• Text of LISP function description

• Examples
• specific,
• possibly a large set of examples required
• exemplification of many attribute instances
• (FIRST (A B)) ? A(FIRST ((A) B)) ?
  (A)(FIRST (A (B C))) ? A(FIRST (A)) ? A

The Function First
Informational Equivalence of different learning
materials
14
A unifying model (KIWi-Model)Schmalhofer (1998)
common sense
related domain

knowledge
situation model
text repres.
Sensory encoding
direct experience
text
15
Regressive eyemovements during reading
16
Reading times for studying examples
17
Memory retrieval after learningfrom text or
examples

• text novices
• text experts
• example novices
• example experts

18
Correct Responses in Example Verification Task
as a Function of Different Amounts of Text
19
Summary of the tested predictions of KIWi-Model

• KA is a goal-driven process, consisting of
  construction and integration phases
• Text and examples can be equated for
  informational contents
• The material-related representations are
  constructed by general heuristics, the situation
  model depends on domain knowledge
• Experts construct and use deep knowledge
  (situation model), novices rely on
  material-related representations
• Integrative KA (perception, language and memory)
  instead of the dominance of one source

20
Research on

• Text Comprehension(learning from text)
• memory issues
• encoding processes
• retrieval
• representational issues

• Concept Formation(learning from examples)
• search processes
• hypothesis formation
• concept identification

and
memory paradigm
Problem solving paradigm
Two alternative ways of acquiring knowledge
21
Comparison to Other Cognitive Models
22
Importance of Unified Theories

• Consistent and consensual theories drive
  cumulative progress
• The chronology of research produces a need for
  readjustment of the mappings between theoretical
  constructs and empirical data
• Natural Texts versus Textoids
• Changes in Task Types (Priming, Reading time,
  Memory)
• Changes in Available Methods (lateralized
  presentation,fMRI, ERP)
• Possibility that some controversies can be
  resolved by synthesis into a unified theory
• For Example Predictive and Bridging Inferences

23
Types of inferences (Graesser et al., 1994)

• Referential
• Case structure assignment(role e.g. agent)
• Causal antecendent
• Superordinate goal
• Thematic
• Character emotional reaction
• Causal consequence
• Bridging
• Predictive

• Mary heard the ice-cream van coming.She
  remembered the pocket money.She rushed into the
  house.
• What types of inferences are there and when are
  they drawn?

24
Landscape of Inferences (from Graesser, 2003 HC)

• TYPES OF INFERENCES
• Referential
• Case structure role assignment
• Instantiation of a noun category
• Superordinate goal
• Superordinate goal or action
• Instrument
• Causal antecedent
• Causal consequence
• Character emotional reaction
• Emotion of reader
• State
• Themes
• Authors intent

Causal consequence The inference is on a
forecasted causal chain, including physical
events, psychological events, and new goals,
plans, and actions of agents. Causal
antecedent The inference is on a causal chain
that bridges the current explicit action, event,
or state to the previous passage context.
25
Construction and persistence of predictive and
bridging inferences (e.g. McKoon Ratcliff,
1986 Potts et al., 1988 Keefe McDaniel (1993)
26
Overarching theoretical assumptions

• Kintschs (1998) C-I theory
• Multi-level representation surface-level,
  propositional text representation and situation
  model
• Processing cycles with construction-integration
  phases
• Enhancing assumptions
• Situation models are built from perceptual
  symbols (Zwaan et al. 2001) they often build a
  visuo-spatial representation (Fincher-Kiefer,
  2002)

• Instead of a nominal distinction of inference
  types, like predictive, bridging, causal etc.
  inference,a functional description of cognitive
  processes
• Similar to object constancy in visual perception,
  a situation constancy is postulated in the
  formation of situation models
• Inferencing achieves this situation constancy,
  i.e. inferencing as a pattern completion process

27
The 2nd processing cycle for the explicit and
predictive conditions
28
Model predictions of the inference encoding
scores for Keefe McDaniel data

• MODEL
  DATA
• Input Cycle 2 3 2 3
• __________________________________________________
  ____
• Explicit 33 32 30 33
• Predictive Inference 24 6 35 4
• Bridging Inference 22 22
• __________________________________________________
  ____

29
Evaluation of experimental predictions

• Experiment 2 (instructions) x 4 (text) mixed
  factorial design
• 1) situation condition elaborate on the context
  described in the passage
• 2) concentrate on the precise wording of the
  sentences
• Textmaterials from McKoon Ratcliff (1986),
  Potts et al. (1988), .
• Latencies in word pronunciation task
• Sentence recognition task Pr(yesold) as
  dependent measure

30
Reaction time in ms in Pronunciation Task (to
inference targets)
Predictive Bridging Explicit Control
Situation 569 556 571 614 Word
595 576 550 609  
31
Text- and situation focused reading (3-rd
processing cycle)

• Model Data
• Reading Focus text situation text situation
• __________________________________________________
  ________________
• Condition
• Explicit 23 63 59 43
• Predictive inference 5 61 14 45
• Bridging inference 21 41 33 58
• __________________________________________________
  _______________

32
Sentence Recognition Task

• Explicit probe
• The cameraman was preparing to shoot closeups.
• Inference probe
• The actress was pronounced dead.
• Elaboration probe
• The actress died from her injuries.
• Inconsistent probe
• The actress lived a long life.

33
Pr (explicitly mentioned) in sentence
recognition task
Text Types Predictive Bridging
Explicit Control _______________________
__________________________________________________
________ Reading focus text
situation text situation text situation text
situation ________________________________________
_____________________________________
Explicit .72 .82 .75 .94 .77
.86 .51 .52 Inference .22
.29 .27 .36 .93 .87 .15
.11 Elaboration .24 .33 .26 .42
.23 .38 .15 .16
Inconsistent .11 .12 .14 .05 .11
.04 .12 .11
34
Strength of situational representation for the
critical consequence as d-values (from
elaboration and inconsistent statements)
35
Griesel, Friese Schmalhofer (2003)
36
Modeling predictive and bridging inferences in
comparison to explicit statements

• Differences in interconnectivity are the key
• High interconnectivity at situation level
  (predictive)
• compared to
• High interconnectivity at the propositional level
  (explicit)
• Schmalhofer, McDaniel, and Keefe (2002)
• Focus on situation and time course may even keep
  predictive inferences activated in a later
  processing cycle (this was also the model
  prediction)
• McDaniel, Schmalhofer, and Keefe (2001)

37
Application to Beeman, Bowden and Gernsbacher
(2000) data

• Differential contribution of LH an RH in
  inference generation
• LH fine semantic coding
• Strong activation of small semantic fields
• RH coarse semantic coding
• Weak activation of large semantic fields

• Both hemis process in parallel with sharing at
  critical times

• Activation of hemis assessed at predictive and
  bridging inference points

• Mapping to CI model
• LH is verbatim and propositional
• RH is situation

38
Experiment of Beeman et al. (2000)
predictive LH - RH
bridging LH RH
predictive LH - RH
39
Beeman, Bowden and Gernsbacher (2000)
40
(No Transcript)
41
Relative frequency of yes-responses in the
verification task for left visual and right
visual field presentations (Griesel et al., 2003)
42
Mean latencies in ms of the yes-responses in
the verification task for left visual and right
visual field presentations
43
Summary

• Experimentation for differentiation Theorizing
  for integration
• Theories of text comprehension can be
  instantiated to simulate data from multiple
  experiments in detail
• Systematic relation of dependent and independent
  variables to the different conceptual entities in
  models
• Integration of existing data and theories is
  exciting, especially in view of ERP and new brain
  imaging data, related to inferencing