Language Embedded in Systems of Perception and Action

1
Language Embedded in Systems of Perception and
Action

• Rolf A. Zwaan
• Florida State University

2
Experiential Traces
3
Experiential Traces
linguistic
referential
4
Experiential Associations
intra-linguistic
intra-referential
5
Experiential Associations
cross-domain
6
Hebbian Learning

• Co-occurrence primary factor in the forming of
  associations
• Intra-linguistic
• Intra-referential
• Cross-domain (grounding).

7
Experiential Traces

• Linguistic Traces
• Auditory
• Motor
• Visual
• Tactile

• Referential Traces
• Visual
• Motor
• Auditory
• Olfactory
• Tactile
• Somatosensory

8
Capturing Second-order Correlations
r2 .30
9
Capturing Second-order Correlations
r2 .65
10
Utility of Linguistic Traces/Associations

• Additional layer of associations.
• May enhance fluency.
• Referential bootstrapping.

11
Referential Bootstrapping

• Name this animal
• What has horns like a giraffe, a deer head, a
  horse neck, and legs like a zebra?
  http//www.westerville.k12.oh.us/ZOO/Orlich/pages/
  Okafrtpg.htm

12
Referential Bootstrapping
13
Resonance

• Experiential traces resonate with linguistic
  input.
• Visual resonance.
• Motor resonance.
• The role of resonance in comprehension.

14
Predictions

• Prediction 1
• Language comprehension affects subsequent
  perceptual performance.
• Prediction 2
• Incidentally acquired visual representations
  affect subsequent language processing.
• Prediction 3
• Language comprehension affects concurrent motor
  performance.

15
Comprehension Affects Perceptual Performance
(Zwaan, Stanfield, Yaxley, 2002)

• The ranger saw the eagle in the nest.
• SAW, RANGER, EAGLEIN, EAGLE, NEST
• The ranger saw the eagle in the sky.
• SAW, RANGER, EAGLEIN, EAGLE, SKY

16

• The ranger saw the eagle in its nest.
• The ranger saw the eagle in the sky.

17
Picture Recognition and Naming Latencies
(Zwaan, Stanfield Yaxley, 2002)
RT (ms)
18
Language may Involve Dynamic Representations(Zwaa
n. Madden, Yaxley, Aveyard, 2004)
The midfielder passed the ball to you.
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
Zwaan, Madden, Yaxley, Aveyard, 2004
24
Prediction 2

• Incidentally acquired visual representations
  affect subsequent language processing.

25
The Visual-Memory Paradigm (Aveyard, Zwaan,
Radach, Vorstius, in preparation)

• Memory Phase
• Word-picture verification (4 times)
• Reading Phase (ostensibly unrelated)
• Eye movements tracked

26
Memory Phase
eagle
27
Memory Phase
28
Reading Phase

• The ranger was in the park on his morning round.
  It was a bright day. In the sky there was an
  eagle, soaring above the prairie.

29
First Fixations
30
Gaze Durations
31
Visual World Visual Memory

• Situated
• Visual Search
• Auditory Presentation
• DV eye fixations during display viewing

• Re-situated
• Memory Search
• Visual Presentation
• DV eye fixations during reading

32
Prediction 3

• Motor traces are activated by linguistic input.

33
Action Compatibility Effect(Glenberg Kaschak,
2002)

• Sensibility judgments
• Close the drawer.
• Andy delivered the pizza to you.
• Liz told you the story.
• Break the cloud.

34
Action Parameters and Motor Resonance

• Actions have various parameters.
• Direction.
• Force.
• Manner (e.g., open hand vs. closed hand).
• Speed.
• Duration.
• Which action parameters exhibit resonance in
  language comprehension?

35
Measuring Response Force
36
Measuring Response Force
37
Response Force(Zwaan Taylor, in prep.)

• Sensibility judgments.
• Verbs.
• He pushed the car. (high)
• He started the car. (low)
• He admired the car. (no)
• Nouns.
• He broke the baseball bat. (high)
• He broke the bread. (low)
• He broke the promise. (no)

38
Response Force Verbs
Response Amplitude (kPa)
39
Response Force Nouns
Response Amplitude (kPa)
40
Manner of Action

• Sustain vs. Release
• He gripped the hammer.
• He dropped the hammer.
• Predictions
• Sustain faster than Release (ACE).
• Sustain longer than Release (ACE).

41
Sustain vs. Release Response Speed
42
Sustain vs. Release Response Duration
10
43
Sustain vs. Release Response Duration gt10
44
Motor Resonance

• What is the role of motor representations in
  language comprehension?
• Instrumental (Glenberg)
• Ornamental (many skeptics)
• Useful/necessary in specific cases (Jackendoff)

45
Motor Resonance

• What is the role of motor representations in
  language comprehension?
• Useful/necessary when actions are
• Concrete
• Punctate
• Simple
• Lets first ask How is motor resonance modulated
  by linguistic input?

46
How is motor resonance modulated by linguistic
input? (Zwaan Taylor, 2006, JEP General)

• Reading by rotating.
• Every 4 deg. new frame.
• He /realized /that /the music /was /too loud/so
  he /turned down/the /volume.
• 60 psychology undergraduates.

47
Reading-by-Rotation
48
Is Motor Resonance Short-lived or Focus-bound?
(Taylor Zwaan, in prep.)

• Perspective change.
• After/lighting/the candles/for the/romantic/evenin
  g,/he/dimmed/the/lights.
• Maintaining focus on the action.
• He lit/the candles/for the/romantic/evening./He
  noticed/the bright/lights,/which
  he/dimmed/slowly.
• Reading-by-rotating.

49
Motor Resonance is Focus-bound
50
Strong Test of Linguistic Focus Hypothesis

• Action-modifying vs. subject-modifying adverbs
• which he turned down slowly.
• which he turned down obediently.
• Stay tuned.

51
Conclusions Outlook

• Language is embedded in systems of perception and
  action.
• Rapid exchange of information.
• But it needs to be determined to what extent it
  is in bed with these systems.
• Perceptual motor resonance ornamental or
  instrumental?

52
Acknowledgements

• Grants from the National Science Foundation (NSF)
  and the National Institute of Mental Health
  (NIMH)
• Graduate students (past present)
• Rob Stanfield, Carol Madden, Rich Yaxley, Mark
  Aveyard, Larry Taylor
• Postdoctoral Fellows (past present)
• Barbara Kaup, Dave Therriault, Johanna Kaakinen,
  Tobias Richter
• Faculty Collaborators
• Mike Kaschak, Ralph Radach, Katinka Dijkstra

53
(No Transcript)