Kathryn Wheeler

1
Action Verbs and Hand Motion An ACE Experiment
Kathryn Wheeler University of Hawaii
Manoa October 9, 2004
2
Visual Imagery

• When people process language, they imagine the
  visual scene.
• An eagle is in the sky. causes comprehenders to
  visualize an eagle, in above location, with
  spread wings.
• Monkey Neurons (Gallese et al. 1996 Rizzolatti
  et al. 1996)
• Mirror neurons become active during perception
  and execution of actions.

3
Visual Imagery

• fMRI studies (Buccino et al. 2001 Nyberg et al.
  2001 Wheeler et al. 2000)
• Action observation and recall activates the same
  neural structures as performing the action.
• Language and Shape/Orientation (Stanfield Zwaan
  2001 Zwaan et al. 2002)
• Verbal input affects visual processing.

4
Motor Imagery

• While visual imagery involves creating a visual
  image of a scene, motor imagery involves
  imagining performing an action.
• Ehrsson et al. 2002 Lotze et al. 1999 Porro et
  al. 1996 Kosslyn et al. 1999.
• As in the Leaving the House scenario.

5
What do we know?

• Hand shapes (Klatzky et al. 1989)
• Hand shape priming facilitates action
  comprehension.
• When subjects are primed with a kind of
  handshape, they are much quicker to make
  decisions about a sentence involving that shape.
• Image-Verb Matching (Bergen et al. 2003)
• Interference arises between effector-specific
  verbs and picture stimuli when they depict
  different actions that use the same effector.

6
Action-Sentence Compatibility Effect

• ACE (Glenberg Kaschak 2002)
• Subjects read sentences implying motion away from
  the body or toward the body. In making a
  sensibility judgement, they were much quicker to
  make a movement to respond in the same direction.
  
• You handed Andy the pizza. (motion away)
• Andy handed you the pizza. (motion toward)
• There is an interaction between implied
  sentential direction and action execution.
• Therefore, action execution and language
  understanding use shared neuro-cognitive
  mechanisms.

7
Two Questions Remain

• When processing language that is not about you
  (the comprehender), do you still simulate action
  as measured by a facilitatory response to
  compatible actions? (Bailey 1997 Bergen Chang
  2004)
• The delivery boy handed Andy the pizza.
• To what level of detail to you simulate in order
  to understand? Is it simply direction of motion,
  or does this include more fine motor details?

8
Theory

• In order to understand motion language that
  implies motor action, we must simulate the fine
  motor skills necessary to perform the action
  (Bailey 1997).
• Eg. The student is grabbing the telephone.

9
Method

• We presented subjects sentences that implied
  either a closed fist hand shape (grab, punch) or
  an open palm hand shape (pet, slap).
• The singer is gripping the microphone. (fist)
• The nanny is patting the cushion. (palm)

10
Forced Choice Task

• We then had them respond to a sensibility task by
  pressing a button with a hand shape that was
  either compatible or incompatible.
• palm
• fist

11
Response

• In the instructions, subjects were told to press
  the response button with one hand shape (fist or
  palm) if the sentence made sense and the other
  hand shape if the sentence did not make sense.
• Sensible
• The whale is spouting.
• Not Sensible
• The ants are bubbling.
• The nurse is digging the rope.
• Half-way through, the function of the handshapes
  was reversed.

12
Procedure Trials

• Fixation Cross appears until subject is ready to
  continue.
• Subject presses and holds Enter key.
• Visual stimulus appears in the center of the
  screen and remains until the Enter button is
  released. (Average 2058 ms)
• When the subject decides whether the sentence is
  sensible, she releases the Enter key in order to
  respond. (Average 556 ms)
• Subject presses black pedal to respond with the
  pre-assigned fist or palm for yes or no.
• Fixation Cross for the next trial appears

13
Items

• Each subject saw each item once, either in the
  first half or second half.
• Item totals
• 8 palm, 8 fist 16 critical trials
• 144 fillers (transitive, intransitive half
  sensible, half non-sensible) 160 items total

14
Experiment Versions

• Four versions of the experiment were created to
  fully cross the two blocks (1 and 2) with the two
  hand shape responses fist and palm in both
  the yes and no condition.
• The order of the answer pairs (Fistyes, Palmno
  vs. Fistno, Palmyes) was reversed for half of
  the subjects (Glenberg Kaschak 2002).
• Subjects saw both Block 1 and Block 2 in either
  1-2 order or 2-1 order.

15
All Conditions
The order of which block and which hand meant yes
first was fully crossed.
16
Blocks Practice

• The two blocks were pseudo-randomized.
• Each block had the same number of sentences of
  each type (palm grab or fist slap).
• Subjects had a practice session with 16 trials
  prior to starting the main experiment.
• Since subjects had to reverse the response hand
  for the second half, the second practice session
  included 20 trials.

17
Hypothesis

• Subjects should respond more quickly when the
  sentential hand shape is compatible with the hand
  shape required to respond.
• Because motion verbs prime information about the
  action (Glenberg Kaschak 2002).

18
Subjects

• 65 subjects participated for class credit or five
  dollars.
• All subjects were right-handed, native English
  speakers.
• 18 were eliminated due to incorrect task
  performance or an error rate outside 3 StdDev of
  the subject mean.
• 47 remaining subjects were analyzed 10-13 per
  version.

19
Predictions

• We predict a compatibility effect between the
  type of hand shape implied in the sentence
  (grabbed fist, petted palm) and the type of
  hand shape the subject makes to respond (fist or
  palm).

20

• F(1, 47) 2.48, P .12

21
Results

• The response button RT showed numbers approaching
  significance in the interaction between the
  sentence type handshape and the response
  handshape.
• This indicates a compatibility (ACE) effect.
• Compatibility is in the right direction.
• With more subjects, this number should reach
  significance.

22
Results

• The release RT showed no significant results,
  unlike the Glenberg Kaschak (2002).
• May be due to processing time course. Glenberg
  looked at broad motor actions which may be
  processed first, while fine motor actions may
  occur later.

23
Implications

• If there is facilitation between action language
  and action performance to the level of fine motor
  detail, this is further support for the theory of
  simulation in language comprehension.
• Understanding involves not only broad aspects of
  lexical semantics, but also details of motor
  performance.
• Not just accessing lexical items but initializing
  the components of simulated action.

24
Future Directions

• Passivized sentences Does the effect persist
  when the agent is backgrounded?
• The microphone was grabbed by the monkey.
• Simultaneous processing and performance We would
  expect interference as in the Bergen et al.
  (2003).
• Aspect Do you get more simulation with
  imperfective aspect?
• The butcher was patting the cat.
• The butcher had patted the cat.

25
Mahalos!

• Ben Bergen
• Kristen Ciano
• Amy Schafer
• Meylysa Tseng

26
References

• Bergen, B., S. Narayan, and J. Feldman. 2003.
  Embodied verbal semantics evidence from an
  image-verb matching task. In Proceedings of the
  Twenty-Fifth Annual Conference of the Cognitive
  Science Society.
• Buccino, G., Binkofski, F., Fink, G.R., Fadiga,
  L., Fogassi, L., Gallese, V., Seitz, R.J.,
  Zilles, K., Rozzolatti, G., and Freund, H.J.
  2001. Action observation activates premotor and
  parietal areas in a somatotopic manner an fMRI
  study. Eurpoean Journal of Neuroscience, 13 (2)
  400-404.
• Ehrsson, H.H., Geyer, S., and Naito, E. 2003.
  Imagery of voluntary movement of fingers, toes,
  and tongue activates corresponding body-part
  specific motor representations. J. Neurophysiol.
  90 3304-3316.
• Gallese, V., Fadiga, L., Fogassi, L., Rizzolatti,
  G. 1996. Action recognition in the premotor
  cortex. Grain 119 593-609.
• Glenberg, Arthur M., and Kaschak, Michael P.
  2002. Grounding Language in Action. Psychonomic
  Bulletin Review, 9 (3), 558-565.
• Klatzky, Roberta L., Pellegrino, J., McCloskey,
  B., and Doherty, S. 1989. Can you Squeeze a
  Tomato? The Role of Motor Representations in
  Semantic Sensibility Judgements. Journal of
  Memory and Language 28, 56-77.
• Kosslyn, S.M., Ganis, G., and Thompson, W. L.
  (2001). Neural foundations of imagery. Nature
  Reviews Neurosci, 2, 635 -642.

27

• Lotze, M., Montoya, P., Erb, M., Hülsmann, E.,
  Flor, H., Klose, U., Birbaumer, N., Grodd, W.
  (1999) Activation of cortical and cerebellar
  motor areas during executed and imagined hand
  movements An fMRI study, Journal of Cognitive
  Neuroscience, 11(5) 491-501
• Rizzolatti, G., Luciano, F., Vittorio, G.,
  Fogassi, L. 1996. Premotor cortex and the
  recognition of motor actions. Cognitive Brain
  Research, 3131-141.
• Stanfield, R.A. Zwaan, R.A. (2001). The effect
  of implied orientation derived from verbal
  context on picture recognition. Psych Science,
  12, 153-156.
• Porro CA, Francescato MP, Cettolo V, Diamond ME,
  Baraldi P, Zuian C, Bazzocchi M, di Prampero PE
  (1996) Primary motor and sensory cortex
  activation during motor performance and motor
  imagery a functional magnetic resonance imaging
  study. J Neurosci 1676887698.
• Nyberg, L., Petersson, K.-M., Nilsson, L.-G.,
  Sandblom, J., Åberg, C., Ingvar, M. (2001).
  Reactivation of motor brain areas during explicit
  memory for actions. NeuroImage, 14, 521-528.
• Wheeler, M. E., Petersen, S. E., Buckner, R. L.
  (2000). Memorys echo Vivid remembering
  reactivates sensory specific cortex. Proc. Natl.
  Acad. Sci. USA 97 1112511129.
• Zwaan, R.A., Stanfield, R.A., Yaxley, R.H.
  (2002). Do language comprehenders routinely
  represent the shapes of objects? Psychological
  Science, 13, 168-171.

28
Accuracy Rate

• Sensible Sentences 92
• Not Sensible Sentences 90

29
Design

• The experiment was designed using E-prime.
• Responses were recorded on a Sony video camera
  and later transcribed to check accuracy.
• Both release RT and response RT were recorded.
• All subjects were run in a sound-attenuated booth.

30
Add me?

• Insert stuff about what we expect
• Right after subjects, before graph
• Remember!