Hearing Lips and Seeing Voices

1
Hearing Lips and Seeing Voices

• by Harry McGurk and
• John MacDonald
• presented by Carmela Toews

2
Main claim and Background

• Vision influences speech perception
• Based on a study in which subjects who heard the
  sound baba dubbed onto the lip movements for
  gaga heard dada.
• If the sound gaga was dubbed onto the lip
  movements for baba, they heard bagba or
  gaba.
• If they just heard the sound with out seeing
  anyone talking, they reported accurately.

3
Current Study

• The goal of this study is to confirm and
  generalize the previous findings.
• Four films were made with contradictory audio and
  visual signals.
• baba/gaga
• gaga/baba
• papa/kaka
• kaka/papa

4

• Three groups of people were tested
• pre-school children (ages 3-5)
• primary school children (ages 7-8)
• adults (predominantly male, ages 18-40)
• The experiment had two parts
• Auditory/Visual
• Subjects watched a film and repeated what they
  heard the speaker saying.
• Auditory only
• Subjects faced away from the screen and repeated
  what they heard the speaker saying.

5
Results

• For the auditory only component, accuracy rates
  were high (above 91, 97 and 99 for
  pre-schoolers, school children and adults
  respectively).
• Errors were unsystematic
• For the auditory/visual component, accuracy was
  extremely low (41, 48 and 8 for the same three
  groups)
• Errors followed certain patterns

6
Auditory-Visual Response Patterns

• There were five types of responses
• Auditory (subject followed the auditory cue)
• Visual (subject followed the visual cue)
• Fused (subject combined the audio and visual cues
  into a new element)
• Combination (subject reported some unmodified
  elements from each modality)
• Other (everything else)

7
(No Transcript)
8
Percentages of Responses
9
Observations

• This experiment confirms the findings of the
  previous study
• Subjects tended to give fused responses for the
  ba-voice/ga-lips video and the pa-voice/ka-lips
  video
• The effect of the visual input seems more
  pronounced with ba/ga than with pa/ka

10

• Adults pay more attention than children to visual
  cues during the perception of speech.
• The results of this experiment are statistically
  significant.
• These effects do not habituate over time. People
  are still susceptible to them, even when they are
  aware of what is happening.

11
Conclusions

• The reason for the fused responses (hearing
  dada when the auditory signal is baba and the
  visual signal is gaga) may be that the auditory
  signals for ba and da have something in
  common that ga doesnt have, and the visual
  signal for da and ga have something in common
  that ba doesnt have.

12

• Therefore, when subjects are presented with
  auditory ba and visual ga, they hear
  something that partially matches with both.
• When subjects are presented with auditory ga
  and visual ba, the signals are contradictory
  because they have no shared features, so they
  vary between choosing b and choosing g (as in
  babga)
• These are interpretations of the results, not
  really conclusions.