DO MOUTHS SIGN DO HANDS SPEAK

1
DO MOUTHS SIGN? DO HANDS SPEAK?

• Bencie Woll
• University College London
• NIAS Workshop May 19-20, 2006

2
Introduction

• Gesture systems (home sign) can appear in the
  absence of linguistic input
• The sign languages of Deaf communities have
  sometimes been regarded as primitive
  communication systems and, the reasoning follows,
  as a precursor to spoken languages
• Sign languages are not primitive and are in fact
  full natural languages with complex grammars
• Nevertheless it is possible that sign languages,
  have features in common with manual precursors of
  spoken language
• Is there a plausible mechanism by which primarily
  manual actions could have transformed themselves
  into vocal actions?

3
Theories of language origins

• Ding-Dong onomatopoeia is the source of the
  first words imitative sounds mimicking the
  sounds of the world around them.
• Bow-wow imitation of animal sounds.
• Pooh-pooh - sighs of pleasure, moans of pain, and
  other semi-involuntary cries, which then became
  the names of the phenomena
• Uh-oh - warnings to other members of the human
  band.

4
Theories (cont.)

• Yo-he-ho - rhythmic chants and vocalisms uttered
  by people engaged in communal labour.
• Watch the Birdie - communication became
  elaborated because humans found it advantageous
  to be able to deceive other humans. Since
  exclamations and vocalisms can involuntarily
  reveal your true mental state, humans learned to
  feign them in order to deceive others for selfish
  advantage.
• Ta-ta - oral gestures that began in imitation of
  hand gestures that were already in use for
  communication.

5
Historical Perspectives

• there are actions of the mouth which are
  commonly performed under certain circumstances
  and which seem to be due to imitation or some
  sort of sympathy. Thus persons cutting anything
  may be seen to move their jaws simultaneously
  with the blades of the scissors. Children
  learning to write often twist about their tongues
  as their fingers move, in a ridiculous fashion.
• Charles Darwin, The Expression of Emotions in Man
  and Animals (1872)

6
Henry Sweet

• "Gesture .. helped to develop the power of
  forming sounds while at the same time helping to
  lay the foundation of language proper. When men
  first expressed the idea of 'teeth', 'eat',
  'bite', it was by pointing to their teeth. If the
  interlocutor's back was turned, a cry for
  attention was necessary which would naturally
  assume the form of the clearest and most open
  vowel. A sympathetic lingual gesture would then
  accompany the hand gesture which later would be
  dropped as superfluous so that ADA or more
  emphatically ATA would mean 'teeth' or 'tooth'
  and 'bite' or 'eat', these different meanings
  being only gradually differentiated".

7
Richard Paget

• The earliest human language was a language of
  gestures, in which features originally made by
  hand were unconsciously copied by movements or
  positions of the mouth, tongue or lips.
• Human Speech some observations, experiments and
  conclusions as to the nature, origin, purpose and
  possible improvement of human speech (1930)

8
Morris Swadesh

• The shape of objects is imitated in human
  gestures and from there passes into vocalisation.
  This is due to two circumstances. One is that in
  humans as in other primates, the lips are
  flexible and can be used to copy shapes, such as
  round or flat. The other is that the passage of
  air through spaces gives a resonance that is
  related to their shape e.g. a word like the
  Latin CAPIO I TAKE or English CAPTURE
• the formation of the K sound at the back of the
  mouth, while the lips are open, is comparable to
  the open hand. The closing of the lips then is
  analogous to the fingers closing with the thumb
  as one takes hold of an object. Thus the
  pronunciation of the root CAPIO is like the
  action of taking. Of course not all words are to
  be explained in this way, in fact only a few. And
  yet the possibility that some words developed in
  this way is not denied by other qualities also
  evident in language.

9
Allotts tests for these theories

• Evidence that gesture historically preceded
  speech and that the gesture-language fairly
  precisely expressed, by shape and movement, the
  objects and events referred to
• Evidence for a historical process by which those
  overt gestures were reflected, reproduced in
  miniature, in gestures particularly of the tongue
  and lips which were associated with the
  production of speech-sounds
• There can be no evidence for this as a
  historical process and Paget presents little to
  explain or justify the hypothesis as a reasonable
  physiological speculation

10
Allotts tests for these theories

• Evidence that gesture historically preceded
  speech and that the gesture-language fairly
  precisely expressed, by shape and movement, the
  objects and events referred to
• Evidence for a historical process by which those
  overt gestures were reflected, reproduced in
  miniature, in gestures particularly of the tongue
  and lips which were associated with the
  production of speech-sounds
• There can be no evidence for this as a
  historical process and Paget presents little to
  explain or justify the hypothesis as a reasonable
  physiological speculation
• Is there any contemporary evidence?

11
Summary

• Most of these writers suggest that the mouth
  actions themselves would share underlying imagery
  with the iconic manual gesture
• In the absence of any plausible mechanism or
  historical evidence, the notion of a hand-mouth
  link remains as speculative as any other theory

12
Signs and gestures

• full human languages
• used by modern humans with language-ready
  brains
• But they share some features with gestures
• common articulators
• greater iconicity than spoken languages
• How could a hypothesised highly iconic manual
  communication system have led to the creation of
  a vocal communication system in which the links
  between symbol and referent are for the most part
  arbitrary?

13
The articulators of sign language

• Not just manual there is a rich and complex
  role for other articulators body, face, and, in
  particular, the mouth.
• The research reported here focuses on one
  subgroup of these mouth actions 'echo phonology
• a repertoire of mouth actions not derived from
  spoken language
• form an obligatory accompaniment to some manual
  signs in a range of sign languages
• characterised by 'echoing' on the mouth certain
  of the articulatory actions of the hands.

14
Data

• narratives in 3 different European sign languages
• evidence from bilinguals
• functional imaging studies (fMRI) with deaf
  signers and hearing non-signers
• These provide examples of a possible mechanism
  in the evolution of language by which the units
  of an iconic manual communication system could
  convert into a largely arbitrary vocal
  communication system.

15
Neurobiological Perspectives

• Studies of neurons in the monkey brain by
  Rizzolatti and colleagues since 1996 have
  identified mirror neurons, which fire when the
  animal observes another individual making
  specific movements (primarily for reaching and
  grasping)
• The mirror system, in temporal, parietal and
  frontal regions, is part of a system specialised
  for perceiving and understanding biological motion

16
Hand and mouth - Gentilucci

• when humans are asked to open their mouths while
  grasping objects, the size of the mouth opening
  increases with the size of the grasped object
• Grasping larger objects and bringing them to the
  mouth induces increases in the size of mouth
  opening and voice spectra of syllables pronounced
  simultaneously
• Observing another individual grasping or bringing
  different sizes of objects to the mouth also
  affects the articulation of syllables

17
The hand and the mouth shared actions

• the anatomical closeness of hand and mouth
  related neurons in the premotor cortex may relate
  evolutionarily to the involvement of both in
  common goals
• The relationship between mouth actions related to
  eating, and those found in spoken language, have
  been discussed in detail by MacNeilage

18
Non-manual actions in sign languages (after Woll,
2001)
Mouth actions
Eyes, brows, head, body...
Mouthings derived from spoken language
Mouth gestures not derived from spoken language
Echo phonology
Adverbials
in loans e.g. to disambiguate
Enactions
in contact signing
19
The Mouth in Sign Language

• extensive use of the mouth for a variety of
  functions
• Mouthings borrowed from spoken words used to
  disambiguate manually homonymous forms
• Adverbials used to signal manner and degree
• Enaction (mouth-for-mouth) the mouth
  represents an action directly (e.g. CHEW, BITE
• Echo phonology

20
Echo Phonology

• obligatory in the citation forms of lexical signs
• not derived from spoken language
• specifiers of meaning, not adverbials added to
  citation forms
• not enactions
• the mouth action includes movement either the
  exhalation or inhalation of breath, or a change
  in mouth configuration during the articulation of
  the sign.
• features based on the articulatory features of
  the manual movement
• 3 BSL signs meaning succeed or win
• Three different oral patterns occur in these
  signs one cannot be substituted for the other
• the action of the mouth, while echoing that of
  the hands, is not in itself iconic

21
Examples of EP syllable types

• pa (SUCCEED)
• associated with one or two active hands, movement
  consists of hand separation and twisting, with
  single sharp action
• ?? (EXIST)
• wriggling or fingers, repeated shaking or
  twisting of wrists(s), no path movement
• hww or hyy (WIN)
• repeated shaking of wrist, no path movement
• ?p (THANK-GOD)
• closing and twisting of hand(s), sharp movement
• am or ?m (TRUE)
• hand closes and contacts passive hand, sharp
  movement
• ?p (DISAPPEAR)
• hand(s) close, sharp movement with abrupt stop
• pipipi (VARIED)

22
Cross-linguistic study

• Comparison of narratives in three sign languages
• the occurrence of echo phonology was compared
  with other types of mouth action

• of Echo phonology is low information-adding
  functions have priority over those where
  information is redundant

23
Some evidence from bilinguals

• Data from CODAs (hearing native signers)
• Written texts in CODA-CODA correspondence (ASL)
• Father fork in throat (father is frustrated)
• Echo phonology

24
Echo Phonology without the hands

• Have you done that poster?
• ??? (NOT-YET), Ill do it tomorrow
• It was terrible. It was like ?mp
  (END/absolutely over)
• I couldnt get a straight answer from anyone. It
  was completely pipip (VARIED/inconsistent)

25
Functional Imaging

• sign languages embody the structural and
  communicative properties of spoken language,
  while existing entirely within a wholly
  visuo-gestural medium
• Among other insights, they enable investigators
  to clarify the core components of language
  processing in distinction to those that reflect
  input or action characteristics of the language
  system.

26

• An fMRI study was undertaken to explore the
  following conditions
• silent speech (SS)
• BSL signs that require mouthings for
  disambiguation (DM)
• BSL signs that exhibit echo phonology (EP)
• BSL signs that require no specific mouth action
  (NM)

27
Stimulus characteristics
28
Research Questions

• to what extent does the pattern of activation
  during speech perception and sign language
  perception differ?
• does the perception of mouthings (DM) differ from
  signs without mouth (NM)?
• does echo phonology (EP) generate distinctive
  activation compared with mouthings (DM)?
• how do hearing non-signers differ from deaf
  signers?

29
Study participants

• Thirteen (6 female mean age 27.4 age range
  18-49) right handed participants
• congenitally, severely or profoundly deaf native
  signers, having acquired BSL from their deaf
  parents
• comparator group of 13 hearing English-speakers,
  with no knowledge of BSL, matched for age,
  gender, IQ and educational background.

30
Experimental stimuli
31
To what extent do the patterns of activation for
speech perception and sign language perception
differ?
32
Deaf native signers
No mouth
Echo phonology
Silent speech
Disambiguating mouth
33
Speechreading

• The major area of activation was perisylvian
  (superior temporal and inferior frontal) , with
  somewhat more extensive activation on the left
  than the right
• These findings conform with other recent studies
  and confirm that silent speech can activate
  regions in deaf peoples brains that have been
  identified as auditory speech processing regions
  in hearing people.

34
Deaf native signers
No mouth
Echo phonology
Silent speech
Disambiguating mouth
35
Sign language

• In all three sign language conditions, there is
  also activation in perisylvian regions
• This affirms that sign language in Deaf native
  signers activates core language regions that are
  typically found when hearing people listen to
  speech.
• Although both sign language and speech involve
  perisylvian regions, sign language perception
  activated more posterior and inferior regions

36
Does the perception of signs with mouthings (DM)
differ from signs with no mouth (NM)?

• If language (speech vs. sign) is the crucial
  reason for the more posterior activation found in
  BSL perception, then signs with disambiguating
  mouth and signs without mouth should be processed
  identically
• On the other hand, if the articulators used
  determine the areas of activation, then DM and NM
  signs should differ, with more posterior
  activation for the NM signs
• The data support the second alternative anterior
  activation characterised DM and posterior
  activation NM
• These findings are very similar to those
  exploring distinctive patterns of activation when
  viewing hand and mouth gestures that are
  unrelated to sign language.

37
Deaf native signers
No mouth
Echo phonology
Silent speech
Disambiguating mouth
38
Does echo phonology (EP) generate distinctive
activation compared with other mouthings (DM)?

• The contrast between DM and EP provides further
  insight into the cortical correlates associated
  with observing specific articulators within sign
  language
• DM generated relatively greater activation in a
  circumscribed region of the left middle and
  posterior portions of the superior temporal
  gyrus, while EP produced relatively greater
  posterior activation
• This can be considered to reflect the fact that
  DM is more speech-like than EP
• Thus EP appears to occupy an intermediate
  position between signs without mouth and signs
  with mouth actions derived from spoken language.

39
Deaf native signers
No mouth
Echo phonology
Silent speech
Disambiguating mouth
40
Implications

• Obligatory mouth actions differentially activate
  a circumscribed region within the middle and
  posterior portions of the superior temporal
  gyrus.
• But generally, a strong similarity between the
  patterns of distinctive activation for mouth
  actions and for hand actions
• core language processes appear to be similar for
  sign language and speechreading in Deaf native
  signers.

41
How do hearing non-signers differ from deaf
signers?

• Silent speech Hearing participants extensively
  activated the regions associated with listening
  to and articulating speech
• Sign language Knowledge of SL generates greater
  activation in posterior superior temporal
  regions, including Wernickes area, and greater
  activation in left than right cortical regions.
  Only when material can be linguistically
  processed does differential activation extend to
  the superior parts of the temporal lobe
• Echo phonology The only condition to generate
  differential activation (Deafgt hearing) in
  frontal regions, including premotor regions. In
  line with mirror neuron theory, this condition
  requires signers to engage motor circuitry
  possibly because the mouth actions observed
  reflect those of the hands in their dynamic
  patterning.

42
Speech
Deaf signers Hearing
nonsigners
43
No mouth
Deaf signers Hearing
nonsigners
44
Echophonology
Deaf signers Hearing
nonsigners
45
Conclusions

• One issue for those concerned with suggesting a
  link between gesture and word has always been how
  visually-motivated gestures could have been
  transformed into the largely arbitrary words of
  spoken language
• Echo phonology provides evidence for a possible
  mechanism

46

• the phenomenon appears to be fairly common across
  different sign languages (needs research in
  non-European sign languages)
• the oral activities in echo phonology are
  themselves not visually motivated.
• the inventory of elements looks very much like a
  system of maximal contrasts in a spoken language
  phonology
• The oral components can be used in speech
• fMRI suggests that echo phonology occupies an
  interestingly intermediate position between signs
  and words

47

• This paper represents only a very preliminary
  exploration of echo phonology Further research
  may provide more insights into the origins of
  phonological structure in spoken language, and
  from that into the evolution of human language.

48
Collaborators on the ECHO project

• Nijmegen University
• Els van der Kooij, Onno Crasborn, Annika
  Nonhebel, Wim Emmerik
• Stockholm University
• Johanna Mesch, Brita Bergman
• UCL/City University London
• Dafydd Waters, Bencie Woll, Rachel Sutton-Spence

49
Collaborators on Imaging the deaf Brain

• Cheryl Capek
• Dafydd Waters
• Bencie Woll
• Mick Brammer
• Ruth Campbell
• Tony David
• Philip McGuire

• Mark Seal
• Mairéad MacSweeney
• Jordan Fenlon
• Tyron Woolfe
• Zoë Hunter
• Steve Williams and the imaging team