Prsentation PowerPoint

1
Phonology grounded in sensori-motor
speech Elements of a morphogenesis theory
J.L. Schwartz, C. Abry, L.J. Boë, M.A. Cathiard,
H. Loevenbruck, N. Vallée, A. Vilain ICP
(Institut de la Communication Parlée) Grenoble -
France
2
1. Perceptuo-motor links in speech
perception The Perception for Action Control
Theory (PACT) 2. Stabil loop, a cortical
circuit for stabilizing percepts and building
speech units 3. Vocalise to Localise 4.
Elements of a morphogenesis theory of speech
units 5. A neuro-computational project about
the grounding of human language in
perceptuo-motor cognition
3
1. Perceptuo-motor links in speech
perception The Perception for Action Control
Theory (PACT) 2. Stabil loop, a cortical
circuit for stabilizing percepts and building
speech units 3. Vocalise to Localise 4.
Elements of a morphogenesis theory of speech
units 5. A neuro-computational project about
the grounding of human language in
perceptuo-motor cognition
4
Auditory and Motor Theories of speech
communication
Two symmetric difficulties
5
The perceptual theory shortcomings The example
of vowel reduction
/i/
/i/
/a/
A common target is hidden in this acoustic
pattern (the speakers intention) It may be
recovered through procedural motor knowledge
(Viviani)
6
Paulo Viviani, the v r 1/3 law of human
gestures and the role of procedural knowledge in
action perception
7
The motor theory shortcomings The example of
vowel systems
8
Speech Gestures are shaped by Sensory Processing
(e.g. Stevens Quantal Theory)
9
A perceptual theory The Dispersion-Focalisation
Theory of vowel systems
Dispersion could mistakenly be considered as
supporting the  larynx theory  But actually,
the larynx is NOT needed to utter i, a, u
Boë et al., 1999, 2002
10
The  Perception for Action Control Theory 
(PACT) unification
Speech is shaped by the perceptuo-motor
link Learnt in development (in a developmentally
plausible scenario) Computationally implemented
through perceptuo-motor relationships (speech
robotics) Cortically implemented in a
temporo-parieto-frontal loop in the brain (the
dorsal route)
Speech is shaped by the perceptuo-motor link
Speech is shaped by the perceptuo-motor
link Learnt in development (in a developmentally
plausible scenario)
Speech is shaped by the perceptuo-motor
link Learnt in development (in a developmentally
plausible scenario) Computationally implemented
through perceptuo-motor relationships (speech
robotics)
Gestures are selected according to the
information they may provide for recovering
controls from stimuli
11
1. Perceptuo-motor links in speech
perception The Perception for Action Control
Theory (PACT) 2. Stabil loop, a cortical
circuit for stabilizing percepts and building
speech units 3. Vocalise to Localise 4.
Elements of a morphogenesis theory of speech
units 5. A neuro-computational project about
the grounding of human language in
perceptuo-motor cognition
12
Verbal transformations A window inside verbal
working memory and perceptuo-motor links in
speech perception
Marc Satos thesis, ICP 2001-2004
13
VERBAL TRANSFORMATION EFFECT in Adults
An experiment with French n-stable syllables
"PSE" is the winner/"EPS" the loser
Sato et al., Perception Psychophysics (2006)
14
The STABIL-LOOPAn ENACTIVE PERCEPTUO-MOTOR
WORKING MEMORY
left SMA
Wernicke
SMG
The Articulatory Loop network is recruited for
Verbal Transformation e.g.  SPE -gt  PES 
(Sato et al., Neuroimage 2004)
15
The STABIL-LOOP basic Circuit
An ENACTIVE PERCEPTUO-MOTOR WORKING MEMORY
16
Patient 1   Broca pars triangularis (BA45)
Target condition patapatapatapata
Control condition papapapapapatatatata
Active syllabic parsing in the IFG (intracranial
EEG) (Jean-François Lachaux, Philippe Kahane,
Anahita Basirat, Marc Sato, Jean-Luc Schwartz)
17
Patient 2   Broca pars triangularis (BA45)
Target condition patapatapatapata
Control condition papapapapapatatatata
18
1. Perceptuo-motor links in speech
perception The Perception for Action Control
Theory (PACT) 2. Stabil loop, a cortical
circuit for stabilizing percepts and building
speech units 3. Vocalise to Localise 4.
Elements of a morphogenesis theory of speech
units 5. A neuro-computational project about
the grounding of human language in
perceptuo-motor cognition
19

Language as showing by the voice

• Deixis (pointing) is involved during

- lexicon construction (at 8-10 mos) Semantic
grabber  whats that? 
- morphosyntax emergence (at 16-20 mos) Word
deictic gesture (Volterra Caselli, 1986 
Goldin-Meadow, 1999  Bates Dick, 2002).

• demonstratives in acquisition Syntactic
  precursors
• -  That!  presentative
• -  Thats Mum!  presentative
• -  Thats the sugar that goes in there!  
  relativizers
• -  Thats Mum that said that the sugar goes in
  there! 
• (Tomasello, 2003 cf. Lieven, Diessel)

20

• Acoustically
• deixis F0 raising lengthening on focal
  constituent
• F0 lowering on pre-focal constituent
• Deaccentuation of post-focal constituent
• Dohen Lvenbruck (2004)

21

• Articulatorily
• -hyper-articulation of the focal constituent
• -with optional hypo-articulation of post-focal
  constituent
• Lvenbruck (1999) Dohen Lvenbruck (2006)

22

23

• LIFG activation in all (deixisbaseline)
  contrasts

• The LIFG was therefore activated during verbal
  pointing at the agent of the action, through
  prosody and syntax.

• The results are consistent with
  neuropsychological and neuro-imaging studies on
  thematic-role tracking in syntactic processing.

• Our findings are also in line with neuroimaging
  studies on the observation and mental imagery of
  action.

• All these observations support our claim that
  the role of the LIFG is that of an
  action-structure parser
• In morphosyntactic encoding / decoding, the LIFG
  would handle the parsing of the predicate and its
  arguments,
• i.e. the monitoring of who-does-what-to-who
  m.

24

• Left insula in all (deixisbaseline) contrasts

• Involvement of the left precentral gyrus of the
  insula in articulatory planning during speech
  (Dronkers, 1996).

• Prosody has acoustic and articulatory correlates
  (e.g. Beckman et al., 1992 Dohen et al., 2004).

• The production of prosodic deixis (wrt the
  baseline) may require more accurate planning of
  the movements of the tongue and jaw and larynx.
• Hence significant increase in activation of the
  left insula.

• Similarly, syntactic deixis (wrt the baseline)
  requires more accurate articulatory planning,
  given the larger number of consonant clusters
  involved (due to schwa deletion).

25

• LSMG temporal cortex activation in the
• (prosodic deixisbaseline) contrast alone

• The SMG is often considered an association area,
  part of a network for spatial awareness, that
  integrates distributed multimodal sensory signals
  (somatosensory, visual, auditory) to form an
  interactive representation of space (Andersen,
  1997 Mesulam 1981, 1999).

1. Evidence for the role of SMG in manual
pointing (Astafiev et al., 2003).
2. Role of the left SMG in linguistic manual
pointing (i.e. deixis in sign language, Bellugi
et al., 1989.)
3. Role of SMG in working memory (Paulesu et al.,
1993 Jonides et al., 1998). Hickok Poeppel,
2000 inferior parietal cortex () serves to
interface sound-based representations of speech
in auditory cortex with articulatory-based
representations of speech in frontal cortex. gt
The Temporo-Parieto-Frontal network
26
The  That-Path  (la  ça-voie )

• Prosodic deixis, i.e. expressive orofacial
  (manual and facial for sign language) deixis may
  be considered in continuity with manual pointing.

sensory expectation
auditory description
action analysis

• Grammaticalization automatisation (no
  requirement of sensory expectations from ones
  own vocal actions)
• gt deparietalization

27
1. Perceptuo-motor links in speech
perception The Perception for Action Control
Theory (PACT) 2. Stabil loop, a cortical
circuit for stabilizing percepts and building
speech units 3. Vocalise to Localise 4.
Elements of a morphogenesis theory of speech
units 5. A neuro-computational project about
the grounding of human language in
perceptuo-motor cognition
28
A morphogenesis theory of language patterns in
the PACT framework should include both

• an action system producing contrasted gestures,
• in a developmentally plausible scenario

• and a perceptual system able to recover actions

29
The Vocalize to Localize framework (Abry et
coll.) Speech as the meeting point of a double
set of evolution mechanisms, providing the basis
for the birth of the double articulation of oral
language
Orofacial deixis Extending facio-manual deixis
towards words and syntax (Vocalize-to-Localize,
Abry et al.)
Jaw cyclicities Extending ingestion mechanisms
towards syllables for communication
(Frame-Content, MacNeilage Davis)
30
Abry, Vilain, Ducey
31
a. The birth of syllabic frames (Théorie Frame
then Content McNeilage Davis)
Coocurences /ba/ /di/ /gu/
32
b. The birth of orofacial controls From 7 to 18
mths, learning the mastering of independent and
coordinated control of vowels and consonants (
 contents )
Lalevée, Vilain
33
c. Two syllables in a word
Ducey, Lalevée, Abry, Vilain
34
d. Perceptual viability of speech
units (Dispersion-Focalisation)
35
e. Sequencing syllables

• The  Labial - Coronal  (LC) effect
• Human languages prefer variegated patterns
• (bana gt baba)
• LC effect more labial-coronal than
  coronal- labial sequences (pata gt tapa)

A jaw - tongue - lips synergy?
36
The Haken - Kelso speeding paradigm inspired from
synergetics
It is possible to concatenate two syllables on
one jaw cycle
provided that the two articulation places are
different
1. Speeding tapa provides pata (an
articulatory/perceptual ground to LC?)
1. Speeding tapa provides pata (an
articulatory/perceptual ground to LC?) 2.
Speeding papa more difficult than pata / tapa (an
articulatory ground to OCP?) (Rochet-Capellan
Schwartz 2005)
37

• The perceptual consequence
• In the verbal transformation effect, pata more
  stable than tapa
• (Sato, Vallée Schwartz 2006)

/ / / / / /
ta pa ta pa ta pa ta pa ta pa ta
38
Development of orofacial and deixis control
 provides  syllables, CV co-occurences,
bi-syllabic words
The Dispersion-Focalisation Theory  provides 
/a i u/ and /p t k/
Articulatory synergy / economy  provides  /pt/
sequences
Etc (the pati papa project)
39
1. Perceptuo-motor links in speech
perception The Perception for Action Control
Theory (PACT) 2. Stabil loop, a cortical
circuit for stabilizing percepts and building
speech units 3. Vocalise to Localise 4.
Elements of a morphogenesis theory of speech
units 5. A neuro-computational project about
the grounding of human language in
perceptuo-motor cognition
40
The speech robotics framework
sound
Direct model
Inverse model
gesture
sound
41
Language emerging from face-to-face,
mouth-to-mouth, brain-to-brain  mirror 
interactions
Mutual Attention System
Perceptuo-Motor Interactions
42
Thank you!