PERCEPTION OF MUSIC BY PATIENTS WITH COCHLEAR IMPLANTS

1
PERCEPTION OF MUSIC BY PATIENTS WITH COCHLEAR
IMPLANTS

• Jaan Ross (Tartu and Tallinn)
• Inna V. Koroleva and Jelena A. Ogorodnikova (St.
  Petersburg)

2
Cochlear implantation (1)

• Cochlear implantation is one of the most
  perspective directions in the rehabilitation of
  patients with significant hearing losses
• Cochlear implant is a surgically implantable
  device that provides hearing sensation to
  individuals with severe-to-profound hearing loss
  who do not benefit from hearing aids
• People with hearing losses in such range have
  absent or malfunctioning sensory cells in the
  cochlea

3
Cochlear implantation (2)

• In a normal ear, sound energy is converted to
  mechanical energy by the middle ear, which is
  then converted to mechanical fluid motion in the
  cochlea. Within the cochlea, the sensory cells
  (the inner and outer hair cells) are sensitive
  transducers that convert that mechanical fluid
  motion into electrical impulses in the auditory
  nerve.

4
Cochlear implantation (3)
Cochlear implants are designed to substitute for
the function of the middle ear, cochlear
mechanical motion, and sensory cells,
transforming sound energy into electrical energy
that will initiate impulses in the auditory nerve
Implant systems consist of both internal and
external components
5
Cochlear implantation (4)
The external components (over or next to the ear)
include (1) a microphone, which converts sound
into an electrical signal, (2) a speech
processor, which manipulates and converts the
signal into a special code and (3) a transmitter,
which sends the coded electrical signal to the
internal components
The surgically implanted components include (A) a
receiver, which decodes the signal from the
speech processor, and (B) an electrode array,
which stimulates the cochlea with electrical
current
The systems are powered by batteries located in
the speech processor
6
Rehabilitation (1)

• Rehabilitation is the very important
  post-surgical stage of cochlear implantation
• It is a complex and long process which
  essentially depends on individual characteristics
  of the patient (auditory experience, general
  abilities and the absence of mental defects)
• It is accepted to distinguish between the two
  groups of patients with cochlear implants
• pre-lingual - the hearing loss occurred before
  the patient has learned to speak (in general,
  before the second year of life) and
• post-lingual - when the hearing loss occurred in
  the age of four years or later (after the patient
  has learned to speak)

7
Rehabilitation (2)

• The main task of rehabilitation
• for post-lingual patients - the regeneration of
  the ability to perceive and to understand spoken
  language (the forming of connections between the
  distorted auditory information and previous
  internal patterns of speech and other sounds)
• for pre-lingual patients - the development of
  the foundations of auditory perception from the
  beginning (includes the basic elementary
  operations - detection and discrimination of
  sounds) and building up the whole system of
  spoken language
• So for all listeners with cochlea implants a
  primary rehabilitation problem is the creation or
  correction of new auditory patterns

8
Rehabilitation and music

• cochlear implants have been designed primarily to
  enhance speech perception
• many implant recipients anticipate enjoying music
  following the implantation
• in fact, up to 38 per cent of implant recipients
  report they do not enjoy listening to music with
  their device
• up to 86 per cent of post-lingual recipients
  report lower scores of listening habits after
  implantation
• musically oriented rehabilitation programs should
  be used in order to narrow the gap between
  pre-surgical expectancies of the implant
  recipients and their post-surgical experiences

9
Hypothesis

• pitch has been reported to be recognized with
  more difficulty than rhythm or timbre by implant
  recipients (because frequency encoding is robust
  with an implant), therefore temporal and timbral
  aspects of music are expected to be perceived
  better than pitch

10
Method

• about 150 cochlear implant receivers in St.
  Petersburg area by 2005
• they are interviewed in order to study their
  post-surgical adaptation to listening music
• interviews are conducted at the Institute of
  Otolaryngology in St. Petersburg as a part of
  general rehabilitation program
• selected interviews are videotaped

11
Training system for rehabilitation program

• The system has been developed in collaboration
  between the Pavlov Institute of Physiology, and
  the Institute of Otolaryngology in St. Petersburg
  where the surgery and post-surgical
  rehabilitation of the patients is carried out
• In practice, the system consists of the laptop,
  two loudspeakers, and a console for patients
  under school age
• The software for the computerized training and
  rehabilitation system includes standardized
  available components as well as original modules
  developed for the tasks to be fulfilled in the
  framework of the project

12
Interview

• Did you listen to speech, music, or noise?
• If this was music, was it singing or playing an
  instrument?
• If this was singing, was there just one performer
  or many of them? Were the singers male or female?
• If this was playing an instrument, was there just
  one instrument or many? String, wind, or
  percussion?
• Was this music joyful or sad? Fast or slow?
  Pleasant or not? Simple or complicated?
• Did this music sound in high, medium or low
  register?

13
Choice of music for the interview

• Four excerpts from world music, with different
  sound characteristics
• Japanese bamboo flute shakuhachi one voice, long
  notes, timing plays little role
• Saami jojk (Mari Boine) female singing, one
  voice, heterogeneous rhythm
• Lithuanian sutartine polyphonic female singing
  (facilitates harmonic dissonances), repetitive
  rhythm
• Tuvan overtone singing (xöömij) a low
  fundamental of special timbre, with higher
  overtones made audible one-by-one

14
Long-term spectra of (1) Japanese bamboo flute
shakuhachi (top left), (2) Saami jojk (top
right), (3) Lithuanian dissonant polyphony
(bottom left), and (4) Tuvan overtone singing
(bottom right)
15
Excerpts from an interview (1)

• - a post-lingual patient
• - has used the implant for 4 years
• - recognizes Lithuanian polyphonic and Tuvan
  overtone singing
• - confuses shakuhachi sound with Saami jojk
• - claims to prefer the jojk to other excerpts

16
Excerpts from an interview (2)

• a post-lingual patient
• has used the implant for a couple of weeks
• describes Tuvan overtone singing as sound of a
  Jewish harp
• describes shakuhachi sound as singing
• confuses Saami jojk with Lithuanian polyphony
• noticeable progress during the interview
• admits using extramusical features (sound volume)
  to solve the task

17
Interview results for 10 CI-recipients (age 10 to
55 years, post-surgical experience from 2 weeks
to 4 years)

• flute jojk sutartine xöömij
• music 10 10 10 7
• singing 2 10 10 6
• female singing 1 7 7 2
• joyful 0 0 8 1
• fast 0 0 8 0
• pleasant 8 8 8 0
• low pitch 2 3 1 8
• medium pitch 3 1 1 4
• high pitch 2 0 6 1

18
Listening preference score for ten CI-recipients
19
General conclusion

• Post-lingual patients with cochlear implants are
  strongly motivated to attend music even when its
  perceived acoustical characteristics only
  remotely resemble the patterns they are familiar
  with from the period preceding their hearing
  loss. This may be explained by an ecologically
  important function music fulfills in peoples
  everyday life.

20
A related study of intonation

• statements and questions may be distinguished
  from each other on the basis of intonation
• statements tend to have falling intonation while
  questions tend to have rising intonation

21
Thanks to

• Elvira I. Stoljarova of the Pavlov Institute of
  Physiology, St. Petersburg, for video recording
  of the interviews
• Wissenschaftskolleg zu Berlin for providing me
  with a three-month Mellon scholarship