Visual Aid For the Hearing Impaired

1
Visual Aid For the Hearing Impaired

• By Jason Vieira, Katherine Andrade, Carlos
  Castillo, Frank Taranto, Jr.

2
Purpose and Target Customers

• Lip Information Complemented with Speech improves
  intelligibility in environments with low
  signal-to-noise ratio (SNR)
• Applications in telephone conversations, computer
  interfacing, and television
• Lip information associated with speech assists
  hearing impaired

3
Procedure

• Record all 42 English phonemes among several
  subjects
• Photograph different subjects as they pronounce
  all the phonemes and measure lip shape parameters
• Fitting parabolas to approximate phoneme lip
  shapes and animate coherent speech
• Using COLEA toolbox to obtain LPC coefficients
• Utilizing neural networks to best associate
  coefficients to respective lip shape

4
Design Changes

• VRML animation scrapped because of softwares
  lack of versatility and speech synchronization
• LPC coefficients preferred over cepstral
  coefficients because of computational savings and
  smaller scope of our design
• Neural Network Application over Lip Shape
  Interpolation

5
Sample Audio

• Short a, /a/, as in flat
• Long E, /E/, as in me
• m sound, /m/, as in my
• kw sound, /kw/, as in quick
• ks sound, /ks/, as in box or exam
• z sound, /z/, as in zoo

6
Phoneme Acquisition
7
Measuring Lip Parameters

• Measure inner width
• Measure inner height of upper lip
• Measure inner height of lower lip
• Measure outer width
• Measure outer height of upper lip
• Measure outer height of lower lip

8
Using Parabolas To Simulate Lip Shapes
9
Theory of Lip Animation

• We begin with a parabola, because its properties
  resemble a lip
• By varying d1 and d2 we obtain various
  configurations of a parabola y ax2c
• 4 parabolas are used to apply 2-D effect

D1
d2
Fig. 1
Fig. 2
Fig. 3
10
Long a, /A/, as in Fonzies Greeting
11
Other Phonemes
Short a, /a/, as in flat
b sound, /b/, as in ball
12
Important Compilation Notes

• The sequencing of lip movements should appear
  continuous (approx. 30 fps)
• Voice signal and video present integrated,
  enchancing info and should be in sync
• Audio and visual info can independently render a
  comprehensible phonemes, but are synergistic only
  when synchronized

13
Organs of Speech and Linear Predictive Coding
(LPC)

• A technique for modeling the vocal tract
• Ideal for pitch and formant detection determines
  area functions of vocal tract
• Two types of methods for LPC analysis
• Autocorrelation method
• Covariance method

14
Extracting LPC coefficients

• Linear Predictive Coding is a means to compress
  a continuous signal
• LPC coefficients are derived from previous values
  
• a0s(j)a1s(j-1) .... ans(j-n)
• aCoefficients
• s(j)Present sample
• Procedure is repeated over set of n samples
• Optimum number of coefficients between 10 and 20

15
Other Considerations

• In contrast to LPC analysis, one can obtain the
  cepstrum of a signal which does a better job of
  capturing formants (mathematical manipulation)
• Apply a Hamming window (low-pass filter) to the
  sound signal to increase SNR

Cepstrum DFTlog( DFTsignal )
16
Using Neural Networks

• Neural Networks complement the measurement of LPC
  coefficients
• Operates by supplying network with training set
  of data (LPC inputs and parabolic coefficient
  outputs) and performing least squares for varying
  sets
• Robust method for determining lips shapes for
  people with different pitches and vocal tract
  elasticity

17
Future Progress

• Obtain database of LPC coefficients for all 42
  phonemes
• Apply Neural Network Tools for Signals from
  different subjects to develop smart lip drawing
  software
• Add more aesthetic features to lips to convey
  further communication information

18
Questions ?