Evaluating the Effect of Predicting Oral Reading Miscues

1
Evaluating the Effect of Predicting Oral Reading
Miscues

• Satanjeev Banerjee, Joseph Beck, Jack Mostow
• Project LISTEN (www.cs.cmu.edu/listen)
• Carnegie Mellon University
• Funding NSF IERI

2
Why Predict Miscues?

• Reading Tutor helps children learn to read.
• Speech recognizer listens for miscues (reading
  errors)
• E.g. listen for hat if sentence to be read has
  word hate
• Accurate miscue prediction helps miscue detection.

3
Real Word Substitutions

• Miscues substitutions, omissions, insertions
• Real word substitution misread target word as
  another word
• E.g. read hat instead of hate
• Most miscues are real word substitutions
• ICSLP-02 predicted real word substitutions
• Here evaluate effect on substitution detection

4
How Evaluate Substitution Detection?
substitution
substitution undetected
false alarm
substitution detected

false alarms
False alarm rate

words correctly read
5
Evaluation Data

• Sentences read by 25 children aged 6 to 10

6
Rote Method

• Uses the University of Colorado miscue database.
• For each target word
• Sort substitutions by children who made them.
• Predict that the top n substitutions will
  reoccur, for this word.

7
Extrapolative Method

• Predict the probability that a word is a likely
  substitution for another word
• Pr ( substitution hat target hate)
• Use machine learning to induce a classifier
• Train using University of Colorado miscue
  database.

8
Extrapolative Method contd

• Given a target word, predict substitution if
• Pr ( substitution candidate target word ) gt
  threshold

9
Combining Rote and Extrapolative

• Aim Get n substitutions for a given word.
• Step 1 Use top n substitutions from rote.
• Step 2 If rote predicts k substitutions, k lt n,
• Then add top n k substitutions from
  extrapolative.
• Intuition rote is more accurate, so use when
  available. If not available, fall back on
  extrapolative.

10
Results from Combining Algorithms
Truncation The first 2 to n-2 phonemes of a
word models false starts. /K AE/ and /K AE N/
for /K AE N D IY/ none for hate Theoretical
max use only those miscues the child actually
made.
11
Conclusion

• Evaluated effect on substitution detection of
• Two previously published algorithms
• A combination of the two algorithms.
• Combined approach improved on current
  configuration (truncations) by
• Reducing false alarms by 0.52 abs (12 rel)
• Increasing miscue detection by 1.04 (4.2 rel)
• Take-home sound byte Listening for specific
  reading mistakes can help detect them!