Towards Improving Spoken Presentation of Lists

1
Towards Improving Spoken Presentation of Lists

• Brian Langner
• Joint Speech Seminar
• 11 November 2005

2
Outline

• Background
• Experiment and Results
• Future directions

3
Problem Background

• Dealing with lists (and other complex
  information) intelligently is challenging for
  speech systems
• There are 23 flights from Chicago to New York
  USAir at 718, Northwest at 744,
• This strategy often will fail miserably at
  conveying any information
• How do people solve this problem?

4
Possible Solutions

• Ask (follow-up) questions
• Limited to interactive applications
• Problem occurs in other areas as well
• Group items
• But, in what way?
• Any bus that comes whose number is prime and
  letter is not C
• Ignore some parts of the list
• What if you leave out the one important thing?

5
Experiment

• Goal identify presentation methods that result
  in improved human understanding
• Subjects asked to listen to 10 sentences of
  diphone synthesis with bus numbers and travel
  locations
• Needed to draw a line on a map showing the
  locations and correctly identify the set of buses

6
Experiment Map
7
Experiment

• Number of buses to identify varied from 2 to 28,
  out of 32 possible choices
• Buses randomly presented as a list, grouped
  together, or larger groups with exceptions
• All subjects heard the same content, in the same
  order, only differed in list presentation

8
Set Presentation Types

• List all
• H11, H12, H21, H25, etc.
• Groups
• All of the 25s
• Large Groups with Exception
• Any of the 40s except the 48

9
Results

• Most people found this task difficult
• 22 subjects
• Average Error Rate 51
• Range from 15 to 93, median 50
• Error Rates by condition
• List 40 Grouped 49 Exceptions 58
• Whats going on here?
• Well, several things

10
Results

• There were two kinds of grouped
• By numbers (the 42s and 48s)
• By letters (the H J and L 25s)
• Grouping by number was better than by letter (45
  to 63 error rate)

11
Results

• There were two kinds of subjects
• Natives (15) / Non-natives (7)
• Natives generally were better than non-natives
  (by 20-30 absolute error rate)
• except for groupings by number (natives
  51 to non-natives 35)
• Natives had 51 error for both grouping types
• List condition still had best results for
  natives!
• 29 vs 51/51/50

12
Other Issues

• Sentences presented a variable number of buses to
  identify
• Missing one out of 10 shows low error compared to
  missing one of 3
• Evaluate based on score rather than error
• Each sentence is worth 1 point if all buses
  identified with no false positives, 0 otherwise

13
Results

• Overall scores (out of 10)
• Average 1.63 (1.67 native, 1.57 non-native)
• Range from 0 to 4, median of 1
• Average score by condition (from 0 to 1)
• List 0.25 Grouped (Number) 0.11
• Grouped (Letter) 0.19 Exceptions 0.14
• Why does just list everything seem better?

14
List everything better?

• These sample sizes are too small to show
  statistical significance
• Especially with the interesting looking
  native/non-native splits
• Better here is still pretty bad anyway
• Anecdotally, people get annoyed and frustrated
  with systems that do just that
• People dont do that

15
But it seems like it works better!

• In specific cases, the simple list everything
  strategy is best
• Short lists
• Lack of logical simple groupings
• But this is not always the case
• Best overall result was with a set of 4 buses
  that were easily grouped next best had 8!
• People who heard the grouped versions generally
  scored better than people hearing the listed
  versions

16
Then what should we do?

• The key seems to have less to do with the style
  of presentation than with the amount of
  information
• Biggest complaints are when a system lists dozens
  of items
• People do better with fewer pieces of info
• Not particularly surprising, given the oft-cited
  7 2 rule
• This data shows that rule is optimistic at best
• So, limit the number of pieces of information

17
How much can we tell a person?

• Not very much
• Ideal subjects
• Still found the task difficult
• Generally struggled with anything more than 2
• Maximum range is probably between 2 and 4 need
  to investigate this properly
• What about non-ideal (ie, general public)?
• Is this domain-dependent?
• Or worse, user-knowledge-dependent?

18
Where do we go from here?

• Find the real limit of how much information you
  can give a person
• Identify and employ strategies to stay below that
  limit
• In interactive settings, follow-up/clarifying
  questions can narrow the list, along with
  grouping
• Non-interactive may try some grouping, ordered
  (limited) presentation
• In many (all?) settings, complex grouping should
  be avoided

19
New Experiment

• Use a dialog system (such as Lets Go) to
  identify the limit of amount of information
  reliably given using simple lists
• Once identified, see what strategies work best to
  provide larger lists of information
• Ask questions, different levels of grouping, etc.
• Try to generalize results to a different
  system/domain

20
Questions?