To Example or Not To Example, That is the Natural Language Question

1
To Example or Not To Example, That is the
Natural Language Question

• Frederick Parkinson, PhD, Project Manager, User
  Interface Designer
• Deborah Rapsinski, Senior User Interface Designer

2
Introduction

• This paper presents best practices around the
  prompting of natural language recognition states
• Structure of prompt
• Use of examples

3
What we will talk about

• Why we need best practices around Natural
  Language (NL)?
• What makes designing for NL challenging?
• What makes for a good NL response?
• The Nuance best practice for NL
• Evidence to support this strategy

4
Why we need best practices for NL

• NL deployments are becoming more common
• NL deployments incur additional cost, so the
  demand for quick ROI is more acute
• Best practices allow adopters to achieve ROI
  goals more quickly by allowing designers to
  leverage lessons learned from previous
  deployments

5
Designing for NL is challenging

• Directed Dialog versus Natural Language
• Why NL is appealing
• The Natural Language Paradox
• Natural Language states are unnatural
• While callers are encouraged to respond
  naturally, the fact that they are interacting
  with a computer makes the interaction unnatural

6
How Natural Language systems work

• Two tasks
• recognition of words in the utterance
• interpreting the meaning of those words
• How Natural Language systems are developed
• trained against corpora of tens of thousands of
  utterances
• training data is tagged by hand by humans
• To work well, the recognizer has to pick out a
  few salient words that map to specific caller
  intentions

7
What makes a good Natural Language response

• A few meaningful words
• four to ten words long
• one or two meaningful words
• Good examples of responses
• I need to locate a pharmacy, I want to order a
  refill,my power is out, close account
• Bad examples of responses
• I need help, inquiry,I need to check about
  amounts of several drugs that I already have
  when it's time to order new ones I don't want to
  order more than I can use

8
The Nuance best practice

• Provide the caller with examples
• pro addresses the deer in the headlights some
  callers face
• con humans dont offer examples
• Examples provide just enough structure to help
  callers formulate their responses
• In a few words, please tell me why youre
  calling today. You can say things like I need
  my account balance, or whats the status of my
  order? Okay, go ahead.

9
What makes a good example?

• Shows callers how to respond 1 statement, 1
  question

Succinct and Unambiguous Fewest number of turns
to destination
Makes use of caller profile If past
due Id like to make payment
arrangements.
10
Best Practice, Continued

• Rotate examples to teach callers of what can be
  handled
• Offer 2 examples at the initial prompt
• Reduce to 1 for re-prompting or looping through
  NL state
• Balance caller goals with business goals
• Use examples that will, e.g. encourage callers to
  pay their bill or inform them of new product
  offerings

11
Evidence

• Usability Testing
• Inferred success through completed tasks
• Post-test interviews
• Experiments
• No example, 2 statements, 2 questions, 1
  statement and 1 question
• Deployment Data
• Fluency, delays
• Number of words
• Value of responses

12
Evidence Number of Words Used
No Example
Example Used
13
Evidence Fluency
No Example
Example Used
14
Conclusion

• Weve presented the Nuance best practice for the
  structure of the Natural Language question
• Include examples
• Examples include one statement, one question
• Examples balance callers goals and business
  goals
• Weve presented evidence in support of this best
  practice that shows that callers prefer hearing
  examples and that examples improve the quality of
  callers responses to the Natural Language
  question