A Scientific Approach to Improving VUI Design

1
(No Transcript)
2
A Scientific Approach to Improving VUI Design
3
Agenda

1. Principles of Gourmet Customer Service
2. Data Collection Toolbox
3. The Project Life-Cycle
4. Conclusion/QA

4
Principles of Gourmet Customer Service

• Nearly every effort to improve service makes at
  least one of these mistakes
• No data is collected from actual callers.
• The data collected does not relate to business
  goals.
• There is not enough data collected to be
  statistically significant.
• There is no context or understanding of the data
  (for example what is a good satisfactions
  score?).
• No action is taken based on the data collected.

5
The Scientific Approach

• Set Goals
• Use relevant business goals, not technical
  metrics
• Gather data
• Multiple kinds of tests may be needed to get a
  complete picture
• Take action
• Validate
• Repeat the tests from Step 2.
• Commit Go back to Step 1.

6
Data Collection Toolbox

• There is no one way to collect data about a VUI
  or call center.
• A VUI designer who only uses one test method is
  like a carpenter who only owns a hammer.

7
(No Transcript)
8
The Toolbox General Tools

• Test Goals
• Statistical Considerations
• Margin of Error
• Probability of Finding Problems
• Prototypes
• Functional Prototype (aka Rapid Prototype)
• Wizard-of-Oz Prototype

9
The Toolbox Idea Generating Tools

• Heuristics
• Best Practices
• Expert Opinions, Audits, Assessments
• Agent Feedback
• Call Recording and Monitoring

10
(No Transcript)
11
The Toolbox Data Generating Tools

• Controlled Testing
• Lab Tests
• Rapid Assessments
• Large Controlled Tests
• Follow-Up Surveys
• End-of-Call Surveys
• Not Recommended due to extreme sample bias

12
(No Transcript)
13
The Toolbox More Data Generating Tools

• Friends Family Employee Tests
• Not Recommended due to unrepresentative sample
• Automated Load Testing
• Traversal Testing
• Call Logs and Stats

14
The Project Life-Cycle

• Every project goes through certain phases.
• Analysis
• Design
• Code
• Test
• Release
• Operational
• Every phase has different testing requirements

15
Analysis Phase

• Lay out the project scope, goals, and budget
• Ask the big questions
• Who are these callers, and how do they live?
• Why are they using the phone and not (e.g.) the
  web?
• What is the competition doing?
• Look at the big picture and big ideas
• Generate ideas from consultants, vendors, agents,
  and customers
• Develop baselines benchmarks

16
Analysis Phase Tools

• Controlled Testing
• Benchmarking, competitive analysis, baseline
  measurements
• Need to understand your callers before you start
• Agent Feedback
• Follow-up Surveys
• Call Recordings
• Heuristics
• Best practices, consultants opinions, etc.

17
Design Phase

• This is when you plan the details of the VUI
• Use a Prototype ? Test ? Prototype ? Test
  iterative design cycle
• The keys to success are speed and flexibility
• Test several designs, sometimes in the same day
• Dont blow the entire testing budget on The
  Usability Test

18
Design Phase Tools

• Prototypes
• WOZ or Functional Prototype
• Controlled testing
• Rapid Assessment or Lab Test
• Lets you use an iterative design process
• Nothing ends arguments faster than solid data.

19
Code Phase

• Implement the design
• Most testing is for debugging
• Unit testing and some functional testing
• Relatively little VUI testing takes place here
• Implementation may require some VUI changes
  these should be tested

20
Test Phase

• Last stage before rolling out the application to
  live callers
• This is where the most thorough testing takes
  place

21
Test Phase Tools

• Traversal test
• Ensures that the implementation matches the
  design
• Large controlled tests
• Designed to catch even small usability glitches
• Automated load testing
• Done last because changes in the first two test
  phases can affect load handling

22
(No Transcript)
23
The Test Phase can also be the Acceptance Phase

• Business goals are what are important, not the
  exact design
• They are what should be in the contract.
• Tests which measure business goals should be the
  acceptance criteria.
• Many of these tests can be used in the design
  phase.

24
Release Phase

• This is the process of rolling out the
  application
• By this time, you should be confident that the
  system works
• Confident doesnt mean cocky
• This is the time for a phased roll-out
• Some call it a pilot test
• This is a deployment strategy, not a testing
  strategy

25
(No Transcript)
26
Operational Phase

• The work doesnt end when the system is live!
• Application Drift
• External changes
• Company
• Caller Lifestyle
• Competition
• This looks a lot like the analysis phase for the
  next project.

27
Operational Phase Tools

• Benchmarking (Controlled Testing)
• Competitive analysis (Controlled Testing)
• Follow-up surveys
• Agent feedback
• Monitoring call stats and logs
• Listening to call recordings

28
Lessons Learned

1. Good service does not have to be more expensive
   than bad service
2. Customers are smarter than machines
3. Every system can stand some improvement
4. The best VUI designs come from paying attention
   to the right things at the right time

29
Q A