Survey Mode Impact Upon Responses and Net Promoter Scores

1
Survey Mode Impact Upon Responses and Net
Promoter Scores

• Frederick C. Van Bennekom
• Northeastern University, Great Brook
• Sam Klaidman
• Middlesex Consulting

2
Summary

• Surveys heavily used gauge of customer
  centricity
• Mixed mode used to increase response rates
• Net Promoter Score (NPS) is common summary
  attitudinal indicator
• However, businesses are unaware of how changes in
  survey practices can affect responses
• Research findings on survey mode
• Telephone survey mode elicits higher scores on
  the response scale than web-form survey mode
• NPSs threshold effect amplify the mode effects

3
Previous Research Non-Measurement Error

• Survey Mode impacts
• Non-measurement error Whether the person
  responds
• Measurement error How the person responds
• Non-measurement error
• Non-response bias
• Telephone mode garners higher response rates
  (Groves 1989, Nunley, 2013)
• Composition bias
• Response rates will differ by demographic groups
  for each mode

4
Previous Research Measurement Error

• Measurement Error
• Response effects are greater with telephone mode
  yielding higher, more positive scores
• Acquiescence yes saying (Bowling 2005)
• Social Desirability (Bowling 2005)
• Primacy Recency (Christian et al. 2007,
  Bethlehem 2012, Kreuter 2008)
• Scale truncation effect
• Our belief is that the telephone presentation of
  the scale with endpoint anchors as the only
  information guides leads to the tendency toward
  extreme responses

5
Previous Reasearch Net Promoter Score

• New statistic applied to the Likelihood to
  Recommend question
• Reichheld (2003), based on Sambandam and Hausser
  (1998)
• Reichhelds argument
• Best summary measure to indicate likelihood of
  future profitability

6
Why the NPS Statistic?

• Its one number
• Rather than report both Top Box and Bottom Box
• People get percentages
• Highly responsive to changes
• Real or artifacts of the survey process
• Provides focus to the low end of response scale
• However, highly controversialdue to lack of
  reproducibility (Morgan and Rego 2008)

7
Research Venue

• Large, anonymous capital goods manufacturer with
  after-sales service
• Transactional survey conducted in both telephone
  web-form modes after service events
• Web-form if email address captured by field
• Telephone if otherwise
• Three question survey
• Likelihood of recommendation NPS question
• Overall satisfaction with XXX as a service
  provider
• Overall satisfaction with the last visit
• Access to December 2011 data

8
From a Research Design PerspectiveGood News, Bad
News

• Bad news
• Not a controlled experiment
• Good news
• Real company data used, not a convenience sample
  of college students

9
Survey Statistics

• Of 133 districts, 48 had no web-form responses
• These districts eliminated due to possibility of
  confounding factor
• Telephone survey scores tested between districts
  with and without email address. p 0.65
• Item non-response led to Unusable Responses

10
Frequency Distribution Recommendation Question
11
Frequency Distribution Chi Square Test Results
Phone 1.96 0.15 0.01 0.83 1.01 2.83
email 21.42 1.59 0.09 9.10 11.06 30.99
Chi square values
12
Additional Testing

• Recommendation question p 5.09 E-16
• Overall satisfaction question p 2.29 E-16
• Visit satisfaction question p 0.0035
• Less difference in scores since high for both
  modes

13
Mode Measurement Error Combined with Composition
Effect on NPS
Survey Method Top Box Promoters (9s10s) Passives (7s8s) Bottom Box Detractors (0s to 6s) Net Score (Promoters Detractors)
Phone 67.8 22.5 9.7 58.1
Web-Form 44.4 29.2 26.4 18.0
Combined Phone Web 65.8 23.1 11.1 54.7
Combined with 10 shift to Web 63.5 23.7 12.8 50.7
14
Implications for Surveying Practices

• Develop mode adjustment factors
• Complicated, expensive, not transparent
• Change survey delivery practices
• Fully anchored 10-point scales not practical for
  phone
• Track modes separately
• Simple, but how to summarize for a district
• Discontinue mixed-mode surveying
• Need to educate consumers of survey data about
  the many errors present, especially when
  comparing data across companies