Results and Discussion

1
Effect of Pay Structure on Interviewer
Productivity Michelle Cantave (REDA
International, Inc JPSM), Frauke Kreuter (JPSM,
University of Maryland), Elham-Eid Alldredge
(REDA International, Inc)

• Study Background
• JPSM 2008 Practicum Survey
• Primary purpose of the study was to see if there
  was an effect of filter question format (grouped
  vs. interleafed) on respondent answers.
• Additional experiment to test if interviewer pay
  structure had an effect on productivity and the
  quality of the data collected.
• Telephone survey conducted for the University of
  Maryland, College Park JPSM Practicum class.
• Data collected by REDA International, Inc. from
  August 2008 to September 2008. Sample released in
  replicates.
• N 1,745 825 in grouped and 827 in interleafed
  condition. Test for successful randomization on
  frame variables. AAPOR Response Rate (4) 14.
• Acknowledgements Stanley Presser, Roger
  Tourangeau, Katharine Abraham, Susan McCulloch

• Experimental Design
• Questionnaire Format Respondents were randomly
  assigned to a Grouped or Interleafed
  questionnaire format
• Interviewer pay structure
• All interviewers were paid by the hour at the
  beginning of the study.
• At the time of release of the 2nd replicate, a
  random subset of interviewers (n20) were
  switched a paid-by-case payment scheme. Remaining
  interviewers (n16) continued with normal hourly
  payment.

• Measures Examined
• Productivity
• Dialings Per Log-on Hour The number of telephone
  calls the interviewer made on average for every
  hour he or she was logged on. Interviewer is
  considered productive if the rate is high.
• Completes Per Log-on Hour The average number of
  completes the interviewer get for every hour
  logged on to the system. Interviewer is
  considered productive the higher this rate is.
• Dialings Per Complete This rate is a measure of
  how good the interviewer is at gaining the
  cooperation of respondents and how difficult the
  respondents are. The lower this rate, the more
  successful the interviewer in gaining
  cooperation.

• Consequences for Data Quality
• Interviewers in pay-by-case scheme have an
  incentive to shorten the interview. Thus
  interviews conducted in this payment scheme
  might
• Take on average less time
• Show fewer affirmative responses to filter
  questions

Results and Discussion

• Results
• Effect on productivity indicators
• No difference in average number of dialings
  before and after the switch in interviewer pay
  structure.
• Increase in the average completes per log-on hour
  after the change in interviewer pay structure
  (plt0.05).
• Decrease in the average dialings per complete
  after the change in interviewer pay structure
  (plt0.05).
• Data Quality
• No significant effect on interview length (569
  interviews conducted in pay-by-case payment
  scheme took on average 18 minutes 95 interviews
  conducted in hourly payment condition took on
  average 20 minutes).
• There was not a significant different in the
  average number of affirmative answers to filter
  questions when comparing individual interviewers
  results before and after the switch in payment,
  nor when comparing two payment conditions for the
  second replicate sample.

• Summary and Discussion
• The switch in interviewer pay structure had a
  positive effect on interviewer productivity.
• Results are preliminary. Further controls for
  experience throughout the surveys needed.
• It appears that paying interviewers by the
  completed interview, as opposed to by the hour,
  does not reduce the frequency of affirmative
  answers to filter questions.
• We compared hourly payment to case payment with a
  floor (that is interviewers were told they could
  earn more in the per-case condition but not
  less). The effect on data quality in a pure
  by-the-case comparison scheme might be different.

• The effect of compensation arrangement for
  decentralized in-person surveys (where monitoring
  is unusual) might differ from that in centralized
  telephone surveys (where monitoring is routine).

REDA International, Inc. Quality
research guiding policy and change Since
1992 www.redainternational.com