Using eInterviewers Improves Web Survey Data Quality

1
Using e-Interviewers Improves Web Survey Data
Quality

• by
• Karl G. Feld humanvoice inc.
• Steven A.Wygant Brigham Young University

2

• dyadic from the Greek duadikos. Designating or
  pertaining to a relationship between two
  entities, or a predicate expressing such a
  relationship. Meaning originating in the late
  nineteenth century.
• The Oxford English Dictionary, 1993

3
The Dyadic Relationship Improves Data Quality

• Bourque, L.B. and E. P. Fielder. 1995. How To
  Conduct Self-Administered and Mail Surveys.
  Thousand Oaks, CA Sage Publications, Inc
• Conrad, F.G. and M. F. Schober. 2000. Clarifying
  Question Meaning In A Household Telephone Survey.
  Public Opinion Quarterly. 641-28.
• Krosnick, J. 2000. Peering Into The Future of
  Thinking and Answering A Psychological
  Perspective on Internet Survey Respondents.
  Presentation at the 2000 Gallup Nebraska
  Symposium on Survey Research. 13-15 April.
  Lincoln, Nebraska.
• de Leeuw, E. and M. Collins. 1997. Data
  Collection Methods and Survey Quality An
  Overview. Survey Measurement and Process Quality.
  New York John Wiley Sons, Inc.
• Saris, W. E. 1991. Computer-Assisted
  Interviewing. Newbury Park, CA Sage
  Publications.

4

• computer assisted methods do not provide a
  panacea (or even a general palliative) for survey
  non-coverage, non-response, and measurement
  error. Nor do they free the survey manager and
  survey designer from a need to understand the
  broader survey literature on questionnaire
  design, field work procedures, and process
  quality.
• Nicholls, Baker and Martin, 1997

5
Why Use e-Interviewers?

• 1. Reduce mid-termination or dropout rates
  significantly.
• 2. Increase open-end response length.
• 3. Screen out ineligible and professional
  respondents.
• 4. It is inexpensive.

6
University Alumni Publications Study

• Survey measured attitudes of alumni about
  university publications they receive.
• Split mode methodology was used to collect
  comparative data.
• Self administered online survey
• e-Interviewer administered online survey
• Interviewer administered CATI

7
Technologies Control Data Quality Across Modes

• Integrated CATI and web survey software to
  allowed uniform, simultaneous programming.

• List Rotations
• Complex Skip Patterns
• Forced Responses
• Prevent Order Effects
• Respondent Privacy

8
Technologies Control Data Quality Across Modes

• Internet versions hosted in-house to guarantee
  respondents server access 24/7.
• Servers were tied directly to the Internet
  through a Tier-1 provider.
• Servers were redundant and load-balanced to
  prevent overload during peak periods.

9
(No Transcript)
10
Quotas Based On Winter 2000 U.S. Internet
Population

• Age
• Census Region
• Gender
• Income
• Occupation

11
Sampling From Known Universe

• CATI VERSION
• Entire known universe (141,337 names) used.
• Selected every nth point based on geographic,
  gender and age quotas.

• INTERNET VERSIONS
• Those with e-mail addresses not contacted by
  telephone (11,892 names) used.
• Separated into demographically equivalent
  files, one for each version.
• Every nth name selected based on geographic,
  gender and age quotas.

12
Data Collection Procedures

• CATI
• Interviewer hours evenly divided by gender.
• No interviewer worked two days sequentially.
• Daily quotas by quota area using sample
  proportional to final quotas.
• Dialed sample from 9AM to 8PM after initial
  evening attempt.
• Callbacks were scheduled upon request.

• e-Interviewer Web
• Interviewer hours evenly divided by gender.
• No interviewer worked two days sequentially.
• E-mail invitations with survey URL (and
  reminders) sent out daily in equally-sized
  batches proportional to final quotas.
• Interviewers available 24/7.
• Respondents could exit and re-enter surveys.

13
Data Collection Procedures

• Self-Administered Web
• E-mail invitations with survey URL (and
  reminders) sent out daily in equally-sized
  batches proportional to final quotas.
• Survey available 24/7.
• Pop-Up Exit Survey on Terminations
• Respondents could exit and re-enter surveys.

• e-Interviewer Web
• E-mail invitations with survey URL (and
  reminders) sent out daily in equally-sized
  batches proportional to final quotas.
• Survey available 24/7
• Pop-Up Exit Survey On Terminations
• Respondents could exit and re-enter surveys.
• Interviewer hours evenly divided by gender.
• No interviewer worked two days sequentially.

14
Select Final DispositionsCalculated Using
Council of American Survey Research Organization
(CASRO) and American Association for Public
Opinion (AAPOR) Standards
15
1. Significantly reduce mid-termination or
dropout rates.
16
Mid-Termination Rates Across Modes
17
Comparative Mid-Term Rates
18
Established Causes of Mid-Termination Solved By
Using e-Interviewers

• No rapport with interviewer
• Presence of open-end questions
• Respondent technical difficulties
• No opportunity to request question clarification
  or verification of sponsor
• No option for respondent to change interview mode
  at will

19
Solving Some Knotty Internet Challenges

• Reducing Mid-Terms Decreases Sample Panel Fatigue
  and Self-Selection Bias
• Click to Callback and Instant Messaging Allow
  Resolution of Technical Problems Or Interview
  Mode Change
• Increase by 30 the number of respondents 55
  years who complete surveys

20
2. Increase open-end response length.
21
Findings Consistent With Previous Research

• Fewer terminations on open-end questions when
  interviewer present
• Average word length of open-end responses and
  follow-up probes is longer

22
Average Open-End Word Length
23
3. Screen out ineligible and professional
respondents.
24
The Harsh, Researched Truth

• Frequency of professional respondents in the
  U.S. is on the rise
• Self-administered surveys have always suffered
  from ineligible respondents
• Self-selection rates are high in sample panels
  with high mid-termination rates

25
www.money4surveys.com
26
Ineligible Respondents Caught By Version

• CATI n300 5 ineligibles
• E-Interviewer n117 2 ineligibles
• Self-Administered n122 ?????????
• Security required unique PIN and year of birth.

27
4. It is inexpensive!
28
So What Does It Cost?

• n600
• 50 questions
• Listed, 80 incidence sample
• Re-invite after initial sample send
• Using SurveyGuardian 24/7

CATI 11356 USD
E-Interviewer 7200 USD
Self-Administered 4550 USD
29
Why Use e-Interviewers?

• 1. Reduce mid-termination or dropout rates
  significantly.
• 2. Increase open-end response length.
• 3. Screen out ineligible and professional
  respondents.
• 4. It is inexpensive.

30
Using e-Interviewers Improves Web Survey Data
Quality

• by
• Karl G. Feld humanvoice inc.
• Steven A.Wygant Brigham Young University