About ORC Macro

1

CELL PHONES AND PUBLIC-SECTOR SURVEY RESEARCH
Are Incentives Really Needed? National AAPOR
Conference Anaheim, CA May 18th, 2007
James Dayton, MBA Daniel Gundersen, M.A
Cristine Delnevo, PhD, Zi Zhang, MD, MPH Lori
Westphal, PhD, MPH Michelle L. Cook, MPH Susan
J Cummings, BSN, CPHQ Diane Aye, MPH, PhD
Randal ZuWallack, MS Naomi Freedner, MPH Ruth
Bernstein, BA
2
Cell Phone Surveys

• The potential to measure the cell phone
  population, particularly cell-phone only, is
  still relatively unknown.
• There are advantages to interviewing respondents
  on cell phones.
• Reaching a new generationmany adults,
  particularly younger adults, are opting to use
  cell phones as their primary communication medium
• As a personal device, cell phones provide a
  direct communication link to respondents at
  virtually any time of day.

3
Barriers

• Legislative and ethical
• Laws prohibiting the use of predictive dialing
  means that all numbers must be manually dialed
• Labor intensive and increased likelihood of
  dialing errors
• Incoming calls to cells often eat-up minutes and
  can result in real financial costs to
  participants
• Sense that cell phones are private
• Usage, privacy and safety
• Unavailable for a survey (driving?, working?,
  public places?)
• Respondents are not necessarily located where
  their area code and exchange suggest they are
• Inbound calls to cells are often use plan minutes
  or have a definable cost to respondents
• Increasing risk of backlash as volume of
  unsolicited calls increases
• Sampling methods are relatively crude in
  comparison to contemporary RDD practices
• List assisted methods are not applicable since
  there is no directory of cell phone numbers
• Resort to traditional equal probability samples
  of cell phone numbers
• With restrictions prohibiting automated dialing
  of cell phone numbers, sampling efficiencies are
  all the more important.

4
Study Specifics

• Participating States
• Massachusetts, Connecticut, New Jersey, Florida,
  Montana, Texas
• Questionnaire
• Shortened BRFSS core questionnaire that had been
  piloted in other states
• Survey introduction included IRB-approved
  informed consent, security/privacy, and a safety
  provision
• Average interview length - 10 minutes
• All records must be manually dialed
• Incentive / compensation offered in Massachusetts
  only
• 15 in compensation to all contacts.
• No offer of compensation in other states
• Sample Design
• Purchase randomly generated cell numbers based on
  assigned cell exchanges and 1,000 blocks from
  Genesys

5
Methodology

• Methodology
• Fielding dates Late October 2006 February 2007
• Target calling times between 900 am and 900 pm
  on weekends and between 700 and 900 pm on
  weeknights.
• Generally, only requested call-backs were be made
  between 900 am and 700 pm weeknights.
• Minimum of 6 attempts on non-terminal records
• Two weeknight, four weekend attempts over at
  least a sixteen day calling period
• Leave message on Voicemail on 1st and 5th
  attempts
• Must be an adult to complete survey
• No household randomization assumed cell phone
  was not shared
• Completed interviews with both cell only
  households and cell/landline households
• Single refusal policy

6
Completed Interviews

• Original completed interview target per state
  up to 500
• Actual completed interviews
• Connecticut 352
• Florida 320
• Massachusetts 302
• Montana 389
• New Jersey 300
• Texas 298

7
Incentives

• Identification of control
• We elected to compare MA to NJ
• Both states demonstrate similar response
  characteristics in traditional BRFSS surveys, and
• MA and NJ had a similar ratio of refusals before
  or during the introduction relative to the number
  of respondents who continue past the intro
• 2.75 in MA vs. a 2.65 ratio in NJ.
• Conclusion with limited evidence incentive did
  not impact participation.
• The completion rates are 41.7 and 40.8 for MA
  and NJ, respectively.
• 60 out of the 302 respondents completed the
  survey, but did not leave a name or address for
  receipt of the incentive.

8
Sample Usage
9
Interviewing Costs
Assumes 20 Minute average interview length for
BRFSS
10
Cell Phone Only?
11
State-Level Sample Targeting
12
Safety Concerns?
13
Will the Boss be Impressed?
14
Discussion

• Are incentives needed?
• The limited scope of the research suggested they
  are not yet
• What if incentives are not offered?
• Ethical concerns
• IRB concerns
• Will the impact of incentives change as the
  number of unsolicited calls to cell phones
  increase?
• If incentives are offered
• Practical means of transferring incentive that
  does not require the respondent to reveal a great
  deal of personal information
• PayPal???
• Respondents not terribly concerned about safety
  do we need to be more concerned for them?
• Laws prohibiting use of auto-dialer adds
  substantially to costs
• Even with incentives, cell surveys might be the
  most cost-effective way to increase response
  rates among young adults, especially males.

15
Contact Information

• James Dayton
• James.J.Dayton_at_orcmacro.com