Response Rates and Results of the Advance Letter Experiment

1
Response Rates and Results of the Advance Letter
Experiment

• 2004 RRFSS Workshop
• Toronto, June 23, 2004
• David A. Northrup, Renée Elsbett-Koeppen and
  Andrea Noack
• ISR, York University

2
Outline

• general comments on response rates
• how response rates are calculated
• a very brief history of response rates
• what strategies have/are being put in place to
  deal with declining response rates

3
Outline (continued)

• response rates and RRFSS
• what did it take to get the 62 rate for 2003
  RRFSS
• number of calls
• refusal conversions
• results of the advance letter experiment

4
Calculating Response Rates

• Completions / estimate of number of eligible
  households (HH)
• eligible HHs include completions, refusals,
  callbacks, and a of the never answered
• ISR method same as BRFSS,
• aka CASRO 3
• RRFSS 2003 62, exclude callbacks 71

5
Response Rates for American Election Study
6
Response Rates for BRFSS
7
Strategies for Improving Response Rates

• interviewer training
• increase call attempts
• convert refusals
• use advance letters
• payments (as a lottery, to completers, to the
  whole sample)

8
Data Collection at ISR for RRFSS Response Rates

• minimum number of 14 calls (more when there is
  reason to think extra calls might obtain a
  completion)
• limitation of one month sample release
• costs about 3 to 7 points on response rate
• at least one attempt to convert almost all
  refusals

9

• 390,106

10
RRFSS Fun with Numbers 1 (2003 Data)

• number of calls 390,106
• percent of interviews completed first call 21
• number of interviews completed on the 10th or
  subsequent calls 3,158
• number of interviews completed after a refusal
  2,678

11
RRFSS Fun with Numbers 2 (2003 Data)

• average number of calls per completed interview
  4.65
• most calls made for a single completion 33 (for
  two (different) interviews)
• response rate if 10 plus calls and refusal
  conversions are dropped 48.2
• number of complaints about interviewer calling
  registered at ISR 13

12
Characteristics of Refusers2003 RRFSS Data
variable standard converted
mean age 47.99 53.86
education gt than high school () university () 16.7 47.8 23.0 39.2
employed 61.7 52.4
saying health fair or poor 11.8 16.5
doctor told high blood pressure 22.3 27.1
smoke 100 cigarettes 52.5 55.0
of cases standard 24,700, converted
2,640 all differences significant
13
Characteristics of Easy and Hard to Reach 2003
RRFSS Data
variable easy to reach hard to reach
mean age 50.43 43.50
education gt than high school () university () 18.6 45.6 12.5 52.8
employed 54.8 73.6
saying health fair or poor 13.8 8.1
doctor told high blood pressure 25.0 16.8
smoke 100 cigarettes 52.7 50.6
of cases easy 17,000, hard 3,150 all
differences significant
14
Letter Experiment 1

• six Health Units participated (Durham, London,
  Grey Bruce, Halton, Waterloo, Sudbury)
• test two versions of letter ISR and HU
• needed to work with our monthly target and wanted
  to acknowledge random variation in response rates
  per HU per month
• used sample replicates to implement experiment

15
Letter Experiment 2

• Month one replicate 1, ISR letter replicate 2,
  HU letter replicate 3, and 4 (when used),
  control group
• changed presentation in months 2 and 3
• copy of letter at the end of this set of handouts
• exactly the same text, different letterhead,
  signature envelope
• Except Halton

16
Letter Experiment 3

• survey introduction exactly the same except one
  additional sentence
• Recently, we sent a letter to your household
  about an important research project.
• questions about the letter the same
• except Durham

17
Why the Letter Might Improve Response Rates to
RDD Surveys

• reduces the possibility that the telephone call
  catches people by surprise
• increases legitimacy of research project in the
  eye of the potential respondents
• demonstrates social value
• improves the confidence of the interviewer

18
Why Advance Letters Might Not Improve Response
Rates to RDD Surveys

• letter does not reach, or is not read by,
  respondent
• ceiling effects
• survey topic subpopulations
• they give timid participants a chance to
  prepare to say no

19
Response Rates for Months 1 2 of the
Experiment
p value .035 (for letter (1,200) versus no
letter (1,345)) p value .025 (ISR (600) versus
HU (600))
20
Response Rates Months 1 2 All Six Health
Units
See next slide for numbers p values
21
RR by HU and Treatment
treatment, RR () treatment, RR () treatment, RR () P value P value P value
HU none ISR HU none/ISR none/ HU ISR/ HU
Durham 58.9 63.6 65.3 .422 .275 .803
London 57.8 64.5 69.2 .254 .048 .475
Grey 70.5 73.2 78.7 .618 .128 .367
Halton 57.5 57.0 68.7 .925 .053 .087
Waterloo 64.7 60.2 65.8 .441 .851 .416
Sudbury 69.8 63.2 71.4 .255 .776 .213
Number of cases per HU ISR 50, HU 50, none
100
22
Mean Calls per Completion
mean of calls mean of calls mean of calls P value P value P value
None ISR HU none/ISR none/ HU ISR/ HU
Durham 5.89 5.69 5.57 .825 .721 .905
London 6.41 6.06 4.43 .751 .049 .060
Grey 4.36 4.58 4.69 .763 .622 .896
Halton 5.18 6.42 6.53 .157 .124 .935
Waterloo 4.82 6.42 4.38 .049 .510 .047
Sudbury 5.95 5.14 5.42 .445 .597 .809
Number of cases per HU ISR 50, HU 50, none
100
23
Mean Calls per Completion by Letter Status
Letter/no letter p .890, ISR/HU p .230, not
see/saw p .001
24
of First Call Attempts Leading to Completions
Refusals
letter/no letter p .204, ISR/HU p .008
25
At the Start of the Interview
26
Awareness of Letter
Variable (based on 602 cases) total
R indicated saw letter at intro 30
R indicated letter came to house 21
total respondents aware of letter 51
personally read the letter 40
got more info (web site, 1-800) 1
letter made a lot of difference to decision to participate 26
27
Data Characteristics
Variable ISR (n300) HU (n300) none (n620) P
year of birth 1955 1954 1954 .820
male () 58 56 57 .800
employed () 59 59 60 .627
health excellent 22 22 22 .951
smoked at least 100 cigarettes () 50 56 56 .197
28
Costs Month One

• cost of materials 314 staff cost 1,919
• total 2,233
• per case cost 3.62
• buys 72 interviews or 12 per HU
• need to estimate savings from making fewer calls,
  and making fewer refusal conversion calls

29
Conclusions

• HU letter (seems to) increase response and
  warrants consideration as a tool to improve RRFSS
  response rates
• affect on variable distributions minimal, but
  small sample size limits scope of examination
• social-political distance between respondent and
  sender probably matters
• letters may have value other than just increasing
  response rates

30

• Questions