Title: Responsive%20Design%20for%20Household%20Surveys:%20Illustration%20of%20Management%20Interventions%20Based%20on%20Survey%20Paradata
1Responsive Design for Household Surveys
Illustration of Management Interventions Based
on Survey Paradata
- Robert M. Groves, Emilia Peytcheva, Nicole
Kirgis, James Wagner, William Axinn, University
of Michigan, USAWilliam Mosher, US National
Center for Health Statistics
Research partially supported by contract with the
US National Center for Health Statistics,
Contract No. 200-2000-07001
2Definition Responsive Design
Survey designs that
- Preidentify a set of alternative features
potentially affecting costs and errors of
statistics - Identify a set of indicators of the cost and
error properties of those feature - Monitor indicators in initial stages of data
collection - Alter the active features of the survey based on
cost/error tradeoff decision rules - Combine data from separate phases into a single
estimator
3The Whys of Responsive Designs
- One-off surveys are mounted with large
uncertainties (e.g., eligibility of frame
elements, effort required to contact, cooperation
rate, length of interview) - Most survey budgets are relatively fixed at start
of project - Some survey errors are functions of effort during
production - Hence, quality is out of control of researcher
unless designs are permitted to change based on
production experience
4The NSFG Dashboard
Effort
Active Sample
Productivity
interviews
occupied
Irs working
hours
eligible
cum. interviews
production
nonworked
hours/interview
X
calls/day
calls/interview
noncontacts
mean calls
calls/hour
peak calls
8 calls
Data Set Balance
scrnr/main calls
locked bldgs
resistant
response rate
hard appt.
with kids
propensity
sexually active
group rates
CV group rates
5Evaluation of Two Interventions Based on Paradata
- Survey setting face-to-face survey screening
(3-5 min.) to locate 60 of households with 15-44
year old one eligible sampled (60-80 min.) - Interventions
- Increasing relative effort on screening
interviews versus main interviews with selected
respondent - Increasing relative effort on a small subset of
cases with high selection weights and high
propensities to respond
6Ratio of Screener Calls to Main Interview Calls
by Day by Quarter
7Autoregressive Time Series Coefficients for Model
Predicting Daily Number of Screener
Calls(p-values for coefficients)
Q1 Q2 Q3 Q4
Intercept 279.9 (lt.0001) 385.0 (lt.0001) 345.3 (lt.0001) 294.8 (lt.0001)
Day -0.73 (.48) -3.2 (0.28) -2.1 (.21) -2.0 (.17)
Screener Week 28.3 (.29) 97.5 (.009) 24.6 (.56) 39.0 (.29)
8Autoregressive Time Series Coefficients for Model
Predicting Daily Number of Screener
Interviews(p-values for coefficients)
Q1 Q2 Q3 Q4
Intercept 63.6 (lt.0001) 94.5 (lt.0001) 64.6 (lt.0001) 63.4 (lt.0001)
Day -0.51 (.032) -1.23 (0.001) -0.54 (.16) -0.69 (.09)
Screener Week 6.1 (.30) 18.5 (.01) 12.3 (.20) 6.1 (.54)
9Second Intervention Increased Emphasis on Subset
of Active Cases
- In the last weeks of Phase 1, a subsample of
cases with high propensities and high selection
weights are identified - These cases are chosen to improve balance of
respondent pool - Interviewers are asked to give greater emphasis
to these cases
10Mean Expected Probability to be Interviewed on
Next Call, Screeners (Red) and Main (Green) by
Day of Data Collection by Quarter
11Analytic Approach
- Not all interviewers active workloads contain
both intervention cases and non-intervention
cases - We limit the analysis to those who have both
types of cases - We examine two indicators of success
- Mean number of calls (imperfect)
- Response rate in intervention period
12Comparison of Mean Screener Calls During
Intervention Period for Intervention and
Nonintervention Cases
Little evidence of increased calling on
intervention cases
13Comparison of Screener Response Rate During
Intervention Period for Intervention and
Nonintervention Cases
Little evidence of increased response rate on
intervention cases
14Comparison of Mean Main Calls During Intervention
Period for Intervention and Nonintervention Cases
Mixed evidence on higher calls on intervention
cases
15Comparison of Main Response Rate During
Intervention Period for Intervention and
Nonintervention Cases
General tendency to higher response rates for
intervention cases
16Logistic Regression For Likelihood of Main
Interview
Quarter 2 Quarter 3 Quarter 4 Quarter 5
Intercept -2.13 -1.97 -0.81 -0.84
Intervention -0.07 0.72 0.13 0.50
Weight 0.00 0.00 0.00 0.00
Propensity 5.46 2.98 2.02 0.65
17Conclusions
- Intervention 1 Management direction to focus on
screeners vs. main increases calls, sometimes
dramatically screener interviews follow - Intervention 2 Effectiveness at focusing on
individual cases greater for main interviews than
screener interviews
18Next Steps on Responsive Design with Paradata
- Responsive design requires effective central
management direction of interviewer behavior - Were still learning how to communicate these
directives consistently well