Question-Answer Sequences in Survey Interviews

1
Question-Answer Sequences in Survey Interviews

• Wil Dijkstra
• Free University
• Amsterdam

Lazarsfeld symposium, June 4-5, 2004
2
De Volkskrant, April 19, 2004 Sixty percent
fears fundamentalists Feels very
threatened 12 Feels a bit threatened 48 Does
not feel threatened at all 38 Dont know 2
Lazarsfeld symposium, June 4-5, 2004
3
Data About 17000 Question-Answer sequences
from 6 surveys, three face-to-face, three
telephone
Lazarsfeld symposium, June 4-5, 2004
4
I Whats your age, if I may ask? R Im nearly
40 I Shall I type 40 then? R Oh, thats OK, it
differs only a few months
Lazarsfeld symposium, June 4-5, 2004
5
I Whats your age, if I may ask? R Im
retired I OK Interviewer types 65
Lazarsfeld symposium, June 4-5, 2004
6
I Whats your age, if I may ask? R My
age? I Yes R Im an elderly man I I already
thought so, great R But Im all there I It
clearly shows, okay Interviewer types 80
Lazarsfeld symposium, June 4-5, 2004
7
I How many days a week do you watch
television? R Uhmm most of the days I Seven
days? R Well, maybe Interviewer types 7
Lazarsfeld symposium, June 4-5, 2004
8
1. strongly disagree, 2. disagree,, 5. strongly
agree
I TV commercials come at the wrong
moments R Uh, well no If I, if I dont like
them, then they always come at the wrong
moments I Yes R Depends on the
moment Interviewer types ?
Lazarsfeld symposium, June 4-5, 2004
9
1. strongly disagree, 2. disagree,, 5. strongly
agree
I TV commercials come at the wrong
moments R Uh, well no If I, if I dont like
them, then they always come at the wrong
moments I Yes R Depends on the
moment Interviewer types 1 strongly disagree
Lazarsfeld symposium, June 4-5, 2004
10
Mismatch answers In the six surveys, the
first answer to a question was a mismatch in
15-31 percent of the sequences
Lazarsfeld symposium, June 4-5, 2004
11
Why mismatch answers? Respondents may
conceive a survey-interview as a common
conversation Mismatch answers are
conversational answers
Lazarsfeld symposium, June 4-5, 2004
12
I And where did you watch television
yesterday? R uhmm well the news I Mmm... and
where do you watch them? R And games and quizzes
and ... I Ok. And where do you watch
them? R Well ehm ... where I watch them? It it
starts at seven oclock, and always the Wheel of
Fortune. I Mmm. And do you look at home, or
somewhere else? R Always at home. I Ok.
(at home, at work, someone elses home,
restaurant/pub)
Lazarsfeld symposium, June 4-5, 2004
13
I I was so sick once I thought I might
die R Huh actually Ive been dead,
haha I Haha R Yes I OK, then I fill in
yes R Yes it happened to me. I didnt knew
anything about it, but it happened I Really? R Y
es, they told me later, after a surgical
operation I OK, yes.
Lazarsfeld symposium, June 4-5, 2004
14
I How many people live in your household,
including yourself? R Household? I Yes R People
who clean up the house? (interviewer explains)
Lazarsfeld symposium, June 4-5, 2004
15
1. strongly disagree, 2. disagree,, 5. strongly
agree
I The government should take care that people
can use smoking-free pubs and restaurants R Uhm
I nearly agree I Strongly disagree, disagree,
neutral R Uhhhh I disagree or strongly
disagree R No, I am I nearly agree, so I uh
nearly not disagree Forgot what you
mentioned I Strongly agree, agree,
neutral R Agree, I agree I OK, agree
Lazarsfeld symposium, June 4-5, 2004
16
Cognitive problems Respondents had cognitive
problems in 6-9 percent of the sequences
comprehension retrieval judgement formatting
Lazarsfeld symposium, June 4-5, 2004
17
I Did you read about the campaign Use your
bike held by Milieudefensie? R Oh yes sure,
that was some years ago I And were did you read
about this campaign? R Well that was in the maga
magazine of Milieudefensie and also in De
Volkskrant, yes. I Oh yes, thats possible
Lazarsfeld symposium, June 4-5, 2004
18
Cognitive mismatches
z 8.8
mismatch
considerations
adequate
z -10.3
Lazarsfeld symposium, June 4-5, 2004
19
Conversational mismatches
z -9.8
mismatch
considerations
adequate
z 12.3
Lazarsfeld symposium, June 4-5, 2004
20
Causes of mismatches Depends of the
formulation of the question
Compare How many days a week do you watch
television? What is the number of days a week
you watch television?
Lazarsfeld symposium, June 4-5, 2004
21
Reaction of interviewer to mismatch repeats
response alternatives 25 repeats
question 6 accepts answer 24 suggestive
probe 20 interprets answer 6 repeats
answer 11 other 8
Lazarsfeld symposium, June 4-5, 2004
22
Problems in the interaction 22 to 42 percent
Lazarsfeld symposium, June 4-5, 2004
23
Recommendations formulate questions
non-conversational formulate alternatives
conversational conduct pilot study train
interviewers to recognize mismatches inform
interviewers about the meaning of the questions
and how this relates to the purpose of the study
Lazarsfeld symposium, June 4-5, 2004
24
Methodology (thanks to AAPOR) Fighting the war
against error!
I How many minutes or hours a day do you watch
television? R Well, I have a full-time job, but
my wife, my wife she she looks about ten hours a
day by now. I Ten? Wow! R Yeah I OK Interviewe
r types 10 hours
Lazarsfeld symposium, June 4-5, 2004
25
I What is your weight? R Oh well, Im pregnant
at the moment Usually it is nine 58
kilo I Yes, but I think we should take the
present situation R No I think we should take
the usual situation
Lazarsfeld symposium, June 4-5, 2004