Asking questions

1
Asking questions

• HON 101

2
Smoking and praying

• A priest and a nun share a tobacco habit, and
  they are wondering about the compatibility of
  smoking and praying..

3
Small changes in wording

• The importance of exact wording on questions
  seems obvious and hardly worth dwelling on. The
  fact that seemingly small changes in wording can
  cause large differences in responses has been
  well known to survey practitioners since the
  early days of surveys. Yet, typically, the
  formulation of the questionnaire is thought to be
  the easiest part of the design of surveysso
  that, all too often, little effort is expended on
  it
• Sudman, Seymour and Bradburn, Norman M.
• (1982) Asking Questions A Practical Guide to
  Questionnaire Design. Jossey Bass, pp. 1

4
Effects on answers

• Wording
• Ordering
• Interviewer
• Context

5
Types of questions

• Close-ended questions
• Open-ended questions
• Check questions
• Probing questions

6
Closed questions

• Definition
• Respondent must choose from a set of possible
  answers that are pre-determined by the researcher
• Yes/no/dont know
• Multiple choice
• Strongly agree/agree/neutral/disagree/strongly
  disagree (Likert Scale)
• Pro
• Makes recording and coding easier
• Helps avoid ambiguity
• Con
• Researcher must anticipate
• all alternatives (Other option)
• Encourages instant answers
• Categories may lead respondents

7
Open-ended questions

• Definition
• No restrictions on response
• What? What? How?
• Pros
• Richer data
• Maintains interest
• Cons
• Less efficient
• Makes coding harder
• Incomplete answers

8
Closed versus open

• Famous study
• Howard Schuman Stanley Presser (1979) The Open
  and Closed Question. American Sociological
  Review, V.44, No.5, pp. 692-712.
• Test of standard Gallup Opinion Poll question in
  closed and open formats

9
Closed format

• Which of the following is the most important
  problem facing the country today?
• 1. Food and Energy Shortages 6.0
• 2. Crime and Violence 34.9
• 3. Inflation 12.6
• 4. Unemployment 19.7
• 5. Decreased Trust in Government 9.9
• 6 .Busing 1.1
• 7. Breakdown of Morals and Religion 9.2
• 8. Racial problems 1.6
• 9. Other 1.8
• 10. Dont know 0.2
• 11. Not applicable 3.0

10
Open format

• What is the most important problem facing the
  country today? (Gallup)
• 1. Food and Energy Shortages 1.7
• 2. Crime 15.7
• 3. Inflation 13.3
• 4. Unemployment 19.1
• 5. Decreased Trust in Government 3.0
• 6. Busing 1.1
• 7. Breakdown of Morals and Religion 5.7
• 8. Racial Problems 2.4
• 9. Other 3.0
• 10. Dont know 1.1
• 11. Not applicable 1.3

11
Closed versus open

• 12. Quality of Leaders 7.0
• 13. Characteristics of People 4.6
• 14. Characteristics of the System 3.0
• 15. Supportive References to WeIfare 1.5
• 16. Unsupportive References to WeIfare 0.9
• 17. National Defense 0.4
• 18. Foreign Affairs 0.9
• 19. The Elections 4.6
• 20. Communism 0.7
• 21. The Economy, Money Problems 3.7
• 22. More than one codable response 5.4

12
Wording rules of thumb

• Use laypersons language
• Keep it short
• Avoid apologetic wording
• Avoid hypothetical situations
• and intentions
• Avoid ambiguity
• Avoid double-barreled questions
• Avoid double negatives
• Avoid loaded words and phrases
• Avoid leading questions

13
Use laypersons language

• Keep it simple!
• Dont ask
• Do you think that TV news programs are impartial?
• Less than 50 interpreted meaning of impartial
  correctly! (Foddy 1993 41)
• What proportion of your working day do you spend
  traveling to and from work?
• Involves math!

14
Keep it short

• Avoid dangling alternatives
• Avoid long lists
• Avoid multiple clauses
• Place qualifying clauses early
• Dont ask
• Do you prefer supermarkets or farmers markets
  when purchasing household needs, excluding
  produce?
• When purchasing household needs, excluding
  produce, do you prefer supermarkets or farmers
  markets?

15
Avoid apologetic wording

• Be direct
• Dont invite refusal or put respondent on guard

16
Avoid hypothetical situations and intentions

• Behavioral intent is not reliable indicator of
  future behavior
• In transport studies intention to use public
  transit is typically 3-4 actual use

17
Avoid ambiguity

• Dont confuse causality and motivation
• When did you last see your doctor?
• Define terms
• Frequency
• Always, often, frequently, sometimes, hardly
  ever, rarely, seldom, never
• Distance
• Near and far
• Collective nouns
• Household, family
• You
• Where do you shop for clothes?

18
Avoid double-barreled questions

• Dont ask
• Do you agree that traffic congestion and parking
  are the most important transportation problems in
  HI?

19
Avoid double negatives

• Dont ask
• Do you agree or disagree that mental health
  facilities should not be located in residential
  neighborhoods?
• Do you think the government should prohibit the
  following kinds of people from teaching in public
  high schools
• Communists
• Gay men
• Sex offenders

20
Avoid loaded words and phrases

• Avoid value-laden terms
• Welfare v. assistance to
• the poor
• Use neutral wording
• Avoid authoritarian negatives!
• Allow v. Forbid
• Do you think the US should allow public speeches
  against democracy? (No 62)
• Do you think the US should forbid public speeches
  against democracy? (Yes 46)
• Roper Public Opinion Research Center 1940

21
Avoid leading questions

• Unbalanced questions may lead respondents to
  think there is a favored answer
• Dont ask
• Do you agree that we need a mass transit system
  in order to reduce congestion on HI?
• Provide alternative answers

22
Interviews are social interactions

• Minimization of personal effort
• and interpersonal disagreement
• Respondents typically do their best to answer
  every question put to themeven questions they
  have difficulty understanding or relating
  toeither by cognitively adjusting the question
  so they can answer it or falling back on
  contextual clues and general attitudes to
  formulate an appropriate answer (Foddy 1993 101)

23
Some biases

• Influence respondent to select one answer in
  preference to others
• Prestige bias
• Agreement bias
• Ignorance bias
• Consistency bias
• Recall Bias
• Ordering bias
• Rapport effect

24
Prestige bias

• Effect of impression management
• Respondents ally themselves with socially
  appropriate response
• Marked under-reporting of socially desirable
  attitudes and behaviors (e.g. voting)
• Estimates of alcohol consumption increase by
  about 1/3 when questions are answered anonymously
  by computer rather than confidentially in person
• 76 of Americans claim to exercise regularly but
  59 are obese!
• Dont ask
• Do you agree or disagree with the Supreme Courts
  ruling that.?

25
Disowning projection

• Respondents reluctant to admit their own values
  and behavior but will project it upon like others
• Do you agree or disagree
• that students sometimes
• commit plagiarism?

26
Agreement (aquiesence) bias

• Others things being equal, respondents want to
  agree with the question and/or interviewer
• Note effect of education
• Note effect of culture

• Humans in general falsely agree from time to
  time, members of certain ethnic groupsin the
  name of deference, hospitality or some other
  cultural normbehave this way more frequently
  (Jayeline 1999 25)

27
Acquiesence bias in Kazakhstan

• Two questions on survey of public opinion on
  government policy
• The government should stop subsidies to prices
  (Agree 55.7 Disagree 33.9 DK 11.1)
• The government should stop subsidies to prices
  (Agree 44.9 Disagree 44.2 DK 10.8)
• Note the swing in responses
• Stop subsidies from 55.7 to 44.2
• Continue subsidies from 33.9 to 44.9
• Balanced forced-choice question
• Do you think the government should stop or
  continue subsidies to prices (Stop 45.7
  Continue 34.7 DK 9.4)
• Significant difference between Kazakhs and
  Russians
• Javeline, Debra (1999) Response Effects in Polite
  Cultures A Test of Acquiescence in Kazakhstan.
  Public Opinion Quarterly 63,1 pp. 1-28.

28
Ignorance bias

• Respondents reluctant to
• admit ignorance and may
• guess to save face
• 25 of respondents answer the question but will
  choose Dont know option if it is available
• Use a filter question to check for opinion
• Have you thought about this issue?
• Do you have an opinion on this matter?

29
Consistency bias

• Respondents will often try to make responses
  consistent between questions and over time,
  especially for beliefs and behaviors that should
  not change (e.g. political views)
• Significantly fewer people will admit that taxes
  are too high if they are first asked if the
  government should increase spending in certain
  areas (Turner and Krauss 1978)
• Respondents will try to be even-handed in their
  treatment of opposed parties
• When asked about international trade policies,
  respondents be more willing to accept right of
  foreign countries to impose tariffs if they have
  already agreed on the right of the US to do so
  (Schuman and Ludwig 1983)

30
Consistency bias

• Markus (1986) conducted panel survey 1973-1982 on
  political positions.
• Found moderate-low correlation on recall of past
  position and actual past position (0.39)
• Found moderate-low correlation between past and
  present position (0.42)
• Found high correlation between present position
  and recall of past position (0.79)
• Therefore
• policy attitudes generally do not have strong
  cognitive representations, are eminently
  changeable, and once they are changed, an
  individual's cognitive autobiography is revised
  so as to render the changes invisible.

31
Ordering bias

• Priming effect
• Discounting effect
• Rapport effect
• Fatigue effect

32
Priming effect or Salience bias

• When content of previous questions suggests
  possible answers
• When respondents are asked questions about gas
  prices and then asked about problems facing the
  country they are much more likely to mention the
  energy crisis (McFarland 1981)
• When respondents are asked about the frequency of
  dating following a question about personal
  happiness there is a strong correlation (r.65)
  but when reversed there is no significant
  correlation (r-.12) (Clark and Shober 1992 41-2)

33
Discounting effect

• When content of previous questions excludes
  possible answers
• When respondents are asked about factors
  affecting their personal happiness after being
  asked questions about their marriage they are
  less likely to mention their marriage (Foddy
  1993 65)
• Note that general happiness tends to by reported
  as higher after questions about marriage

34
Rapport effect

• Trust builds during interview.
• Response to question have you ever been troubled
  by insomnia increases from 12 to 23 when asked
  after general questions on health (Thumin 1962)

35
Fatigue effect

• Often leads to mechanical biases

36
Mechanical biases

• Respondent answers questions according to an
  established pattern
• Response sets
• Halo effect
• Central tendency
• First versus last in a list
• Positive versus negative

37
Measuring attitudes

• Assumption is that attitudes are bi-polar and
  linear, but may be multivariate and non-linear
• Intensity (strength of feeling)
• Centrality (importance of issue)
• Stability (persistence of feeling)
• Commitment (effect on behavior)

38
(No Transcript)
39
Interviewer effects

• The identity of the interviewer affects responses
• Black candidates almost always do less well in
  actual elections than in opinion polls
• Black interviewers increase likelihood of
  respondent saying they will vote for a black
  candidate by 10 (Finkel, Guterboch and Borg 1991)

40
Check questions

• Definition
• Questions asked later in survey to check for
  consistency in answers
• Pros
• Used to verify previous answers
• Cons
• Might make respondent suspicious

41
Probing questions

• Definition
• Clarification of the meaning of question or
  response in cases when
• Respondent does not understand question
• Answer is not apparent
• Responses are contradictory
• Pros
• Natural
• Cons
• Affects reliability of response

42
Rules for probing

• Non-directive
• Dont provide examples of possible answers!
• Scripted
• How do you mean that?
• Could you tell me more about that?
• Anything else?
• Fowler and Mangione (1990, cited in Foddy 1993
  135)

43

• Bradburn, Norman M. (2004) Asking Questions The
  Definitive Guide to Questionnaire Design. John
  Wiley.
• Schuman, Howard (1996) Questions And Answers In
  Attitude Surveys Experiments On Questions Form,
  Wording, and Context. Sage.