Survey Research Methods

1
Survey Research Methods
Week 1. Designing Questionnaires
Steve Fisher, Robert Andersen Anthony Heath
Department of Sociology UNIVERSITY
of OXFORD
2
The Logic of Sampling Measurement
3
General Problem of MeasurementReliability
Validity

• Reliability
• Refers to the replicability of the measurement
  procedure to yield consistent results
• Validity
• Refers to the extent to which the measurement
  procedure actually measures the concept that it
  is intended to measure

4
Reliability ValidityQuestionnaire related
problems

• Wording effectsword choice, loaded questions
• Respondent recall
• Social desirability, knowledge questions,
  threatening questions
• Question order effects
• Response format differences, recency effects in
  closed-ended questions
• Dont know responses
• Middle position in polar alternatives

5
Question wording problemsAssociated factors

• Word selection
• Seemingly trivial changes in wording can affect
  response distributions (e.g., forbid versus not
  allow)
• Ambiguous, lengthy, double-barrelled questions
• Unless you need to establish context, short
  questions are best
• Have only one thought per question
• Bad Could you please tell me how many children
  you have, and what year each of them was born?
• Good What are the ages of your children?
• Biased, leading or loaded questions
• Avoid questions that make the respondent feel
  they should respond a certain way

6
Question wordingExperimental evidence
7
Respondent RecallSituational framing
decomposition

• Telescopingbringing distant events closer to the
  present than they actually occurred
• Limited by Situational framingi.e., placing the
  event in the context of other events, especially
  well known events
• e.g., How many times have you been to the doctor
  since last Christmas?
• Inaccurate calculation of a large number of
  events during a long time period
• Limited by Decompositioni.e., breaking time into
  smaller periods
• e.g., If you want to determine how much alcohol
  people drink in a year, ask them how much they
  drink each week and calculate the year value
  yourself

8
Social desirability, threatening questionsUsing
lead-in statements

• People generally have a desire to present a
  positive image which can override the norm to
  tell the truth
• Errors associated with this can be limited by
  adding lead-in statements
• e.g. 1 Voter turnout
• Did you vote in the last election?
• Many people were not able to vote last election.
  Were you able to vote?
• e.g. 2 Theft
• Did you ever steal as a child?
• Most people have stolen something when they were
  children. Did you ever steal as a child?

9
Question OrderingContext effects

• Question responses can be affected by the context
  in which the question appears
• Question order-effects usually involve a group of
  questions dealing with the same issue
• Replication problem
• Difficult to compare results of different surveys
  over time since it is seldom possible to
  replicate an entire questionnaire.
• This is seldom taken into consideration in trend
  studies, however.
• No real solution other than carrying out
  experiments within the questionnaire to assess
  the impact of context

10
Closed-ended questionsPros cons

• Advantages
• Quick easy for respondents
• Less articulate are not at a disadvantage
• Response choices can clarify alternatives
• Fewer irrelevant answers
• Easy to code and analyse
• Disadvantages
• Responses suggest ideas (e.g., No
  opinion/knowledge still give opinion)
• Frustrates respondents if categories are not
  exhaustive
• Misinterpretation goes unnoticed
• Complex issues forced into simple categories
• Recency effects

11
Open-ended questionsPros cons

• Advantages
• Permits detail, clarification
• Unanticipated answers
• Reveals the logic behind a respondents response
• Disadvantages
• Generalization or comparison difficult
• Coding and statistical analysis difficult
• Irrelevant answers possible
• Bias towards educated

12
Recency effectsExperimental evidence
13
Dont know Responses Missing data or important
meaning?

• Missing data approach
• Standard question does not include DK category
• Interviewers probe in order to force an opinion
• Important meaning approach
• DKs may represent a lack of knowledge, in which
  case they are acceptable responsesprobing
  invents data
• Converse suggests that much of survey errors
  result from random responses by people have no
  real opinion
• Filter questions can be used to determine
  interest and knowledge in a topic before asking
  opinions
• see Converse, P.E. (1970) Attitudes and
  non-attitudes Continuation of a dialogue, The
  Quantitative Analysis of Social Problems.
  Reading, Mass. Addison-Wesley.

14
DK Responses Experimental Evidence
15
Measuring a middle positionShould it be built
into the question?

• e.g., Do you think that state funding for
  universities should be increased or decreased?
• Debate over whether the middle alternative should
  be asked explicitly or simply accepted when
  offered
• Two rationales for omitting the middle category
  are
• Largely people who lean to one side but with
  little intensity
• It attracts those who have no opinion (would
  rather choose it than dont know)
• On the other hand, explicitly offering a middle
  position makes good theoretical sense since those
  who do opt for it may actually favour the middle
  and thus should not be forced into another
  category.

16
Measuring the middle position Experimental
evidence
17
Selective approach to midpoint and DK Rob Johns

• Never have a midpoint without a DK response
• Familiar items should have both, unfamiliar
  should have neither
• Minority is overestimated if indifference is
  common but there is no midpoint, so keep both
• Majority is overestimate if there is strong
  social desirability pressures and a midpoint, so
  omit both
• Familiarity and social desirability can be
  determined from pre-tests.
• Okay to use different formats in the same
  questionnaire

18
Comparative Survey Research

• Difficult to know whether the same question means
  the same thing in different countries
• E.g. Chinese report being more politically
  efficacious than the Mexicans
• King et al (2004) use the following vignettes to
  anchor the question
• Alison lacks clean drinking water. She and her
  neighbors are supporting an opposition candidate
  in the forthcoming elections that has promised to
  address the issue. It appears that so many people
  in her area feel the same way that the opposition
  candidate will defeat the incumbent
  representative.
• Imelda lacks clean drinking water. She and her
  neighbors are drawing attention to the issue by
  collecting signatures on a petition. They plan to
  present the petition to each of the political
  parties before the upcoming election.
• Jane lacks clean drinking water because the
  government is pursuing an industrial development
  plan. In the campaign for an upcoming election,
  an opposition party has promised to address the
  issue, but she feels it would be futile to vote
  for the opposition since the government is
  certain to win.
• Toshiro lacks clean drinking water. There is a
  group of local leaders who could do something
  about the problem, but they have said that
  industrial development is the most important
  policy right now instead of clean water.
• Moses lacks clean drinking water. He would like
  to change this, but he cant vote, and feels that
  no one in the government cares about this issue.
  So he suffers in silence, hoping something will
  be done in the future.

19
Constructing a good questionnaireGeneral advice

• Use social conversation as a guide to both
  question construction and questionnaire design
• Questions should be straight to the point
• One thing per question
• Avoid jargon, slang, abbreviations
• Avoid asking about vague future intentions or
  hypothetical questions
• Avoid wording that is influential or offensive
• If little is known use open questions otherwise
  closed-ended questions with exhaustive and
  mutually exclusive response categories are
  typically better
• Borrow questions from existing literature
• Pre-test

20
Next week

• Please do the readings for your group and for the
  lecture material BEFORE coming to class