Survey Research Methodology

1
Survey Research Methodology

• Gary R. Hanson
• Arizona State University
• HED691 Applied Research Methods

Material adapted from Dillman, D. A. (2000).
Mail and Internet Surveys The Tailored Design
Method. New York John Wiley Sons, Inc.
2

• Creating a survey is like driving in freeway
  traffic while drinking a hot cup of coffee and
  answering an emergency call on your cell phone.

Many things are competing for your attention and
failure to heed any of them can spell disaster.
3
Dillmans Tailored Design Method

• Part 1 identify each aspect of the survey
  process that affects either the quality or
  quantity of response
• Part 2 organize the survey process so the
  design intentions were carried out in complete
  detail

4
Survey Responses as Social Exchange

• Quality surveys are based on two principles
• Responding to a self-administered survey involves
  not only cognition, but motivation
• Multiple attempts are essential to achieving
  satisfactory response rates

Dillman (2000). p. 13.
5
Why TDM is Important !Reduce 4 Sources of Error

• Sampling
• Coverage
• Measurement
• Nonresponse

• Surveying only some and not all elements of the
  survey population
• Not allowing all members of population to have an
  equal chance of being sampled
• Poor wording/poor directions
• Survey non-respondents being different than
  respondents

6
Social Exchange Theory

• Social Exchange Theory asserts that actions of
  individuals are motivated by the return these
  actions are expected to bring from others.

7
3 Critical Factors in S.E.T.

• Rewards
• Costs
• Trust

• What one expects to gain from a particular
  activity
• What one gives up or spends to obtain the
  rewards
• The expectation that in the long run the rewards
  outweigh the costs

8
Ways of Providing Rewards

• Show positive regard
• Say Thank You
• Ask for advice
• Support group values
• Make the questionnaire interesting
• Give social validation
• Inform respondents that opportunities to respond
  are scarce
• Give tangible rewards

Dillman (2000). P. 15-17.
9
Ways to Reduce Social Costs

• Avoid subordinating language
• Avoid embarrasment
• Avoid inconvenience
• Make questionnaire appear short and easy
• Minimize requests to obtain personal information
• Keep requests similar to other requests to which
  a person has already responded

Dillman (2000). P. 17-18.
10
Ways to Establish Trust

• Provide a token of appreciation in advance
• Sponsorship by legitimate authority
• Make the task appear important
• Invoke other exchange relationships

11
Writing Questions

• The Dillman Principles

12
Principle 1

• Choose simple over specialized words
• If words have more than 6-7 letters you can find
  a better word
• Most mistakes are made when you overestimate the
  knowledge level of your participants

13
Principle 2

• Choose as few words as possible to pose the
  question
• The problem with long questions is that the
  reader tries to be efficient or is simply in a
  hurry and will give uneven attention to each
  word. They will skim. Skimming leads to
  misinterpretation.

14
Principle 2 Example

• Do you strongly favor, somewhat favor, strongly
  oppose, or have no opinion on whether race should
  be considered a factor in college admissions?
• a. Strongly favor
• b. Somewhat favor
• c. Somewhat oppose
• d. Strongly oppose
• e. No opinion

• To what extent do you favor or oppose using race
  as a factor in college admissions?
• a. Strongly favor
• b. Somewhat favor
• c. Somewhat oppose
• d. Strongly oppose
• e. No opinion

15
Principle 3

• Use complete sentences to ask questions

Number of years lived in Arizona _____
Years
How many years have you lived in Arizona? _____
Years
16
Principle 4

• Avoid vague quantifiers when more precise
  estimates can be obtained.

• How often did you attend
• religious services during
• the past year?
• Not at all
• A few times
• About once a month
• Two to three times a month
• About once a week
• More than once a week

• How often did you attend
• religious services during
• the past year?
• Never
• Rarely
• Occasionally
• Regularly

17
Principle 5

• Avoid specificity that exceeds the respondents
  potential for having an accurate, ready-made
  answer.

About how many books have you read for leisure
during the past year? _____ Number of books

• About how many books have you read for leisure
  during the past year?
• None
• 1-2
• 3-5
• 6-10
• 11 or more

18
Principle 6

• Use equal numbers of positive and negative
  categories for scalar questions

• How satisfied were you with your academic
  advising last semester?
• Completely satisfied
• Mostly satisfied
• Somewhat satisfied
• Neither satisfied nor dissatisfied
• Dissatisfied

• How satisfied were you with your academic
  advising last semester?
• Completely satisfied
• Somewhat satisfied
• Neither satisfied nor dissatisfied
• Dissatisfied
• Completely dissatisfied

19
Principle 7

• Distinguish undecided from neutral by placement
  at the end of the scale

• To what extent do you agree with this statement
  Living in Arizona is better for a persons
  emotional health.
• Strongly agree
• Agree
• Undecided
• Disagree
• Strongly disagree

• To what extent do you agree with this statement
  Living in Arizona is better for a persons
  emotional health.
• Strongly agree
• Agree
• Neither agree or disagree
• Disagree
• Strongly disagree
• No opinion

20
Principle 8

• Avoid bias from unequal comparisons

• Which of the following do you feel is most
  responsible for recent outbreaks of violence in
  Americas schools?
• Irresponsible parents
• School policies
• Television programs

• Which of the following do you feel is most
  responsible for recent outbreaks of violence in
  Americas schools?
• Parents
• Schools
• Television

21
Principle 9

• State both sides of attitude scales in the
  question stems

• To what extent do you agree with this statement
  It is easier to find work in this community than
  it was about a year ago.
• Strongly agree
• Somewhat agree
• Somewhat disagree
• Strong disagree

• To what extent do you agree or disagree with
  this statement It is easier to find work in
  this community than it was about a year ago.
• Strongly agree
• Somewhat agree
• Somewhat disagree
• Strong disagree

22
Principle 10

• Eliminate check-all-that-apply question formats
  to reduce primacy effects

• In which campus events would you like to
  participate? Please check all that apply
• Sports
• Music
• Lecture
• Art gallery

Do you plan to participate in these events next
semester? Yes No Sports
Y N Music
Y N Lecture Y
N Art gallery Y
N
23
Principle 11

• Develop response categories that are mutually
  exclusive

• From which one of these sources did you first
  learn about the hurricane in Yuma?
• Radio
• Television
• Someone at work
• While at home
• Traveling to work

24
Principle 11, continued

• From which one of these sources did you first
  learn about the hurricane in Yuma?
• Radio
• Television
• Another person

• Where were you when you first learned about it?
• Home
• Work
• Commuting

25
Principle 12

• Use cognitive design techniques to improve recall

• Please think about the most recent time you
  drove or rode anywhere in an automobile, van or
  pickup. During this most recent ride, would you
  say that your seatbelt was fastened . . .
• All the time, every minute the car was moving
• Almost all the time
• Most of the time
• About half the time
• Less than half the time
• Not all during the time the vehicle was moving

26
Principle 12, continued

• When was the last time you drove in a vehicle?
• Today
• Yesterday
• Sometime before that

• During this trip would you say that your
  seatbelt was fastened . . .
• All the time, every minute the car was moving
• Almost all the time
• Most of the time
• About half the time
• Less than half the time
• Not all during the time the vehicle was moving

• During this trip were you the
• Driver
• Front seat passenger
• Back seat passenger
• Other

27
Principle 13

• Provide appropriate time referents

How many times have you played golf so far this
year? _____ Number of times played this year
During the calendar year 2001, about how many
times did you play golf?_____ Number of times
played in 2001
28
Principle 14

• Be sure each question is technically accurate?

• Another activity of the police department is the
  catching and fining of traffic violators. How
  much emphasis should this activity receive?
• Greater
• The same
• Less

• Another activity of the police department is the
  catching of traffic violators. How much emphasis
  should this activity receive?
• Greater
• The same
• Less

29
Principle 15

• Choose question wordings that allow essential
  comparisons to be made with previously collected
  data

• Is this house, apartment, or mobile home
• Owned by you or someone in this household with a
  mortgage or loan
• Owned by you or someone in this household free
  and clear (without a mortgage or loan)
• Rented for cash rent
• Occupied without payment of cash rent?

• Do you own or rent the home in which you live?
• Own
• Rent

30
Principle 16

• Avoid asking respondents to say YES in order to
  mean NO

• Do you favor or oppose not allowing the state to
  raise taxes without approval of 60 of the
  voters?
• Favor
• Oppose

• Do you favor or oppose requiring 60 approval by
  voters in order to raise state taxes?
• Favor
• Oppose

31
Principle 17

• Avoid double-barreled questions

• Should the city build a new swimming pool that
  includes lanes for swimming laps that is not
  enclosed for winter use?
• Yes
• No

• Should the city build a new swimming pool that
  includes lanes for swimming laps ?
• Yes
• No

• Should the city build a new swimming pool that
  is enclosed for winter use?
• Yes
• No

32
Principle 18

• Soften the impact of potentially objectionable
  questions

What was your total income from all sources in
2001? ________ total income for 2001

• What was your total income from all sources in
  2001?
• 10,000 or less
• 10,001 to 20,000
• 20,001 to 35,000
• 35,001 to 50,000
• 50,001 to 100,000
• 100,001 or more

33
Principle 18, continued

• Most religions are a parasite on society. Do
  you agree or disagree?
• Agree
• Disagree

• Here are many different opinions on religions we
  have heard from others. Please tell us whether
  you agree or disagree with each of them.
• Most religions try to help members as well as
  non-members
• Agree
• Disagree
• 2. Most religions are a parasite on society
• Agree
• Disagree

34
Principle 19

• Avoid asking respondents to make unnecessary
  calculations

How many nights did you spend away from home on
trips of any kind during 2001? ___ total nights
away from home
What percent of the nights spent away from home
on trips during 2001 were for business
reasons? ___ percent
How many of these nights away from home were
because of business? ___ nights away from home
for business reasons
35
Dillmans Quality Survey ItemChecklist

• Does the question require an answer?
• To what extent do survey recipients have an
  accurate, ready-made answer for the question they
  are being asked to report?
• Can people accurately recall and report past
  behavior

36
Dillmans Quality Survey ItemChecklist

• Will the respondent feel motivated to answer each
  question?
• Is the respondents understanding of response
  categories likely to be influenced by more than
  words.
• Is survey information being collected by more
  than one mode?

37
Dillmans Quality Survey ItemChecklist

• Is changing a question acceptable to the survey
  sponsor?

38
The End