Survey Research

1
Survey Research

• Chapter 10 (JRM)
• Article Critique Due April 3rd

2
Ch. 10 Elite Interviewing and Survey Research

• Elite Interviewing the process of interviewing
  respondents in a nonstandardized, individualized
  manner.

3
Why do an elite interview vs. sending elites a
survey?

• 1.  Researcher may not know enough about the
  issue to construct a good survey device.
• 2.  Researcher does not want to constrain the
  responses of the elite.
• 3.  Elite respondents may feel a standardized
  survey is beneath them.

4
Survey Research

• Popular form of political science research (also
  called opinion polling).
• Goal is to measure accurately peoples
  attitudes, beliefs, and behaviors by asking them
  questions

5
Survey Research

• Two types of error exist in survey research
• Sampling error error due to the fact that a
  sample of people are used to estimate population
  characteristics (parameters)
• Response error error due to features of the
  survey instrument

6
Sampling Error

• Sampling distribution collection of sample
  statistics calculated from all possible random
  samples of a given size, taken from the same
  population
• Statistical theory tells us what that
  distribution of sample statistics looks like

7
Sampling Error (contd)

• Margin of Error the amount of possible error
  that exists for a sample statistic (with high
  degree of certainty)
• Example An estimate of presidential approval for
  Obama is 60 (4), with 95 certainty
• Margin of error allows you to calculate a
  confidence interval 604(64,56)

8
Sampling Error (contd)
9
Interpreting Margin of Error
10
Response Error

• Survey experts agree BY FAR the largest amount
  of error in survey research comes from response
  error

11
Sources of Response Error

• Survey Administration
• Question Order
• Question Type
• Question Wording

12
Survey Administration

• There are three typical methods used to
  administer surveys
• Interview
• Telephone
• Mail

13
Personal/Telephone Interviewing

• Advantages
• High response rate
• Flexibility/control in questioning
• Observation (personal interviewing)
• Supervision of interviewers (telephone)

14
Personal/Telephone Interviewing

• Disadvantages
• Expensive
• Lack of anonymity
• Characteristics of interviewer may introduce bias

15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
Mail Surveys

• Advantages
• Cheap!!!
• Often avoid bias due to interviewer
  characteristics / social desirability

19
Mail Surveys

• Disadvantages
• Low response rates
• Cannot probe
• Cannot clarify
• Cannot be sure who completes questionnaire

20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
Dillmans Total Design Method for Mail Surveys

• First Mailing 24 response rate.
• Typically, when you send a survey out by mail,
  about 24 of those you sent it to will respond.
• Why do we care about the response rate so much???

25
Dillman Method

• After you send the first survey, wait about two
  weeks and then send a follow-up postcard
• The postcard should be sent only to
  non-respondents to save money
• Disadvantage Loss of anonymity

26
Dillman Method

• The postcard should say something like, In the
  past week or so you received a survey from me
  about XYZ. I know you are a very busy person,
  but it is important for me to include your
  opinions in my research. If you still have the
  survey, could you please complete it and send it
  to me in the stamped addressed envelope I
  included? If it got misplaced, I am happy to
  send you a replacement survey. Please call or
  email me at ABC and I will send a replacement to
  you right away. Thank you so much for helping me
  with this important project. Sincerely, Jane
  Doe.
• The postcard follow-up typically increases
  response rate to about 42.

27
Dillman Method

• About one to two weeks after you send the
  postcard, send the first replacement
  questionnaire
• Attribute the lack of response to the first
  survey getting lost in the mail, accidentally
  misplaced or discarded, or lack of time, etc.
  Stress the importance of that persons response
  to your project.
• The first replacement survey will usually get
  your response rate up to about 59.

28
Dillman Method

• Send a second replacement survey by certified
  mail about two weeks after the first replacement
  survey was sent. This will increase the response
  rate to about 72

29
Dillman Method

• Other pieces of advice to maximize response rate
• Keep it simple
• Make it look like personal mail
• Get a sponsor
• Offer inducements

30
The New Frontier in Survey Administration The
Internet

• Email surveys
• Web surveys

31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
Question Order

• The order in which you place your questions can
  cause the error of a response set.
• A response set may occur when a series of
  questions have the same answer choices. This is
  most likely to happen if questions are on related
  subjects.

35
Question OrderExample Response Set (from the
GSS)

• Listed below are various areas of government
  spending. Please indicate whether you would like
  to see more or less government spending in each
  area. Remember that if you say much more, it
  might require a tax increase to pay for it.
• A. The environment.
• 1. SPEND MUCH MORE
• 2. SPEND MORE
• 3. SPEND SAME
• 4. SPEND LESS
• 5. SPEND MUCH LESS
• 6. CANT CHOOSE

36
Question OrderExample Response Set (from the
GSS)

• B. Health Care.
• 1. SPEND MUCH MORE
• 2. SPEND MORE
• 3. SPEND SAME
• 4. SPEND LESS
• 5. SPEND MUCH LESS
• 6. CANT CHOOSE

37
Question OrderExample Response Set (from the
GSS)

• C. Education.
• 1. SPEND MUCH MORE
• 2. SPEND MORE
• 3. SPEND SAME
• 4. SPEND LESS
• 5. SPEND MUCH LESS
• 6. CANT CHOOSE

38
Question OrderExample Response Set (from the
GSS)

• D. Unemployment Benefits.
• 1. SPEND MUCH MORE
• 2. SPEND MORE
• 3. SPEND SAME
• 4. SPEND LESS
• 5. SPEND MUCH LESS
• 6. CANT CHOOSE

39
Question Order

• Etc.
• What is the danger in a response set and what can
  we do about it???

40
Question Order

• The order in which you place your survey
  questions can cause the error of a saliency
  effect
• Specific mention of an issue in a survey may
  cause a respondent to associate the issue in
  connection with a later question.

41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
Question Order

• Consistency is another problem we might encounter
  due to question order.
• Respondents may strive to appear consistent in
  their answers an answer to a question may be
  constrained by an answer given earlier.

45
Question OrderExample Consistency

• And what about the federal government, does it
  have too much power or too little power?
• FAR TOO MUCH
• TOO MUCH
• RIGHT AMNT OF POWER
• TOO LITTLE POWER
• FAR TOO LITTLE POWER
• CANT CHOOSE
• What if this question preceded the government
  spending questions?

46
Question Order

• A final concern about question order is fatigue
• Respondents may give perfunctory answers to
  questions late in the survey especially if it is
  long
• In addition, response sets are often a problem in
  lengthy surveys toward the end

47
Question Order

• Some final thoughts on question order
• The first few questions should break the ice
  dont jump in with a big question.
• Move from general questions to more specific
  questions.
• Questions on personal or sensitive topics (such
  as income, race, age) should be asked at the end
  with a big disclaimer about why you are asking
  those questions (also helps with fatigue because
  respondents dont have to think about the answers)

48
Question Type

• Closed-ended questions are questions that have
  response options provided by the researcher.
• What are some of the advantages and disadvantages
  to closed-ended questions?

49
Question Type

• Advantages of Closed-Ended
• Easy to answer
• Answers are easy to compare
• Respondents willing to provide more personal
  information
• May help clarify the question

50
Question Type

• Disadvantages of Closed-Ended
• Forces people to accept the categories or puts
  too many responses into the other category.
• Choices listed may communicate the kind of
  response wanted.

51
Question Type

• Open-ended questions allow the respondent to
  write in an answer.
• What are some of the advantages and disadvantages
  to open-ended questions?

52
Question Type

• Advantages of Open-ended
• May better indicate respondents
  thoughts/beliefs/opinions.
• Better when the list of responses might be
  excessive.
• Might get unanticipated answers.

53
Question Type

• Disadvantages of Open-ended
• Respondent may say too much or too little.
• Difficult to code all the various answers and
  analyze them.
• Can be expensive and time-consuming.

54
Question Type

• Single-sided vs. Two-sided questions.
• Single-sided a question that asks respondents to
  agree or disagree with a single substantive
  statement.
• Two-sided a question that offers the respondent
  two substantive choices.

55
Question Type

• Single-sided example
• Do you agree or disagree with the idea that the
  government should see to it that every person has
  a job with a good standard of living?

56
Question Type

• Two-sided example
• Do you think that the government should see to it
  that every person has a job with a good standard
  of living, or should it let each person get ahead
  on his or her own?

57
Question Type

• With a single-sided question a larger percentage
  of respondents tend to agree with the statement
  given.

58
Question Type

• 44 of respondents agreed with the single-sided
  statement.
• Only 30.3 of respondents agreed government
  should guarantee employment and standard of
  living in the two-sided question.
• Two-sided questions reduce the proportion of
  respondents who give no opinion.

59
Question Type

• Social Desirability a form of response error in
  which respondents are not truthful due to the
  fact that they perceive their answer to be out of
  step with the mainstream of society

60
Race Relations and the New South
61
Race Relations and the New South

• Experimental Design
• Random assignment of individuals into two groups
  (national random sample of white Americans)

62
Race Relations and the New South
63
Race Relations and the New South
64
Race Relations and the New South

• Result
• South 42 angry
• Non-South 11 angry

65
Question Wording

• Double-Barreled Questions are two questions in
  one. Example
• Do you agree with the statement that the
  situation in Iraq is deteriorating and that the
  United States should increase the number of
  troops in Iraq?
• What is wrong with how this question is worded?

66
Question Wording

• Ambiguous Question one that contains a concept
  that is not clearly defined.
• What is your income?
• Do you prefer Brand A or Brand B?
• Rate the performance of the two debaters as good,
  bad, or indifferent.
• What is the problem with these three survey
  questions???

67
Question Wording

• The choices you provide on a frequency scale can
  influence how respondents answer. Consider this
  question
• How much tv do you watch per day?

68
Question Wording

• When the response options were
• Up to .5 hours
• .5 to 1 hour
• 1-2 hours
• 2-2.5 hours
• 2.5 hours
• 16 responded they watched 2.5 hours.

69
Question Wording

• When the response options were
• Up to 2.5 hours
• 2.5 to 3 hours
• 3-4 hours
• 4-4.5 hours
• 4.5 hours
• 38 responded they watched 2.5 hours.
• (From Schwarz et al., 1985)

70
Question Wording

• Leading Questions Encourages respondents to
  choose a particular response because the question
  indicates the researcher expects it.
• Dont you think that global warming is a serious
  environmental problem?

71
Question Wording

• Polls conducted by political organizations and
  politicians often include leading questions.
• 1980 poll for the Republican National Committee
  read as follows on the next slide (a dated, but
  classic example that is widely used).

72
Question Wording

• Recently the Soviet armed forces openly invaded
  the independent country of Afghanistan. Do you
  think the U.S. should supply military equipment
  to the rebel freedom fighters?

73
Question Wording

• Word Choice may lead respondents to an answer as
  well. You have to really think about what words
  you use to describe people, things, and events.

74
Question Wording

• Helping the needy compared to those on
  welfare.
• Socialized Medicine vs. National Health
  Insurance
• Fight Against Terrorism vs. Going to War

75
Question Wording

• Linking personalities or institutions to issues
  can also affect responses.
• Would you say that Governor Beshears program
  for promoting economic development has been very
  effective, fairly effective . . .
• A respondents feelings about Governor Beshear
  might affect his or her response to the question
  about economic development.

76
Question Wording

• Push Polls These are relatively new and
  conducted by campaigns, parties, and political
  organizations.
• Interviewers feed respondents false and damaging
  information about a candidate or cause under the
  guise of asking a question.

77
Question Wording

• Do you agree or disagree with Candidate Xs
  willingness to tolerate terrorism in our
  country?
• The goal is NOT to conduct research but to use
  innuendo to spread rumors and lies.
• This is a politically risky strategy and if its
  too blatant, it can back-fire.

78
Question Wording

• Confusing Questions should be avoided. Yes, this
  seems so obvious, but even the experts make
  mistakes sometimes. Double-negatives are
  especially confusing.
• The following question was asked of a national
  random sample by the Roper Center, a respected
  polling organization.

79
Question Wording

• Does it seem possible or does it seem impossible
  to you that the Nazi extermination of the Jews
  never happened?
• A) Impossible it never happened
• B) Don't know
• C) Possible it never happened

80
Question Wording

• The responses were as follows
• Impossible it never happened 65
• Don't know 12
• Possible it never happened 22

81
Question Wording

• Does it seem possible to you that the Nazi
  extermination of the Jews never happened or do
  you feel certain that it happened?
• A) Certain it happened
• B) Don't know
• C) Possible it never happened

82
Question Wording

• The responses were as follows
• Certain it happened 91
• Don't know 8
• Possible it never happened 1

83
The Polls and the 2008 Presidential Election

• http//www.pollster.com/polls/us/08-us-pres-ge-mvo
  .php
• http//www.pollster.com/blogs/the_art_and_science_
  of_choosin.php
• http//pewresearch.org/pubs/964/cell-phones-and-th
  e-2008-vote-an-update