Survey Design

1
Survey Design

• Stats

2
Evaluating Surveys

• Sample size/margin of error
• How sample is selected representative?
• Question Wording
• Non-response

3
Sample Size

• Sample Size 
• Z Z value (e.g. 1.96 for 95 confidence level) 
• p percentage picking a choice, expressed as
  decimal(.5 used for sample size needed)
• c confidence interval, expressed as
  decimal       (e.g., .04 4)

4
Samples

• Size
• Are they representative?

5
Wording Matters

• With which of these statements do you most agree
  protection of the environment should be given
  priority, even at the risk of limiting the amount
  of energy supplies, such as oil, gas, and coal,
  which the U.S. produces or, the development of
  U.S. energy supplies, such as oil, gas, and goal,
  should be given priority, even if the environment
  suffers to some extent?

• What do you think is more important producing
  energy, or protecting the environment?

CBS News / New York Times 11/02
Gallup 3/04
6
Question Wording

• Framing
• Priming
• Does it actually answer the question you want to
  ask?

7
Americans say no tradeoff, but we still ask!

• "With which one of these statements about the
  environment and the economy do you most agree?
  Protection of the environment should be given
  priority, even at the risk of curbing economic
  growth. OR, Economic growth should be given
  priority, even if the environment suffers to some
  extent."

• Do you believe that economic growth should be
  sacrificed for environmental quality, or should
  environmental quality be sacrificed for economic
  growth, or does it not necessarily have to be a
  choice between the two?

Source Wirthlin Worldwide 10/99
Source Gallup 3/08
8
Designing Surveys

• The logistics of creating and running a survey
• Survey design - how to get answers that address
  the research question
• wording individual questions
• overall structure of the survey
• Eliciting information about attitudes (e.g., New
  Ecological Paradigm)
• How to analyze/interpret responses

9
Logistics

• Decide on Topic
• Decide on structure of survey.
• Decide on sample.
• Develop testable hypotheses.
• Draft questions.
• Human Subjects
• Pretest finalize questions
• Field Survey and enter data
• Present Findings
• Mistakes This take time!

10
Sample

• Sample size
• Sample frame how do you select and access these
  people
• Can only generalize from probability sample
• Mistakes Non-representative sample

11
Methods of Data collection

• Personal
• Mail
• Telephone
• Internet
• Multiple modes
• Issues Cost, response rate, social desirability
  bias, sample frame and design, coverage,
  completeness of data
• Response effects response order, acquiescence,
  extremeness, degree of interaction, privacy
• Mistakes
• Go with internet because its easy
• Find out you cant reach your sample
• Too costly
• Takes too long

12
How do you help a respondent respond?

• Comprehend the Question
• Make as specific as possible
• Use words virtually everyone will understand
• Retrieve (remember) the information
• Include all reasonable responses
• Lengthen the questions by adding memory cues to
  improve recall
• When forgetting is likely, use aided recall.
• How many times last month did you recreate
  outdoors? (bike, surf, hike, rockclimb, etc.)

13
Helping the respondent, cont.

• Use judgment and estimation
• Recall and count
• Rate-based
• Impression-based
• Report an answer
• Now think about the past 12 months, from DATE
  through today. We want to know how many days
  youve used any prescription tranquilizer that
  was not prescribed to you or that you took only
  for the experience or feeling it caused during
  the past 12 months. (National Survey on Drug Use
  and Health)

14
Mistakes in Question Wording

• Too complicated
• Make assumptions that arent accurate

15
Order matters for Questions and Answers

• Questions
• General first
• Demographics last
• Always get some roll-off
• Answers
• Randomize order
• Except Yes/No, then affirmative first
• Multiple items start with least popular
• Start with least popular end of scale. Unless
  verbal, then end with least popular.
• Mistakes Reinvent the wheel, or dont invent it
  at all. Use textbooks, etc.

16
Formatting Answers

• Open-ended more information but subject to
  problems
• Unordered responses
• Primacy first option gets picked more (mail)
• Recency more recent get picked more (phone)
• Randomize

17
Formatting Answers, cont.

• Attitudes Likert Scale Ordered
• Scales
• Label all parts of scale
• Branching (D,R,I then strong or weak)
• 7 points best
• Choose center category closest to population
  average

18
Modified New Ecological Paradigm(from Dunlap et.
al. 2000)

• Listed below are statements about the
  relationship between humans and the environment.
  For each one, please indicate whether you
  STRONGLY AGREE, MILDLY AGREE, are UNSURE, MILDLY
  DISAGREE, or STRONGLY DISAGREE with it.
• 1. We are approaching the limit of the number of
  people the earth can support.
• 2. Humans have the right to modify the natural
  environment to suit their needs.
• 3. When humans interfere with nature it often
  produces disastrous consequences.
• 4. Human ingenuity will ensure that we do NOT
  make the earth unliveable.
• 5. Humans are severely abusing the environment.

19
Reviewing Questions

• Expert Review
• Focus groups
• Cognitive Interviews
• Field pretests
• Randomized experiments
• Infinite mistakes can be made
• Regions
• Conservativeanti-environment
• Dont make the mistake of not pretesting

20
Dealing with data and non-response

• Data You must post-code and look for
  problems/patterns
• Non-response
• Keep bugging respondents
• See if you have a pattern to the non-response
• Weighting

21
Weighting

• Types
• Weighting for differential selection
  probabilities
• Weighting to adjust for unit non-response
• Post-stratification

22
Weighting to adjust for unit non-response

• Suppose we want Bren s but
• PhD Students40 with 50 response rate20
  respondents
• MESM Students130 with 90 response rate117
  respondents
• Sample PhD/total23.5
• Respondents PhD/total14.5
• Assume that within cohorts, respondents are
  random sample
• Inverse of response rate used as weight (2 for
  PhD, 1.111 for MESM) (wi3)

23
Post-stratification weighting

• Suppose we have equal s of males and females in
  survey, but we know that Bren is 60 female
• Deflate males by .4/.5.8
• Inflate females by .6/.51.2 (Wi4)

24
Total weights product

• Example
• Male MESM 1.1.8.88
• Male PhD 2.81.6
• Female MESM 1.11.21.32
• Female PhD 21.22.4

25
Interpreting
26
Cross-Tab Analysis of Degree Programs and
Donation of Time to Beach Conservation Efforts
Note p-value is based on 2 degrees of freedom
27
Good Resources

• Take the class!!
• Survey monkey
• Social Science Research Center
• Textbooks
• Rea and Parker Designing and Conducting Survey
  Research
• Groves, et al Survey Methodology
• Pollingreport.com for examples