A Brief Introduction to Survey Research Methods Matthew J' Kotchen

1
A Brief Introduction to Survey Research
MethodsMatthew J. Kotchen
2
Start With a Well-Defined Research Question

• Surveys are a means to an end
• Need to have objectives defined in advance
• Surveys are a tool to elicit information when
  there is no way of knowing it unless people are
  asked directly
• Attitudes/opinions
• Nonuse values
• Skeptics may be critical of response validity
• Need to know limitations and how crucial they are
  to research objectives

3
Quantitative versus Qualitative

• Quantitative surveys
• Survey instrument prepared in advance
• Closed-ended questions
• Statistical analysis
• Qualitative surveys
• Can be unstructured
• Open-ended questions
• Narrative description or case studies
• Can be coded for statistical analysis

4
Defining Your Population

• The population is the group from which you sample
  and administer the survey
• The research question should define the relevant
  population
• The population established the scope at which
  results should be applied
• Example Nonuse values for protecting the
  National Arctic Wildlife Refuge
• State of Alaska?
• Entire US?
• World?

5
Survey Types

• Mail surveys
• Telephone surveys
• In-person interviews
• Intercept or onsite surveys
• Internet surveys
• Choice of type will depend on several factors
• Budgetary limitations
• Nature of questions to be asked
• Time limitations

6
Sampling

• Sample size influenced by time and budget
  constraints
• Some examples
• Sample of convenience Good for story telling
  but not discovering truth
• Random Each individual in the population has a
  equal probability of being included
• Disproportionate stratification Disproportionate
  sampling within different population sampling
  units

7
Sampling and Statistical Analysis

• Sample protocol can have important implications
  for statistical analysis
• Two simple examples
• T-tests
• Equal variance (perhaps between treatments)
• Unequal variance (perhaps between strata)
• Probability weights
• Calculating a mean

8
A Few General Guidelines for Designing a Survey
Instrument

• Tradeoff between length and response rate
• Begin with easy questions and put more
  controversial ones last
• People who answer the first questions tend to
  finish the survey
• Every question should serve a purpose
• Think about statistical analysis of answers in
  advance!

9
General Types of Closed-Ended Question Formats

• Categorical When the answers are categories, and
  each respondent must fall into exactly one of
  them.
• Numerical For real numbers, like age, number of
  months, etc.
• Multiple-choice When there are a finite number
  of options (remember to instruct respondents as
  to the number of answers to select).
• Likert-scale To assess a person's feelings about
  something.
• Ordinal To rate things in relation to other
  things.

10
Standard Yes/No Questions

• Would you recognize a peregrine falcon if you saw
  one in the wild?
• Yes
• No
• Have you ever seen a wild peregrine falcon in
  Maine
• Yes
• No
• Not sure

11
Avoid Double Barreled Questions

• Do you love me and will you marry me?
• Yes
• No
• Do you think the state of Maine should implement
  an HCP to protect peregrine falcons and that 15
  resident pairs is sufficient for continued
  survival?
• Yes
• No

12
Relative Importance Questions

• In general, how important is the protection of
  Maines endangered species to you?
• Very important
• Somewhat important
• Not very important
• Not important at all

13
Fill in the Blank Questions

• How old are you?
• _____ years
• Including yourself, how many people live in your
  household?
• _____ people

14
Indicate all that Apply Questions

• To which, if any, of the following conservation
  or fish and wildlife organizations do you belong?
• Green Peace
• Local Rod and Gun Club
• Audubon Society
• National Wildlife Federation
• Sierra Club
• Nature Conservancy
• Other, please list______________________________
• I do not belong to any conservation or fish and
  wildlife organizations.

15
Likert-Scale Questions

• Example statements
• We are approaching the limit of the number of
  people the earth can support.
• Plants and animals have as much right as humans
  to exist.
• This lecture is boring.

• Response Categories
• Strongly agree
• Somewhat agree
• Neutral (unsure)
• Somewhat disagree
• Strongly disagree

16
Rank-Ordering Questions

• We would now like you to rank, in order of
  importance, the reasons listed above. Please rank
  from most important to least important the
  significance of the above reasons for your
  green-electricity enrollment decision. In each
  blank, write the letter of the appropriate
  response from above (a through e).
• 1 reason_____ (Most important)
• 2 reason_____
• 3 reason_____
• 4 reason_____
• 5 reason_____ (Least important)

17
Refining The Survey Instrument

• Focus Groups
• Interactive session with people after completing
  survey
• Open-ended feedback that can be useful for
  understanding how people interpret survey and
  questions
• Can diagnose potential problems
• Pretests
• Conduct survey on a small sub-sample or different
  group
• Can anticipate problems
• Useful for forcing to think about statistical
  analysis

18
Survey Implementation

• Details will depend on the survey type
• Mail survey example The Dillman Total Design
  Method
• Initial letter
• Letter with survey
• Postcard reminder
• Another letter with survey
• Final letter with survey

19
Response and Non-response

• A sample size is the number of initial
  contactsnot the number of completed surveys!
• Response rates are calculated as
• ( responses) / (sample size undeliverables)
• Response rates vary substantially
• They are very important because of sample
  selection bias
• Sample selection bias occurs when data come from
  a non-representative sample of the population and
  the reason for selection into the sample is
  related to the question of interest

20
The Really Fun Part

• Coding of surveys
• Data entry
• Analysis
• Learning something new about the world