Survey Research

1
Chapter 9

• Survey Research

2
Topics Appropriate to Survey Research

• All three types
• Descriptive
• Exploratory
• Explanatory
• Good for describing populations too large to
  observe
• Good for measuring attitudes
• Often misused to sell, persuade, justify
• Example push polls
• Surveys are NOT always questionnaires
• Questionnaires ARE one type of survey

3
Rationale

• Begin with the research question
• Asking is often more difficult than answering
• What is your research purpose?
• Descriptive, exploratory, explanatory,
  predictive?
• Nomothetic (aggregate group) or ideographic?
• What are the variables, dimensions, and expected
  relationships (if any)?
• What are possible indicators of the variables or
  dimensions?
• Look to the research, ask sample of intended
  population, ask experts, use intuition, etc.

4
Selection Issues

• Is a survey instrument the best tool?
• Are more direct methods are available and
  feasible?
• Questionnaires, phone, interview, Web?
• Resources to design, test, revise, retest etc.
  the survey?
• Factor analysis, validity and reliability
  analyses skills?
• Are there existing instruments that would
  suffice?
• Tradeoff of validity and reliability

5
Survey Limitations

• Survey instrument of last resort
• Abstraction issues
• Surveys examine abstract language, not behavior
• Question bias
• Identity issues

6
Survey Limitations

• Surveys measure
• Reality?
• Perception of reality directly?
• Self-reported perception of reality?
• Surveys can only measure responses to specific
  questions and question sets
• We want them to measure the latent construct(s)
• We should be sure not to confuse the measurement
  with the construct

7
Composite Measurements

• Concepts have multiple dimensions
• So, they have no clear, single indicator
• Accurate social measurements require examination
  of a comprehensive set of indicators
• Several indicators are usually combined into a
  single measure (scales and indexes)
• The measure should be unidimensional
• Scales measure effects, indexes can be causes or
  effects
• Therefore, data generated from items of a
  well-designed scale should be related. Not
  necessarily so with indexes

8
Variables and Dimensions

• E.g. Your two variables are religiosity and
  social support
• Separate each into dimensions (based on
  research)
• Religiosity
• Belief
• Practice
• Spirituality
• Social support
• Family
• Friends
• Workplace
• You need to design a methodology that measures
  the dimensions you identify

9
Qualitative or Quantitative?

• Qualitative unexplored areas, fewer
  participants, ideographic purpose
• Open-ended questions
• Higher validity, lower reliability and
  generalizability
• Quantitative focused expectation, more
  participants, nomothetic purpose
• Closed-ended questions
• Higher reliability and generalizability, lower
  validity

10
Question Design

• Good questions are
• Ones they can answer
• Ones they can answer easily
• Ones they are willing to answer
• Ones they understand consistently
• With a clear understanding of what type of answer
  is desired
• Ones that can be consistently administered
• Ones that are at the highest level of measurement
  possible

11
Incomplete Responses

• Incomplete responses, or responses that vary from
  intended design must be carefully considered
• Exclude them
• Substitute the mean
• Substitute a specific answer
• Analyze missing data to interpret the meaning

12
Review Survey Error

• Sampling error
• Too small a sample
• Coverage error
• Sample population is not representative
• Measurement error
• Inaccurate survey answers poor questions or
  survey construction
• Non-response error
• Large number of participants do not respond, and
  they share relevant characteristics

13
Guidelines for Asking Questions

• Choose appropriate question forms
• Questions or statements? Open-ended or
  closed-ended? Each has its advantages and
  disadvantages
• Open-ended
• Greater flexibility and validity
• More difficult must be coded
• Closed-ended
• Greater uniformity of responses
• Less ambiguity
• Easier to code and work statistically
• May not offer an accurate answer
• Answers must be exhaustive and mutually exclusive

14
Guidelines for Asking Questions

• Make items clear
• Language must not be ambiguous
• Avoid double-barreled questions
• Avoid and in questions creates two questions
• Respondents must be competent to answer
• Do they have the information?
• Are their opinions likely to be accurate?
• Can they remember?

15
Guidelines for Asking Questions

• Respondents must be willing to answer
• Are they apprehensive about sharing their
  knowledge?
• Questions should be relevant
• Do these people have an opinion? Do they care?
• Short items are best cause fewer problems
• Avoid negative items
• They tend to skip over the nots
• If unavoidable, make them STAND OUT
• Avoid biased items and terms
• Your question shouldnt suggest an answer
• See page 250 for examples

16
Bad Questions

• A survey of famous actors including Nicole Kidman
  and Tom Cruise revealed that they are in favor of
  strengthening hydrocarbon emission requirements.
  Do you agree?
• Do you enjoy hiking and skiing?
• Should the recent unfair ruling by the Supreme
  Court be reversed?
• How should biochemists reorganize their ethics
  committee?
• Have you ever thought of cheating on your wife or
  husband?
• Would you support a new law that would help unwed
  mothers?

17
Bad Questions

• Are you patriotic?
• When was the first time you saw a clown on TV?
• Why do you think the Republicans are supporting
  the new and confusing social security initiative?
• Which of the following represents the reason why
  you exceed the speed limit?
• Most people feel that the new law is a step in
  the right direction. What makes it such a good
  law?

18
Guidelines for Questionnaire Construction

• One question per line.
• Dont overcrowd the questionnaires pages
• Use contingency questions when necessary
• Make them very clear, easy to skip
• Format matrix questions so they are easily
  answered
• Avoid response sets alternate response direction
• Put the most relevant questions first
• Put the demographics last (unless theyre
  critical)

19
Guidelines for Questionnaire Construction

• Dont ask questions you wont use (unless you
  have a very good reason)
• Avoid confusing graphics, fonts, colors etc.
• Be aware of issues with ordering items.
• Include instructions for the questionnaire.
• Examples check one only and rank ordering
• Tell the respondents what you will do with the
  information
• Give them a way to get a copy when youre done
• Pretest all or part of the questionnaire.

20
Survey Facts

• Paper color doesnt matter
• Dont reduce font size to reduce of pages
• Stamps on envelopes dont help
• Use a cover letter or message
• Motivation is important
• Use multiple attempts
• Sponsorship raises compliance
• Make it seem important (letterheads, stationary,
  design)

21
Social Exchange Theory

• Perceived rewards, costs, trust
• Show positive regard, say thanks, ask for advice,
  appeal to group values
• Tokens can help (immediate only)
• Make the questionnaire interesting
• Allude to response of reference groups
• Avoid subordinating language

22
Social Exchange Theory

• Avoid inconvenience (supply what they need)
• Avoid embarrassment (e.g. watch language and
  question complexity)
• Make questionnaires short and simple
• Minimize requests for unneeded personal
  information
• Consider a foot in the door technique

23
Secondary Analysis

• Analyzing data collected by another researcher,
  often for a different purpose

24
Question Tips

• Which cell phone brand do you use?
• A.) Nokia. B.) Motorola. C.) Sony. D.) Other
• Make sure they can answer
• How much did you spend last year on gasoline
• _______
• Make sure they can answer accurately
• Agree or disagree Most of our clients are happy
  with our products
• Make sure they have the information
• What is your income?
• _________
• Make sure theyll be willing to answer

25
Question Tips

• Rank the following 20 items from high to low
• Make sure theyll be motivated to answer
• How much overtime do you do? (Very infrequent,
  infrequent, moderate, often, very often).
• Watch for inaccuracy due to vague quantifiers

26
Question Tips

• Open-ended questions
• Good for exploratory information, poor for
  accuracy and consistency
• Difficult to code
• Occupation? ____________
• Closed-ended questions
• Easier to code
• May not be inclusive
• Vague quantifiers are inaccurate, but often
  needed

27
Question Tips

• If you must use vague quantifiers
• How satisfied are you with the new POS?
• Delighted, mostly satisfied, neutral, mostly
  dissatisfied, terrible
• Ask a single question
• Understand that abstraction decreases accuracy
• 1-7 scale

28
Question Tips

• Use simple language, and as few words as possible
• Use complete sentences as questions
• How often do you send email each day?
• Poor Very often, often, sometimes, rarely,
  never
• Better Over 10/day, 6-10/day, 1-6/day, never
• Use the most precise quantifiers available
• How many times did you eat out last year?
• Poor _______ times
• Better None, 1-10, 11-20, 21-30, over 30
• Dont ask for too much specificity

29
Question Tips

• How satisfied are you with the new service?
• Less effective Very poor, Somewhat poor, Poor,
  Neither, Good
• Better Very poor, Poor, Neither, Good, Very
  good
• Use equal numbers of positives and negatives for
  scalar questions
• Agree or disagree The new software is stable
• Poor Strongly agree, agree, neither, disagree,
  strongly disagree
• Better Strongly agree, agree, neither,
  disagree, strongly disagree, No opinion
• Distinguish undecided from neutral

30
Question Tips

• Violence in schools is caused by
• Irresponsible parents, School policies,
  Television programs
• Better Parenting methods, School policies,
  Television programs
• Avoid question bias
• To what extent do you agree with this
  statement?
• Better To what extent do you agree or disagree
  with this statement?

31
Question Tips

• Avoid a Check all that apply question form
• Consider the statistical needs first
• Make sure your answers are mutually exclusive
• Age? 75
• Where did you learn about the new policy?
• At work, at home, on TV, on the radio
• (This is two questions location and media)
• Make navigation obvious and easy

32
Question Tips

• Do you favor or oppose prohibiting management
  from changing HR policies midyear?
• Asks respondents to say yes in order to say
  no
• Should management install a new computer system
  that includes updated OCR software?
• Double-barreled question
• What is your yearly income ______
• Better What is your income range? 20-40K,
  41-60K, Above 60K
• May be an objectionable question

33
Question Tips

• What of your salary is spent on vacations?
• Better About how much did you spend last year
  on vacations? Less than 500, 500-2000, Over
  2000
• Dont make respondents do math
• Watch for primacy/recency effects due to item
  order, or items with unordered categories
• Item order 1) Agree or disagree? Our
  organization should be able to protect its
  proprietary designs and 2) Agree or disagree
  Microsoft should be able to protect its
  proprietary designs vs. the reverse
• Unordered categories Which of the following
  strategic directions would you support?

34
Questions

• I am concerned about my grades. (Y/N)?
• Better I regularly share my concerns about my
  academic progress with my professor
• Ask about observable behaviors rather than vague
  perceptions
• The speaker is an expert in her field. (Y/N)?
• Better The speaker effectively responded to
  questions from the audience.
• Dont you think that the speed limit on campus
  should be reduced so we will be safer? (Y/N)
• Better The speed limit on campus should be
  reduced.

35
Question Tips

• Choose the first question carefully
• Easy, interesting, applies to all
• Dont start with a demographic question
• List answers vertically (check one)
• Like this ___
• And this ___
• And this ___
• Put special instructions where theyre needed,
  not at the front end
• Use white space effectively
• Pretest the instrument if possible. Look for
  ceiling and floor effects

36
Acceptable Response Rates

• 50 - adequate for analysis and reporting
• 60 - good
• 70 - very good
• Administering surveys
• Self-administered
• Mail, home delivery, Internet or phone
• Phone
• Using a researcher
• Face-to-face

37
Mail Surveys

• Poor completion rate
• Usually because its too much trouble
• Include a self-addressed stamped envelope
• Stamps or postage-paid mail?
• Use a letter of introduction (short)
• Include a real signature, and use the
  participants name, not dear sir or madam
• Use return graphs to show trends
• Use follow-up mailings
• Track with IDs

38
Survey Interviewing

• Usually result in a higher response rate
• Can note things not on the survey nonverbals
• Ethics?
• Interviewer must be neutral
• Not just words and ambiguity inflection, dress,
  other nonverbals, confidence in the material,
  friendliness, consistency, accuracy

39
Guidelines for Survey Interviewing

• Dress in a similar manner to the people who will
  be interviewed.
• Study and become familiar with the questionnaire.
• Follow question wording exactly.
• Record responses exactly.
• Probe for responses when necessary.

40
Training for Interviewers

• Discussion of general guidelines and procedures.
• Specify how to handle difficult or confusing
  situations.
• Conduct demonstration interviews.
• Conduct real interviews.

41
Telephone Surveys

• Advantages
• Money and time
• Safety
• Control over data collection
• Less concern over nonverbals (except trust)
• Disadvantages
• Resistance and discomfort
• Surveys that are really ad campaigns.
• Answering machines
• Ability to reach a representative sample

42
Internet

• Same general advice as paper
• Dont make them scroll
• Dont require plug-ins
• Minimize technical requirements and participant
  knowledge
• Recognize limitations
• Only those that have Internet connections can
  participate
• Only those with technological comfort may reply
• Multiple replies are possible from a single
  individual

43
Creating a Scale

• Clearly define the construct to be measured
• This is the conceptualization process
• Determine effects of the construct
• Remember that causes do not necessarily
  correlate, so they will be inappropriate for a
  scale
• Generate an initial item pool
• Use experts, population sample, other measures,
  intuition, etc.
• You need many initial items (100 or more is best)
• Separate useful redundancy from useless
  redundancy

44
Creating a Scale

• Determine the scale and item format
• Thurstone is very difficult and time consuming
• Likert or Semantic Differential more practical
• Decide on number of possible response categories
• Even vs. odd issue
• 5 vs 7 issue
• Consider adding validation items (from DeVellis,
  R.F. (2003) Scale development)
• Social desirability scale (Strahan Gerbasi,
  1972)
• Construct validity items

45
Creating a Scale

• Administer to a pilot sample
• Do reliability analysis
• Determine alpha (at least .70, but above .85 is
  desired)
• Realize that alpha is dependent on number of
  items
• Consider a factor analysis
• Determine the optimum length (number of items)

46
Strengths of Survey Research

• Useful in describing the characteristics of a
  large population.
• Make large samples feasible.
• Flexible - many questions can be asked on a given
  topic.

47
Weaknesses of Survey Research

• Can seldom deal with the context of social life.
• Inflexible in some ways.
• Subject to artificiality.
• Weak on validity.