Surveys, questions and other cheap ways of getting data

1
Surveys, questions and other cheap ways of
getting data

• How to do stuff
• 1/14/02
• Michael Ramscar

2
Plan

• Surveys
• What is correlational research?
• Sampling a population
• Ways of collecting survey data
• How to formulate Good questions
• Scales
• Soothing strategies

3
Correlational Research

• The nature of correlation
• A numerical relationship between two variables
• Predictions
• Will score A covary with score B?
• Can be either negative or positive
• Presence of correlation allows prediction
• Prediction can be tested on samples

4
Correlational Research

• Does drinking beer make you appear cooler,
  funnier and sexier?
• How do you test this as a hypothesis?

5
Correlational Research

• 2 ways
• Experimentally
• Randomly assign people to groups
• Get one to drink no beer other drink 3 beers/day
• Measure the change in their coolness in relation
  to pre-test
• Conduct a survey
• Ask people to estimate how much beer they drink
• Ask people to rate them for coolness, sexiness
  and funniness
• Find the relationship between these two variables

6
Surveys Pros and Cons

• Advantages of survey research
• Can sample a large number of individuals
• Do not have to worry that the situation is
  artificial
• Surveys are often easy to administer

7
Surveys Pros and Cons

• Disadvantages of survey research
• Danger of biased surveys
• Sample may not be representative
• Questions may bias responses
• People may not answer truthfully/competently
• Asking questions may not be the right methodology
• Cannot make statements about causality

8
Correlation and Causality

• Correlation
• A numerical relationship between two variables
• One value need not cause the other
• The third variable problem - spurious
  relationships
• Does drinking beer make you cool?
• Perhaps uncool people are just too intimidated to
  go into bars
• A Mantra Correlation does not equal causation!

9
Sampling
The hard part of conducting a survey is getting a
representative sample
Population
Sampling Frame
Sample
10
Potential sampling problems

• Flawed surveys often reflect poor samples
• Sampling bias can enter in two places
• At selection of sampling frame
• Use of phone directories
• List of subscribers to a magazine
• At selection of a sample
• Use of an email survey
• Emails made to students on Friday and Saturday
  nights

11
Potential sampling problems

• Suppose you were interested in student attitudes
  to beer on campus
• Sampling frame might be the student email
  directory
• You could randomly sample names from this
• What is good and bad about this frame?

12
Types of Samples

• Nonprobability samples
• No guarantee that elements of pop can be included
• Will (almost never) be reflective of the
  population
• May be helpful for studying ideal cases
• May be helpful for studying specialized
  populations
• Accidental samples
• Whoever happens to be around...
• Surveys in the street
• Newspaper call-in polls
• Problems with accidental samples
• Built in bias
• Generalizability?

13
More non-probability samples

• Purposive sampling
• Studying a particular population
• May seek out special individuals
• Survey of types of employee
• May provide insight into what makes good vs. bad
• Generally assume that non-probability samples are
  biased
• True for many mail-in surveys

14
Subject Selection Effects

• Any study in which subjects self-select
  themselves for participation are potentially
  problematic
• Some types of selection are worse than others
• Those willing to participate for money
• Those interested in a particular topic
• Those with a particular viewpoint
• Those with a particular skill
• Problem with passive net surveys?

15
What to do?

• Probability samples
• In a probability sample, there is a better chance
  that the sample will reflect the underlying
  population
• Types of probability samples
• Simple random sampling
• Select N individuals from sampling frame
• Stratified random sampling
• Useful if there are key subgroups in data
• Suppose your students were half male and half
  female
• Divide sampling frame into males and females
• Select N/2 males and N/2 females
• Ensures that this aspect of the sample matches
  the population

16
Survey research designs
Cross-sectional designs Try to get a view of a
population at one time
17
Example

• Suppose we were interested in the number of
  left-handers in the population
• We could conduct a survey
• What if we found that left-handers make up about
  10 of the population up to age 60, and then
  there is a continuous decrease in percentage
• Might conclude that left-handers die earlier
• Other factors?

18
Successive Independent Samples

• Can be used to assess changes in attitude over
  time
• Must be sure that samples are comparable
• Want results to be due to changes in time, not
  changes in population

19
Longitudinal Studies

• Follow a single group of individuals over time
• Requires a lot of time and effort
• Subject mortality a problem
• Drop out -- sometimes literal mortality
• Most often people drop out for some reason
• Must ensure that there is not a systematic reason
  for the mortality
• Left handers again?

20
Summary So Far...

• Research is only as good as the sample you draw
• Must ensure the most representative sample
  possible
• Survey and correlational research cannot
  determine causal relationships
• Next - how do we go about it?

21
Conducting Surveys

• There are many ways to conduct surveys
• Through the mail
• In person via an interview
• Over the phone
• Via the internet
• some of all of the above can apply to this
• Each of these methods has advantages and
  disadvantages

22
Mail Surveys

• Mail surveys are used frequently
• Minimal effort needed to collect data
• Good for personal / embarrassing topics
• Respondents replies are anonymous
• May cut down on the amount of socially desirable
  responding
• Problems with mail surveys
• Respondents cannot ask for clarification
• A big problem is questions are often poorly
  worded
• No follow-up questions can be asked
• No control over the order in which a survey is
  completed
• Big potential for response bias

23
Self-selection in mail surveys

• Respondents in mail surveys always self-select
• Respondent must decide to do survey and mail it
• Controlling response bias in mail surveys
• Try to ensure a high return rate
• Over 50 is pretty good
• Response bias is less severe with return rates
  over 50
• Make survey attractive
• Include a pre-paid reply envelope
• Personal introductions
• Raffle a prize
• But

24
Personal Interviews

• Survey can be conducted in person
• Advantages
• High response rate
• Can control order of responding
• Respondent can ask for clarification
• Interviewer can ask follow-up questions

25
Personal Interviews

• Survey can be conducted in person
• Disadvantages
• There is a potential for interviewer bias
• May suggest the desired response (nonverbally)
• People may give socially desirable responses
• They may not want to express their true beliefs
  to the interviewer
• Difficult to ask about embarrassing topics
• No anonymity for the respondent

26
Telephone Surveys

• Doing surveys over the phone has some advantages
• More anonymity than for personal interviews
• Can control order of responding
• Respondent can ask for clarification
• Interviewer can ask follow-up questions

27
Disadvantages of telephone surveys

• People may not want to be bothered over the phone
• Particularly now that telemarketers often use
  surveys as a sales pitch (think also email spam)
• No control over surroundings
• Respondent may be doing a number of things at
  once
• Still the potential for interviewer bias
• People may provide socially acceptable responses
• They may not want to express their true beliefs
  to the interviewer

28
Constructing Surveys

• Kinds of questions
• Open-ended
• Range of allowable responses not determined in
  advance
• Respondents decide how much information to give
• Open-ended question can be difficult to score

29
Constructing Surveys

• Kinds of questions
• Closed questions
• A fixed set of responses is provided
• Respondent must pick one
• Easy to score
• May miss responses that respondent wants to give
• Limited in the scope of the answer that can be
  given

30
More on questions

• Mixed questions
• A fixed set of responses is provided
• An open-ended choice is given to cover any
  alternatives that may have been missed
• Still has a limited scope
• What meanings can bank have
• 1. A place to keep money
• 2. The edge of a river
• 3. A large snail
• 4. To lean an aeroplane into a turn etc...
• 7 Other. _________________

31
Response Scales

• For closed-ended questions, a list of responses
  is needed
• For some questions, the list is a set of options
• What meanings can bank have
• 1. A place to keep money
• 2. The edge of a river
• 3. A large snail etc.
• Many questions ask for a degree of preference
• Yes/No questions
• Easy response
• No information about strength of preference

32
Response Scales

• Likert Scales
• Labeled points on scale
• Strongly agree, Agree, Disagree, Strongly
  disagree
• Variations on these scales
• Scale from 1-5 with ends labeled
• How many values should be put on a scale?
• People are reasonably good with 72 options
• With more options, people sub-divide the space
• Ratings are not necessarily more accurate on a
  1-100 scale than they are on a 1-9 scale
• False feeling of accuracy

33
How many points?

• How many points
• The number of points you use depends on how much
  your subjects will be able to discriminate, and
  the size of the effect you want to measure.
• If the effect you're trying to measure is very
  small, then you'll need more points on the scale
• If you don't think people can meaningfully
  discriminate more than 3 levels of a variable,
  then you should only have 3 points - having more
  points would just add noise to your data.

34
Even or odd number of points?

• This makes a difference when you're measuring an
  effect in the middle of the scale
• An even number of points is good when you want to
  force people to make a decision that commits them
  to one side of a scale or another
• An odd number of points is good when you're
  interested in when people will take a middle of
  the road stance
• When will they say dont know

35
When might you not use a 7-point scale?

• 1. Measuring a small effect in the middle of the
  scale
• How good at maths are women generally?
• 1------2------3------4------5------6------7-----
  -8------9------10
• not at all..................................
  ...........................................extrem
  ely

36
When might you not use a 7-point scale?

• 2. Getting more info out of your subjects than
  they think they know
• Is this picture old or new?
• 1------2------3------4
• I'm sure it's old.........................I'm
  sure it's new

37
When might you not use a 7-point scale?

• 3. Finding out that people aren't sure
• Is this picture old or new?
• 1------2------3
• I'm sure it's old.................I'm sure
  it's new

38
When might you not use a 7-point scale?

• 4. Another example of forcing a binary decision
• Please rate whether each item below is masculine
  or feminine
• violin ..........M or F
• hammer.......M or F
• chair............M or F
• salt..............M or F

39
Writing Good Questions

• A survey (or test) lives and dies with the
  questions
• Unclear questions can confuse respondents
• Biased questions can skew results
• Vocabulary in questions should be clear and
  simple
• Never use two syllables where one will do

40
Writing Good Questions

• Questions should be clear and specific
• Is something wrong with these sentences?
• How do you feel about the grammar of these
  sentences?
• Please indicate if one of the following sentences
  is grammatical or not, and indicate why.

41
More on questions

• Edit questions for readability
• Use short questions
• Some words are ambiguous meaning that people
  attribute multiple meanings to them. Given this,
  do you think or not that more than one meaning
  could be given to the following words?

42
More on questions

• Do not assume that participants share your
  enthusiasm
• Get to the point
• Did I mention use short questions
• Help us understand memory - do this.
• Q-day!

43
Asking questions

• Participants may not always help you even when
  they want to
• Show me how you stop your children from doing
  naughty things

44
Asking questions

• Participants may not always help you even when
  they want to
• How to get round this?

45
Asking questions - stratagems

• Participants may not always help you even when
  they want to
• I bet you couldnt stop little Jimmy from doing
  something he really wanted to do

46
Asking questions - stratagems

• Participants may not always help you even when
  they want to
• Children especially may be too eager to please /
  to nervous to perform

47
Asking questions - stratagems

• Participants may not always help you even when
  they want to
• Children especially may be too eager to please /
  to nervous to perform
• Deflect the task
• Could you help this really dumb cookie monster to

48
Asking questions - stratagems

• Participants may not always help you even when
  they want to
• Do fat people eat more when they are stressed?

49
Asking questions - stratagems

• Participants may not always help you even when
  they want to
• What happens when people have folk psychological
  theories?
• Or when theories are famous?

50
Things to avoid

• Leading questions
• Questions that suggest the right answer
• People often recognise ambiguities -- or multiple
  meanings -- in sentences. Re-write the following
  sentences in a way that uses different words, but
  preserves their meanings?

51
Things to avoid

• Loaded questions
• Questions that are emotionally charged
• Competent readers can often spot ambiguities when
  reading. Does this sentence have more than one
  possible reading?

52
Things to avoid

• Double-barreled questions
• Packing too much into one question
• Say which aspects of the the following sentence
  are ambiguous and why you think they are.

53
Other things

• Include conditional information before the key
  idea in the question alter responses?
• If you were reading the following sentence, would
  you consider alternative meanings for it?
• Would you consider alternative meanings if you
  were reading the following sentence?

54
Other things

• Consider varying the polarity of the questions
• Some questions should be phrased negatively
• Other questions should be phrased positively
• Controls for a positivity or negativity bias
• Some people just like to say No (or Yes).