Survey or poll: study in which questions are asked to the population or parts thereof

1
Survey or poll study in which questions are
asked to the population or parts thereof

• Survey study to acquire an overview
• usually of the population or parts thereof
• Large-scale field study

2
Genesis

• In Roman time censuses
• In Napoleons time registry of births, deaths
  and marriages
• 19th century studies on social inequality or
  poverty
• (Le Play, Marx, Kemper)
• 20th century studies on social cohesion or
  public opinion
• (Weber, Fromm)

3
Purpose and definition

• Description and explanation of phenomena
• through questioning
• a relatively large number of respondents
• regarding a relatively large number of
  characteristics
• Aim
• Description of phenomena
• Explanation (or possibly prediction) of phenomena
• Examples
• pre-election polls, public opinion polls, market
  research

4
From problem definition to method
problem definition
choice of strategy
survey
in-person, by telephone, self-administered
5
Variants

• In-person interview
• face-to-face
• typically visit interview
• Telephone interview
• oral interview by telephone
• Self-administered questionnaire
• mail survey
• internal school examination, whole-class
  completion
• Computer-assisted survey in person or by
  telephone

6
Does it matter which variant you use?

• In-person interview
• usually best response
• best quality, but expensive and time-consuming
• more social desirability bias
• Telephone interview
• less good response
• quick and relatively cheap but still good quality
• brief interviews
• less social desirability bias
• Self-administered questionnaire
• response is problematic
• slow, cheap method
• less social desirability bias
• no interviewers required

7
Limitations of telephone interviews

• Short duration limitation of the number of
  questions
• 15 minutes 60-70 questions
• no visual or written aids
• brief, clearly formulated questions
• with answers provided or implied in the questions

8
Sampling

• Random sampling
• each element in the population has a known chance
  of being included in the sample survey
• this chance can also be calculated
• Non-random sampling
• Any other type of sample survey
• If you intend to make generalizing statements at
  the end,
• then always opt for random sampling.
• If the survey is not a random sample survey, all
  statistical generalizations will be dubious.

9
Variants

• Single random sampling
• simply take samples randomly (with or without
  replacement!)
• Systematic sampling
• take samples systematically from a list (card
  index, computer list, registry office)
• Stratified random sampling
• select subpopulations, then draw samples from
  each subpopulation
• Cluster sampling
• draw clusters, then units per cluster
• Multi-stage sampling
• draw by multiple, successive stages
• 1 random sample of addresses (postal services) or
  telephone numbers
• 2 random sample of people within population thus
  acquired

10
Drawing up a questionnaire

• Determine what type of information is required
• General questions, research questions,
  observation questions
• List existing valid and reliable questions
• Think of additional valid and reliable questions
• Prevent formulation errors
• Question formats
• Logical design
• Pretesting the questionnaire
• Planning and organization of fieldwork

11
Determine type of information

• What is required?
• experiences
• knowledge
• opinions or set of opinions attitudes
• behaviour
• background information

12
Prevent formulation errors systematic errors
and measurement errors

• Language use no difficult words
• No negative questions
• No questions or answers with adverbs of
  indefinite frequency
• No long questions, no motivations in questions,
  gt
• No ambiguous questions
• No complex questions more than one topic
• No questions containing debatable facts
• No questions with possibly different standards
• No leading questions
• No assuming questions
• Avoid items that may be interpreted differently
• The answer options should match the questions
• Answers should be entirely mutually exclusive

13
Language use no difficult words, no jargon

• Do you think the Dutch police have an image
  problem?
• What do you think of the total performance of the
  police? (Police Monitor Population 1997)
• Do you think the Dutch police should pay more
  attention to persistent offenders (draaideur
  criminelen)?
• The respondents determine what is difficult
• Remember that people with an academic background
  make up only 15 of the Dutch population

14
Some remedies (to prevent difficult words)

• Use an introduction/description
• You may have heard of persistent offenders, who
  immediately commit another offence as soon as
  they have been released from prison.
• Do you think the police should pay more attention
  to this group of criminals?

15
No negative questions

• Do you not think there should be more police on
  the streets?
• Do you not think the police should take harder
  action?
• Easy to improve
• Do you think the police should take harder
  action?
• Yes, I do
• No, I dont
• Depends on the case

16
No questions or answers with adverbs of
indefinite frequency

• How often have you been victim of one of the
  following criminal actions? (often, sometimes or
  never)
• Bicycle theft
• Threat
• Breaking and entering, etc.
• Easy to improve
• In the past year, how many times have you been
  victim of ...
• Bicycle theft? number of times ..
• Threat? number of times ..
• Breaking and entering? number of times.

17
No questions or answers with vague quantifiers

• Do you drink alcohol?
• No, never/ Yes, sometimes/ Yes, regularly
• How often did you drink alcohol in the past 4
  weeks? Number of times.
• How many glasses do you drink on average?
• 1-4 glasses/5-9 glasses/10 glasses or more
• Better
• Do you drink alcohol ? No, never/ Yes, sometimes
• On how many weeknights did you drink alcohol this
  week? nights
• How many glasses on average did you drink on
  these nights? Number
• On how many nights over the past weekend did you
  drink alcohol? .. nights
• How many glasses on average did you drink on
  these nights? Number

18
No long questions, no motivation in the questions
no ambiguous questions

• How often did you report an offence or crime to
  the police, even though you think they will not
  do anything about it?
• Easy to improve
• How often did you report an offence or crime to
  the police?
• Number of times . gt if more than 0 gt
• To what extent are you satisfied about the way
  in which the police dealt with your information?
• satisfied/dissatisfied/dont know

19
No motivation in questions no ambiguous
questions

• Avoid using long, complex sentences linked by
  words such as
• Due to
• Because
• While
• Although
• As
• So that

20
No complex questions not more than one topic in
one sentence

• Do you think there are many young people in this
  neighbourhood who, generally speaking, cause much
  nuisance?
• Have you yourself or has anyone you know ever
  been victim of..?
• In this neighbourhood, young people from various
  cultures (Dutch, Moroccans, Turks, Surinamese,
  Antilleans, etc.) get along quite well,
  generally. Do you agree or disagree with this
  statement?
• (Police Monitor Population 1997)

21
Some remedies (to avoid complex sentences)

• Split up the complex question into separate
  questions
• How many percent of the population in your
  neighbourhood are people younger than 25? Can you
  make an estimate in percentages?
• How many of these young people cause much
  nuisance? Can you make an estimate in
  percentages?

22
No questions with facts that are debatable or
may be assumed to be generally known

• Rising crime in the Netherlands requires harder
  action.
• Does everyone agree that crime is increasing?
• Is crime actually increasing?
• Does the general public know about this?
• Disappearance of traditional values needs to be
  stopped.
• Does everyone agree that traditional values are
  disappearing? What does everyone consider
  traditional values?
• UNLESS..the problem definition relates to assumed
  or perceived developments!

23
No questions referring to possibly different
standards

• Do your children receive a strict upbringing?
• What is a strict upbringing? Does the strictness
  vary in accordance with the age and sex of the
  children?
• Do you sometimes drink beverages containing
  alcohol?
• If yes, how many on average do you drink each
  time?
• a lot, many, not too many, few, very few
• These type of questions are usually formulated
  too generally.
• So be more specific and do not provide vague,
  multi-interpretable answer options!!

24
Leading questions

• Wouldnt you say its about time the police do
  something about it?
• This question leads the respondent to a yes
  answer plus the question contains a negation.
• The answer options may also suggest something.
• How much alcohol do you drink per day? Is that
• none/1-3 glasses/4-6 glasses/6-10 glasses

25
Using leading questions on purpose sensitive or
taboo issues

• Almost everyone has gone through a red traffic
  light at least once in their lives.
• Have you?
• If yes, how often did this happen in the past
  month?
• People have very different opinions about the
  arrival and presence of ethnic minorities, by
  which we mean people who originate from other
  countries such as Turks, Moroccans, Surinamese
  and Antilleans. We would like to know what you
  think of them. To what extent do you agree or
  disagree with the following statements on these
  minorities?
• Most Turks cannot be trusted.
• Most Turks behave themselves very well.
• Balanced scales with alternately
• positive and negative items avoid response
  set!!!!

26
No assuming questions

• I How often do you take your children to a
  party?
• R ???? I dont have any children!!!
• Easy to improve
• Do you have any children? (yes or no)
• Do you have any children under 18? (yes or no)
• How often do you take your children to a party?
• (always, often, sometimes, rarely, never)

27
Avoid multi-interpretable items point questions
or items

• D66 is a political party that appeals to me.
• if someone says no, it may not only be because
  they think D66 is too left-wing, but also because
  they think D66 is too right-wing.
• Do you think the Dutch police is doing a useful
  job?
• someone says no, they make too few arrests
• Someone says no, they make too many arrests
• These types of questions are often formulated too
  generally. So be more specific!!

28
The answer options should match the questions

• Nijmegen Youth Monitor
• Do you like it at school?
• Never/sometimes/often/always
• Are there any fellow pupils with whom you do not
  get on very well?
• None/a few/many/all fellow pupils
• EASY TO IMPROVE

29
Provide complete set of answer options

• I Do you often/sometimes/rarely feel unsafe?
• R ??? I never feel unsafe!!
• Better
• Do you ever feel unsafe? Yes/No
• Do you feel unsafe often/sometimes/rarely?

30
Use mutually exclusive answer options

• For how long do you watch TV each day?
• Less than one hour, between 1 and 2 hours, or
  between 2 and 3 hours?
• Better
• 60 minutes or less, 61-120 minutes, 121-180
  minutes?
• Or
• Can you estimate how many minutes a day you watch
  TV at weekends? Number of minutes
• Can you estimate how many minutes a day you watch
  TV on weekdays? Number of minutes

31
Question formats, answer optionsanalysis options

• Direct or indirect questions
• nominal or ordinal answer options gt
• measuring models that presuppose nominal or
  ordinal data
• Propositions or Likert items
• metric or quasimetric answer options gt
• measuring models that presuppose metric data

32
Logical design

• Start off with a few nice and easy questions
• Place questions that belong together in one block
• The design should be logical for the respondents
• Difficult and sensitive issues begin with
  careful introduction
• do not raise these issues at the very beginning
  or very end of the list

33
Pretesting the questionnaire

• Always pretest the questionnaire
• with a small number of real respondents
• Consult colleagues (fellow students) first, as
  well as possible users of the results

34
References

• H. t Hart e.a. (1998) Onderzoeks-methoden.Meppel
  Boom.
• R. Porst (2000) Zur Formulierung von
  Fragebogen-Fragen in de ZUMA-How-to-Reihe, nr. 2.
• P. Salant D.A. Dillman (1994) How to conduct
  your own survey. New York Wiley.
• E. Taylor-Powell (1998) Questionnaire design
  asking questions with a purpose. Wisconsin
  University of Wisconsin Extension.

35
Useful Internet sources

• Social Survey Section, American Statistical
  Association
• http//www.stat.ncsu.edu/info/srms/srms.html
• (English) brochures on how to design and
  administer surveys
• Homepage Don Dillman
• http//survey.sesrc.wsu.edu/dillman/
• papers on mail anf Internet surveys