Questionnaire design:

1
Questionnaire design
Perhaps now you'll fill in my questionnaire?
2
How does the way you collect data affect the
data collected? How does question wording affect
peoples answers?
3
Practical exercise in questionnaire use You
will (a) design a questionnaire to answer
questions on some aspect of people's behaviour
(smoking, exercise, hobbies, etc.) (b)
administer it to a small sample (20-30
people) (c) produce an SPSS spreadsheet and
analyse your results (d) write a lab-report
summarising what you did and what you found
(using Excel for any graphs) (e) produce a
PowerPoint presentation on your findings.
4
Stages in questionnaire design
1. Formulate the research question(s) clearly.
2. Identify the population and sample.
3. Design the questionnaire think about question
wording questionnaire formatting mode of
administration data analysis.
4. Pre-test the questionnaire.
5. Administer the questionnaire, after it has
been revised in the light of (4).
5
Samples and populations
Sample a subset from a population (e.g.
first-year psychology students).
Population a complete set of things (e.g. all of
humanity).
For valid inferences to be made about a
population's characteristics, a sample must be
representative of its parent population (e.g.
similar in age, SES, IQ, etc.)
6
Methods of obtaining questionnaire data
Postal questionnaire Personal interview Phone interview Internet questionnaire
Cost Low High Moderate Low
Data quality Response rate Respondent motivation Interviewer bias Low Low None High High Moderate Moderate High Low Low Low None
Sample quality Low, unless high response rate High Moderate to high, especially with random digit dialling Moderate, but improving as Internet access widens
7
Postal Questionnaire Personal interview Phone interview Internet questionnaire
Possible interview length Short Very long Long Short
Ability to clarify and probe None High High None
Anonymity High Low Low Low
Dependence on respondents literacy High None None High
Control of context and question order None High High Depends
8
Goals of Questionnaire design 1. To obtain
facts about a person. 2. To obtain information
about their attitudes and beliefs. 3. To find
out what a person has done (behaviours).
9

• Questionnaire wording
• Should be exact.
• 2. Should be simple.
• 3. Avoid biased or emotive words.
• Schuman and Presser (1981) subtle changes of
  wording may influence responses.
• e.g. Should the Government allow public
  speeches by a Communist? produced 25 fewer
  pro-free-speech responses when allow was replaced
  with forbid.

10
4. Make all alternatives clear. e.g. Payne
(1951) "Do you think most manufacturing
companies that lay off workers during slack
periods could arrange things to avoid layoffs and
give steady work throughout the year?" 63 -
companies could avoid layoffs. 22 - couldnt
avoid layoffs. 15 - no opinion. Same question
plus phrase "or do you think layoffs are
unavoidable?" 35 - companies could avoid
layoffs. 41 - couldnt avoid layoffs. 24 - no
opinion.
11
5. Avoid the format "Some people say x do you
agree or disagree?" 6. Avoid unwarranted
assumptions. e.g. "What is your occupation?"
assumes person has a job. 7. Avoid
double-barrelled questions. e.g. "Should
immigrants be repatriated and their possessions
confiscated?" is two questions. 8. Avoid double
negatives. e.g. "Are you against a ban on
smoking?"
12
9. Consider the relative merits of open-ended and
closed-ended questions. Open-ended allow
unconstrained responses. e.g. "How do you travel
to the University?". May produce richly detailed
responses, but hard and tedious to
score. Closed-ended require choice from a
limited range of alternatives. e.g. "Do you
travel to the University by (a) bus, (b) car, or
(c) unicycle (tick one)". Easy to code, but prone
to bias.
13
Closed-ended questions must have (a) a balanced
response scale (b) mutually exclusive
categories (c) facilities for handling "don't
know" and "other" responses.
14
Rating scales The Likert Scale "Criminals
should be flogged".
Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree
1 2 3 4 5
Can be 5-. 7- or 9-point scale (doesn't make
much difference). Visual Analogue
Scale Strongly agree Strongly disagree
15
Problems with questionnaires about attitudes 1.
May not have an attitude - "doorstep
opinions". 2. Attitudes may be complex and
multi-dimensional. 3. Attitudes vary in
intensity. 4. Expressed attitudes may depend on
question wording, sequence and interviewer
effects.
16
Problems with questionnaires about behaviour 1.
Memory limitations - e.g. Chapman and
Underwood's (2000) study of drivers' memory for
accidents and near-misses. Can be counteracted by
(a) asking specific questions (b) asking for
birth date rather than age (c) using a
chronological format (d) re-interviewing. 2.
Response biases due to social desirability or
suspicion, especially for illegal or anti-social
activities. Can be counteracted by ensuring
anonymity.
17
What are the problems with these questions?
1. Do you visit fast food emporia regularly?
Over-complex wording. Exactly what does
"regularly" mean?
2. How many burgers do you eat per month?
Assumes you eat burgers.
3. Some people suggest that fast food is leading
to increased tooth decay and an increase in
obesity amongst teenagers in many parts of the
U.K. Do you agree?
Over-long. Includes two separate questions.
Implies you should agree with the views expressed.
4. Which of the following methods do you use to
travel to your fast-food outlet? (a) Bus (b) Car
(c) Bicycle
Does the questioner want you to choose only one
option, or can you choose more? No option for
responding "other" .
5. Do your children prefer to eat in KFC or
Macdonalds?
Assumes you have children.
18
Conclusions It's hard to design questionnaires
properly! Always be sceptical of survey results
-ask yourself Who were they collected by? Who
were they collected from? How were the questions
worded, exactly? Remember - 8 out of 10 cats
prefer Whiskas" has become "In tests, 8 out of 10
cats who expressed a preference, preferred
Whiskas".
19
Useful references Burgess, T.F. (2001). A
general introduction to the design of
questionnaires for survey research.
http//iss.leeds.ac.uk/info/312/surveys/217/guide
_to_the_design_of_questionnaires Taylor-Powell,
E. (1998). Questionnaire design asking questions
with a purpose. http//learningstore.uwex.edu/pdf
/G3658-02.pdf Another useful resource http//www
.cc.gatech.edu/classes/cs6751_97_winter/Topics/que
st-design/