Monday 10th October 2005

1
Monday 10th October 2005

• Questionnaire Design

2
Survey Analysis

• Is most often used where individuals are the unit
  of analysis
• Individuals, known as Respondents provide data by
  responding to questions.
• The instrument that is used to gather data is
  called a questionnaire. Questionnaires collect
  standardised information, to be used in analysis

3
Questions should ALL be relevant

• Either
• Measures of the dependent variable
• Measures of independent variable
• Measures of test variables
• Background measures

4
Questions elicit information on

• Behaviour what people do or have done
• Beliefs what people believe to be true
• Attitudes what people would like to happen
• Attributes peoples characteristics.

5
Questions

• May be open or closed
• An open-ended question is one where respondents
  formulate their own answers.
• A closed or forced response question is one
  where a number of alternative answers are
  provided and respondents (are forced to) choose
  from these.

6
Open-ended questions

• Allow respondents greater freedom to answer what
  they want and express themselves.
• But they may be less easy to use after theyve
  been collected.
• They also require that people take time to
  construct their responses.
• Example How do you feel about UK policy in
  Iraq?

7
Closed/forced-response questions

• Are useful because they
• are quick to respond to (and so will encourage
  people to respond)
• are easy to code
• allow for easy comparison across cases
• dont discriminate against the less
  talkative/less literate respondent

8
Closed Response Questions

• Some questions are by nature closed (without any
  forcing of responses).
• For example, yes/no questions,
• e.g. Do you have any children?
• Or questions which must have a numerical
  response,
• e.g. How many children do you have?
• How old are you?

9
Closed Response Questions

• With other questions you need to manipulate the
  wording and responses in order to construct a
  closed question. These are forced response
  questions.
• (examples to follow)
• Note A lot of thought must be put into
  developing the responses (possible answers) to
  forced-response questions. Each set of responses
  must be
• Exhaustive every case should fit into one
  category (You dont want to shoe-horn people into
  false positions)
• Mutually Exclusive every case should fit into
  no more than one category

10
Types of forced response questions

• 1.Rating scales (Likert-style formats)
• Providing people with statements and asking them
  to indicate how strongly they agree/disagree
  (either verbally or on a diagram)
• e. g. Question Tony Blair lied to the British
  people about Iraq
• Response Format A Check the box that best
  describes your position on this
• Strongly agree Disagree
• Agree Strongly Disagree
• Cannot decide
• Response Format B Mark the point on the line
  that indicates how strongly you agree or disagree
  with this statement (- is total disagreement
  is total agreement 0 is no opinion)
• - 0
• Response Format C Circle the number that
  reflects the degree to which you agree with the
  statement (1 is Strongly disagree 9 is strongly
  agree 5 is no opinion)
• Strongly Agree Strongly Disagree
• 1 2 3 4 5 6 7 8 9

11
Types of forced response questions

• 2. Semantic differential formats
• Choosing adjectives (or short descriptions) to
  represent two extremes and asking respondents to
  put a mark between the two
• e.g. How would you describe Tony Blair?
• Friendly 1 2 3 4 5 6 7
  Unfriendly
• Liar 1 2 3 4 5 6 7 Honest
• Intelligent 1 2 3 4 5 6 7 Stupid

12
Types of forced response questions

• 3. Checklists
• A list of items, from which the respondent can
  choose one or more (note the list neednt be
  exhaustive unless you need it to cover all
  possibilities)
• e.g. Which of these things did you do over the
  weekend? (check box)
• ? Participate in sport ? Visit art gallery
• Read Feminist theory ? Go to pub
• Watch TV ? Visit family

13
Types of forced response questions
4. Ranking formats Respondents given a list of
alternative answers, but rather than choosing
between them, they are asked to rank their
importance e.g. I chose this job because (rank
the following reasons in terms of how important
they were in your choice) Rank The pay
is good The work is something
that interests me The company offers
good security of employment The
pension plan is good It is easy
for me to get to and from work The
hours and holidays are good I
believe in the aims of the organisation
I could not find any other work
note it may not make sense for people to
rank more than three or four answers after that
ranking gets pretty unreliable.
14
Types of forced response questions
5. Attitude Choices (rather than
agree-disagree) e.g. Theres been a lot of
recent press coverage about Kate Mosss drug
habits. What do you think the companies for whom
she is a public face (such as HM, Chanel and
Burberry) should do? They should fire her
immediately They should keep her on so
long as she gets drug treatment They
should keep her on regardless I dont
care I dont know
15
Vignettes

• Vignettes are stories that are told to give
  context to a question.
• They are useful because they ensure that everyone
  has the same context in their heads when thinking
  about an issue.
• Particular attributes of peoples lives or the
  situation in a vignette can be systematically
  altered to examine the importance of these
  attributes.

16
Vignettes
In the following example the sex of the character
is altered to see whether people react
differently to a man or to a woman. Susie/John
has a part time job as a teacher and has had this
job for 5 years. Her/His mother-in-law, who
lives by herself and is 80, has become very ill
and needs someone to look after her full-time.
Susies husband/Johns wife is a manager in a
large company and earns enough to keep both of
them. Susie/John income from teaching is not
enough to pay for a full-time carer for Her/His
mother-in-law. Associated Questions can be
open-ended i.e. What should Susie/John do? Or
closed i.e. respondents could choose between a
set of alternative solutions.
17
Guidelines for Asking Questions

• Choose appropriate question forms.
• Make items clear (unambiguous).
• Avoid double-barreled questions (Not
  agree/disagree with the statement its
  important to go to college and develop your
  intellectual interests).
• Respondents must be competent to answer. (Not
  how old were you when you first ate ice-cream?)

18
Guidelines for Asking Questions

• Respondents must be willing to answer.
• Questions should be relevant (respondents should
  care/have opinions about the questions you ask
  them).
• Short items are best (more precise /
  unambiguous).
• Avoid negative items as people may not notice the
  negative and so mis-read these (Not I think
  people who have plastic surgery are not vain
  agree or disagree).
• Avoid biased items and terms. (Not Do you
  engage in unsafe sexual intercourse?)

19
The order of questionsThe order should make
sense and encourage people to complete the
questionnaire. Make sure that the order is
logical that questions on the same topic are
grouped and that there is flow from one to the
next.

• Some Guidelines
• Start with easily answered, factual questions
• Do not start with demographic questions like age,
  marital status (as these are often seen as more
  personal)
• Make sure that the first questions are clearly
  related to the stated purpose of the survey
• Go from easy to more difficult questions
• Go from concrete to abstract questions
• Open ended questions should be kept to a minimum
  and put near the end of the questionnaire
• Group questions into sections
• Use filter (contingency) questions
• When using positive and negative items in a scale
  mix them up so that you avoid people always
  giving the same answer
• If the questionnaire is long try and use
  different question formats to keep it interesting.

20
Contingency (filter) Question Format
21
Contingency Table
22
Instructions to Skip
23
Checklist for Questionnaire Construction

• Decide what you need to know
• Ask yourself why you need to know this
• Begin wording questions
• Check you have included all the questions you
  need
• Check wording of each question
• Sort questions into order
• Write out instructions to be included (at start
  and with particular questions)
• Consider layout and appearance
• Decide on your sampling
• Pretest all or part of the questionnaire

24
Instructions

• Remember to include instructions.
• These should be concise, but they should still be
  clear.
• General instructions
• Introduction to the purpose of the questionnaire,
  assurance of confidentiality, information about
  how respondent was chosen, and where to return
  questionnaire (if relevant)
• Section instructions
• Describe what each section will contain, perhaps
  explaining why it will be useful (i.e. why you
  want this type of information).
• Question instructions
• Explain how to answer each question (i.e. how
  many responses can be ticked)
• Go to instructions
• Are for where a filter is used.

25
Piloting (or Pre-testing)
Ideally questionnaires will be developed through
talking to people about the sorts of responses
they may give to particular questions. However
when you have what you think is a workable
questionnaire it is a very good idea to test it
out on a few people similar to those you will
survey. These tests should probably be blind (the
people who youre talking to wont know that its
a test). In a pilot you are testing whether the
questions are doing what you think they will
do. For example ? whether people understand
them ? whether there are some questions to which
a very large percentage of people give a
non-response (i.e. say dont know). ? whether
you have two questions that basically measure the
same thing ? You will also be testing whether
there were particular questions that make
respondents feel uncomfortable, or that were
difficult to ask.
26
Three Types of Surveys

• Self-administered Questionnaires
• Mailed Survey (or email)
• Group Survey (i.e. in a classroom)
• Interview Surveys (face to face)
• Telephone Surveys (including CAT interviewing)

27
(No Transcript)
28
Response Rate

• The proportion of people who are qualified to
  take part in the survey (part of the sample) who
  actually participate. i.e. if you receive 75
  surveys back from a sample of 100 people your
  response rate is 75
• You must keep track of the response rate.

29
Acceptable Response Rates

• 50 - adequate for analysis and reporting
• 60 - good
• 70 - very good

30
Guidelines for Survey Interviewing

• Dress in a similar manner to the people who will
  be interviewed.
• Study and become familiar with the questionnaire.
• Follow question wording exactly.
• Record responses exactly.
• Probe for responses when necessary.

31
Training For Interviewers

• Discussion of general guidelines and procedures.
• Specify how to handle difficult or confusing
  situations.
• Conduct demonstration interviews.
• Conduct real interviews.