Surveys and Questionnaires

1
Surveys and Questionnaires

• Jonathan Long

2
(No Transcript)
3
Some questions for you to answer

• What is the most desirable characteristic of a
  sample?
• Why might researchers be suspicious of findings
  from a phone-in poll?
• How big does your sample have to be?

4
Design Issues

• How the sample is drawn
• What sub-groups must be represented
• What comparisons need to be made
• Whether or not a control group is needed
• What variables will need to be measured (when and
  at what intervals)
• How these measures will be related to external
  events for example social, medical or other
  interventions

5
Ask Yourself

• What do I need to know?
• Why do I need to know it?
• Who is my sample meant to represent? Does it?
• Is the question in my head the same as the one
  the respondents read/hear?
• How am I going to analyse these data?
• Has good practice already been identified?

6
Social Survey Question Bank

• http//qb.soc.surrey.ac.uk

7
Draft Questionnaire assessing questions

• 1. Which age group are you in? 20-25 25-30
  30
• 2. What class would you say you came from?
• A/B C1 C2 D/E
• 3. What leisure activities do you take part in?
• 4. How often do you take part?

8
Draft Questionnaire

• 5. In a typical week during the past 12 months,
  how many hours did you spend on each of the
  following activities
• Walking, incl. walking to work, shopping and
  leisure
• Cycling, incl. cycling to work and during leisure
  time
• Gardening
• Housework such as cleaning, washing, cooking,
  childcare
• DIY
• Other physical exercise such as keep fit,
  aerobics, swimming, jogging and playing sport

9
Draft Questionnaire

• 6. Taking things as a whole, would you say that
  when people are born in this country,
  irrespective of where their parents are from,
  they should adopt British values and take part in
  British customs so that they can fit in better
  with the surrounding community?
• 7. Please indicate the extent to which you agree
  with the following statement. Too much is made
  of this black/white thing in sport everyones
  the same. Strongly agree
  Strongly disagree
• 8. Thinking about how you treat other people,
  would you say you are
• very racist
• quite racist
• a little bit racist

10
Example a much used questionnaireSF36 - Health
Status

• Physical functioning
• Role - physical
• Bodily pain
• General health
• Vitality
• Social functioning
• Role - emotional
• Mental Health

11
Sampling

• A random sample - really?
• Convenience sampling purposive sampling
• How many do I need?
• Absolute cf. relative size
• Sampling error and associated confidence

12
Another example Individual self-esteem
(indirect)
strongly agree ? strongly
disagree 1 I am a useful person to have
around 2 I can do things as well as most
people of my age 3 I have not got much to
be proud of 4 Sometimes I think I am no
good at all 5 I feel I am as good a person
as anybody 6 I feel I can not do anything
right 7 When I do something I always do it
well 8 I am not really getting anywhere
with my life 9 If I cant do something the
first time I keep trying until I can 10 I
avoid trying to learn new things when they look
too difficult for me 11 I give up
easily 12 I seem to be capable of dealing
with most problems that come up in my
life 13 I find it easy to make new
friends 14 I do not know how to handle
social gatherings
13
Summary of Things to Consider...

• How are you going to select your sample?
• How able are the respondents?
• How motivated are potential respondents to take
  part in the research project?
• What sorts of questions do you want to ask?
• What level of response is acceptable?
• Are you under any financial constraints?
• How much time have you allocated to collect your
  data?
• How widely available is appropriate computer
  hardware and software?

14

• Are you under any financial constraints?
• How much time have you allocated to collect your
  data?
• How widely available is appropriate computer
  hardware and software?

15
Validity

• Content Validity to establish that the items or
  questions are a well balanced sample of the
  content domain to be measured
• Concurrent Validity shows how well the test
  correlates with a well validated questionnaire of
  the same topic administered at the same time
• Predictive Validity shows how well the test can
  forecast some future criterion such as future
  examination attainment
• Construct Validity shows how good the link is
  between theoretical assumptions and an abstract
  construct such as intelligence or neuroticism

16
Ethical Issues

• Presser, 1994 (p.446)
• Every survey engages ethical concerns in terms
  of protecting respondents confidentiality.
  Similarly, all surveys engage ethical concerns in
  terms of informed consent, to which the main
  threat is deception

17
Reading Surveys

• Sample Design
• How was the sample drawn up?
• Sample Size
• How big is the sample?
• Refusal Rate
• Who did not respond?
• Questionnaire Design
• What questions were asked?
• Question Wording
• Neutral or leading words/phrases?

18
Limitations of Surveys?

• Cannot attribute cause
• Does not get at meaningful aspects of social
  action
• Takes the particular out of context
• Neglects human consciousness/goals/values
  assumes external determination
• Sterile ritualistic hypothesis testing
• Empiricist, fact collecting, lacking theory
• Some things just not measurable
• Restricted by structured nature of questionnaires
• Interesting questions reduced to incomprehensible
  statistics
• Scientistic, technistic, manipulative, repressive

19
E-questionnaires group exercise
Social researchers are increasingly looking to
electronic means of data capture. How do the
points discussed in this session apply to
e-questionnaires?
20
Related Readings
de Vaus, D. (2001) Research Design in Social
Research. London Sage. de Vaus, D. (2002)
Surveys in Social Research, 5th edition. London
Routledge. Fink, A. (1995) How to Ask Survey
Questions. Thousand Oaks Sage. Marsh, C. (1982)
The Survey Method. London George Allen
Unwin. Oppenheim, A.N. (1992) Questionnaire
Design, Interviewing and Attitude Measurement,
2nd edn. London Pinter. Presser, S. (1994)
Presidential address informed consent and
confidentiality in survey research, Public
Opinion Quarterly 58 pp. 446459.