Surveys, Questionnaire Design and Data Analysis'

1
Surveys, Questionnaire Design and Data Analysis.

• Dr Brendan Burchell
• Faculty of Social and Political Sciences
• Bb101_at_cam.ac.uk
• Wednesday 26th March 2008

2
Aims and Objectives

• To understand the theory regarding sampling, and
  inferences to populations
• To evaluate the pros and cons of administration
  methods
• Wording of Questions and benchmarking
• Determining cause and effect
• Analysing the survey data
• To consider the implications of these findings
  for your own research plans.

3
Sampling Theory

• As random samples increase in size, their
  properties resemble the population from which
  they are drawn more closely.
• SE of sample mean SD / SQRT N
• Note
• Only applies to random samples. Sample size
  cannot combat bias.
• Size of population unimportant.

4
Methods of Administration

• Face-to-face
• Time-consuming, costly?
• Worse social desirability biases?
• Post
• Low response rate
• Phone
• Who are you talking to?
• Internet?
• Improving.

5
Is representative sampling Important?

• To make exact predictions about the nature of the
  population (e.g. to predict an election)
• Essential
• To explore the relationships between variables in
  a sample and generalise to a population
• Desirable
• To explore mechanisms, processes, qualitatively
• ?

6
Methods of selecting random samples

• 1 Random number tables to select from populations
• 2 Random numbers and constant intervals
• (full list of population necessary)
• 3 Random numbers and procedures to infill gaps in
  population lists
• Stratified Samples (boosting small groups?)

7
Multi-Stage Sampling

• Eg. sampling to select organisations,
• then workplaces,
• then individuals
• Weighting may be necessary to re-adjust sample to
  be representative of population

8
Other Sampling Methods

• Telephone random digits
• Administrative lists
• Emails / Web
• Newspaper adverts
• Snowballing
• Theoretical Case-Studies
• Outliers

9
Non-Response

• Reduced Sample Size
• Increased Error
• Costly, but no loss of representativeness
• Increased Bias
• Much more serious

10
Reducing Non-Response

• Appropriate methods of administration
• E.g. face-to-face interviews
• Call-Backs
• Quality of postal questionnaire / covering letter
• Prizes / Rewards
• Relevance of topic

11
Question Wording

• Open Ended
• What are the significant threats to Health and
  Safety involved in this process?________________
• Closed Questions
• Which of the following are significant threats to
  the Health and Safety of this process?
• Human error, malfunction, fire, alcohol,
  inadequate training, operators falling asleep,
  earthquake, terrorism, faulty materials, employee
  sabotage, inadequate supervision, poor
  management, extreme weather, acts of god,
  computer errors, computer viruses,

12
What are you trying to find out?

• Objective facts.
• How many days of manufacturing were lost last
  year due to equipment failure?
• Who knows?
• Can the results be verified?
• Perceptions
• How satisfied are you with the reliability of
  your equipment?

13
Open vs closed questions

• Open
• Not constraining
• Need less piloting
• Get information in their own words
• Closed
• Less time-consuming to code
• Can include other, please specify
• Easier to lead?

14
Overcoming Social Desirability

• When did you last do a health and safety check?
• We all know that if you completed health and
  safety checks as often as the rulebook says you
  should, you wouldnt get anything else done. When
  did you last manage to do a health and safety
  check?

15
Pilot all questions

• Spot the error
• Do you work part time?
• Qualitative pilots?
• Quantitative pilots?

16
Who is the expert? You or the Participant?

• How long have you been doing this job?
• How many errors have you made in the last month?
• OR
• Doing your job, does the number of errors you
  make decrease over time?

17
Resources

• Plagiarism of questions is a virtue!
• See the Questionbank
• http//qb.soc.surrey.ac.uk/

18
Analysing the data from a survey

• Specialist packages, eg Minitab, SPSS
• Or Generic tools, eg MS Excel
• Analyse the data in stages
• Look at questions individually, check for coding
  errors, initial conclusions. Graph data.
• Start to look for evidence of differences between
  batches (i.e. relationships) by comparing means
• Only then, perhaps, check for statistical
  significance.

19
Guiding Philosophies of Data Analysis

• Are you a Judge Jury or a detective?
• Explore all aspects of data
• Not just Central Tendancy
• Mean, Median, Mode
• But Also
• Spread
• Shape
• Outliers