General Issues in Survey Design

1
General Issues in Survey Design

• Dr Desmond Thomas, MA TESOL
• University of Essex

2
Useful References

• Denscombe, M., 1998, The Good Research Guide for
  Small-scale Social Research Projects, Open U.
  Press
• Foddy, W. 1993, Constructing Questions for
  Interviews and Questionnaires - Theory and
  Practice in Social Research, Cambridge Cambridge
  University Press
• Oppenheim, A.N., 1992, Questionnaire Design,
  Interviewing and Attitude Measurement, London
  Pinter

3
The quantitative-qualitative divide

• Q-Q Divide Numbers vs words
• Traditionally some methodologies (inc surveys)
  labelled quantitative while others (eg
  ethnography) are considered qualitative
• Methods are divided similarly eg structured
  questionnaires (quantitative)
• This view of research is now challenged

4
Descriptive vs analytic

• A more useful distinction for surveys
• Descriptive surveys count entire populations
  (as in a census) or representative samples
• Analytical surveys explore relationships between
  different variables. These can be causal but are
  more likely to explore associations. Often less
  emphasis on representativeness. Linking ideas is
  the aim.

5
Questionnaire or interview? Whats the difference?

• Not a great deal, when questionnaire items are
  read out and responses filled in by the
  researcher
• But a world of difference between an anonymous
  structured survey questionnaire and an
  unstructured 1-to-1 interview
• QUESTIONNAIRE OR INTERVIEW WHICH SUITS YOUR
  RESEARCH?
• And what kind of interview?

6
A research questionnaire should

• Be designed to collect information which can be
  used as data for analysis this can be
  quantitative, qualitative or a mix
• Gather information by asking people about issues
  directly related to a research project
  (demonstrating validity)
• Consist of a written list of questions with
  each respondent reading an identical set and
  following identical procedures (reliability)

7
Questionnaires are useful when

• Used with large numbers of respondents in
  multiple locations
• Information required is straightforward
• Standardized data from identical questions is
  required

8
Questionnaire issues Sampling

• Is a sample representative of a total population
  really necessary?
• Or can the sample in a qualitative survey
  represent itself?
• If a representative sample is needed are there
  ready-made sampling frames?
• Will sampling be random or in clusters?
• How important is sampling size?

9
Questionnaire issues Piloting

• What should be piloted? Everything!
• Questionnaire layout, length, question types,
  question wording, order of questions, rubrics all
  need to be tested
• It is impossible to get things right at a first
  attempt second attempts, in turn, will need to
  be piloted

10
Advantages of questionnaires

• Supply a large quantity of data for a relatively
  low cost not labour-intensive
• Standardized pre-coded answers can enable speedy
  data collection, management and even analysis
• Eliminate interviewer bias at least to a
  certain extent
• Face validity

11
Questionnaire problem areas

• Low response rates
• Frustration for the respondent box-ticking can
  deter respondents when no answer seems
  appropriate
• Frustration for the researcher no scope for
  clarification of answers
• One chance only to get it right especially in
  terms of item wording

12
Creating web-based questionnaires

• Go to http//www.surveymonkey.com/
• Design and edit your survey
• Send the link by email to potential respondents
• Collect and analyse the data
• Note limitations of the free vs the paid version
  of this software

13
Types of 1-to-1 interviews

• Structured with tight control over question
  format and possible answers. Like a face-to-face
  questionnaire
• Semi-structured with a clearly-defined question
  schedule but some flexibility and more open-ended
  answers
• Unstructured a narrative prompted by one general
  question, perhaps

14
1-to-1 Interviews are useful when

• Detailed information is needed from respondents
• A smaller number of respondents is acceptable
• Attitudes or feelings are investigated
• Sensitive issues are explored (?)
• Key players are targeted
• Ideas for a questionnaire need to be explored or
  fine-tuned

15
Advantages of 1-to-1 interviews

• Depth of information
• Insights eg gained from key informants
• Flexibility of formats
• Validity direct contact means that data can be
  checked for accuracy and for relevance
• High response rate
• Opportunity for targeted individuals to make
  their voice heard element of advocacy

16
Interview problem areas

• The interviewer effect and power relationships
  face-to-face contact can directly influence
  answers
• The instant position effect respondents feel
  obliged to supply answers of some sort
• Investment of time and resources
• Complexity of data analysis
• Reliability issues for multiple interviews

17
Reliability issues

• How can we know if respondents consistently
  understand the question? Or if they have
  misinterpreted it?
• How can we know if respondents are consistently
  telling the truth? Or if they are misleading the
  interviewer deliberately or at a subconscious
  level?

18
Features of interview design

• How many questions?
• In what order?
• How long/short?
• Format? (open qqs, statements etc.)
• Follow-up probes and/or prompts?
• Wording of questions?
• Mode of recording/analysing answers?

19
Conducting an interview

• Establishing a relationship/trust
• Explaining aims procedures
• Low key presentation of self and involvement
• Active listening
• Clarifications, probes and prompts
• Recording, transcribing and analysing

20
Focus groups

• Consist of a small group of people who are
  brought together by a trained moderator
  (usually the researcher) to explore attitudes,
  feelings and ideas about a particular topic or
  set of issues

21
Main characteristics of FGs

• A question schedule is prepared by the moderator
  to focus the discussion
• Place value on interaction within the group as a
  means of eliciting information a social
  experience
• No requirement to reach consensus
• Less pressure to hold a fixed point of view in
  fact opinions can develop

22
FGs are useful when

• Topics or issues need to be explored in depth, or
  attitudes revealed
• Typical groups of key informants are to be
  targeted eg language teachers

23
Advantages of FGs

• A less directive interview format
• Interviewer bias can be greatly reduced
• Provide a more natural social forum for the
  exchange of ideas. Interviewees are able to take
  the initiative not just respond
• The right to silence

24
FG problem areas

• Less directive means less predictable
• Huge amounts of data can be collected, much of
  which may be unusable
• Perception that FGs are a market research tool
  (Tony Blair New Labour)