Questionnaire

1
Questionnaire Interview Item-writing Part 1

• Dr Desmond Thomas, MA TESOL
• University of Essex

2
Useful References

• Denscombe, M., 1998, The Good Research Guide for
  Small-scale Social Research Projects, Open U.
  Press
• Foddy, W. 1993, Constructing Questions for
  Interviews and Questionnaires - Theory and
  Practice in Social Research, Cambridge Cambridge
  University Press
• Oppenheim, A.N., 1992, Questionnaire Design,
  Interviewing and Attitude Measurement, London
  Pinter

3
Questionnaire or interview? Whats the difference?

• Not a great deal, when questionnaire items are
  read out and responses filled in by the
  researcher
• But a world of difference between an anonymous
  structured survey questionnaire and an
  unstructured 1-to-1 interview
• QUESTIONNAIRE OR INTERVIEW WHICH SUITS YOUR
  RESEARCH?
• And what kind of interview?

4
A research questionnaire should

• Be designed to collect information which can be
  used as data for analysis this can be
  quantitative, qualitative or a mix
• Gather information by asking people about issues
  directly related to a research project
  (demonstrating validity)
• Consist of a written list of questions with
  each respondent reading an identical set and
  following identical procedures (reliability)

5
Questionnaires are useful when

• Used with large numbers of respondents in
  multiple locations
• Information required is straightforward
• Standardized data from identical questions is
  required

6
Questionnaire issues Sampling

• Is a sample representative of a total population
  really necessary?
• Or can the sample in a qualitative survey
  represent itself?
• If a representative sample is needed are there
  ready-made sampling frames?
• Will sampling be random or in clusters?
• How important is sampling size?

7
Quantitative Survey Issues Sampling 1

• If the target population is secondary school
  teachers of English in Bulgaria
• How large a sample?
• 30 or more people? 5-10?
• Problem of self-selection in survey responses.
  How to solve this?

8
Quantitative Survey Issues Sampling 2

• Sample A teachers belonging to five different
  schools (cluster sampling)?
• Sample B randomly selected?
• Sample C randomly selected within large clusters
  (eg the five schools)?
• Sample D Every 10th teacher on a national
  register (or other sampling frame)?

9
Questionnaire issues Piloting

• What should be piloted? Everything!
• Questionnaire layout, length, question types,
  question wording, order of questions, rubrics all
  need to be tested
• It is impossible to get things right at a first
  attempt second attempts, in turn, will need to
  be piloted

10
Advantages of questionnaires

• Supply a large quantity of data for a relatively
  low cost not labour-intensive
• Standardized pre-coded answers can enable speedy
  data collection, management and even analysis
• Eliminate interviewer bias at least to a
  certain extent
• Face validity

11
Questionnaire problem areas

• Low response rates
• Frustration for the respondent box-ticking can
  deter respondents when no answer seems
  appropriate
• Frustration for the researcher no scope for
  clarification of answers
• One chance only to get it right especially in
  terms of item wording

12
Creating web-based questionnaires

• Go to http//www.surveymonkey.com/
• Design and edit your survey
• Send the link by email to potential respondents
• Collect and analyse the data
• Note limitations of the free vs the paid version
  of this software

13
Issues for all types of questionnaire

• What makes a good questionnaire item?
• Why is it so difficult to formulate the precise
  questions that we need to ask in order to obtain
  the required answers?
• Why is it sometimes difficult to interpret the
  answers that we obtain?

14
Fundamental Problem 1

• It is almost impossible to ask a question
  without suggesting answers. The very fact that a
  question is asked implies that the researcher
  thinks the topic is of interest. Moreover, the
  way a question is asked inevitably reflects the
  researchers preconceptions. Unstated
  presuppositions always underlie a question.
  (Foddy 1993 53-4)

15
Fundamental Problem 2

• Q Which soft drink do you usually buy?
• (What is a soft drink and what isnt?
• How often is usually?
• What is understood by buy and by you?
• What if you buy more than one?)
• A Probably the first brand that comes to mind
• The question may not be understood as it is
  intended and results may therefore be invalid.
• (Starting point what information does it seek to
  elicit?)

16
Question questions such as

• How many journeys have you made on London
  Underground over the past month? 
• Do you enjoy going to coffee bars and restaurants
  in the evening?
• Do you regularly do your shopping in large
  supermarkets?
• The library facilities at Essex University are
  adequate for my needs.
• Strongly agree ( ) Agree ( )
• Disagree ( ) Strongly disagree ( )

17
And these as well

• Complete the following sentence.
• I enjoy studying at Essex because
• 2. Do you think that the library should do more
  to improve its facilities for Masters students?
• 3. How important is exercise as part of your
  daily routine?
• Very important ( ) Important ( )
• Not sure ( ) Unimportant ( )
•  

18
Framework for questionnaire design
1. Interviewer Encodes question, taking into account own purposes, presumptions/knowledge about the respondent, and perceptions of the respondents presumptions about self (ie the interviewer.) ? 2. Respondent Decodes question, taking into account own purposes, presumptions/knowledge about the interviewer, and perceptions of the interviewers presumptions about self (ie the respondent) ?
4. Interviewer Decodes answer, taking into account own presumptions/knowledge about the respondent and perceptions of the respondents presumptions/knowledge about self (i.e. the interviewer.) 3. Respondent Encodes answer, taking into account own presumptions/knowledge about the interviewer and perceptions of the interviewers presumptions/knowledge about self (i.e. the respondent.) ?
19
Design principles

• Choose the right instrument!
• Target information required first ? afterwards
  the right questions to ask. Required information
  depends on aims.
• Rigorously monitor all items
• Pilot and re-pilot the questionnaire
• Provide clear instructions and standard
  procedures (for reliability)

20
Open vs closed questions

• OPEN Answers have more depth but data processing
  is more complex.
• CLOSED Easy to process but many useful insights
  are lost and respondents can become irritated by
  being put in boxes. (Oppenheim 1992, p.115)

21
Some pitfalls to avoid

1. Respondents often answer questions when they
   dont really know the answer.
2. Respondents can deliberately not answer or answer
   incorrectly (ie they lie)
3. Respondents misinterpret questions
4. Small changes in wording can produce major
   changes in distribution of responses.
5. Attitudes and opinions can be unstable
   circumstances can change
6. Memory can be unreliable.
7. Relationship between what respondents say they do
   and what they actually do is not strong.
8. Cultural context has an impact on responses (eg
   attitude scale grades)
9. The format itself can affect responses
10. The order of questions and answers to earlier
    questions can affect responses.

22
NEXT WEEK !!!

• Questionnaire Item-Writing Workshop
• Coursebook Feedback Survey

23
Types of 1-to-1 interviews

• Structured with tight control over question
  format and possible answers. Like a face-to-face
  questionnaire
• Semi-structured with a clearly-defined question
  schedule but some flexibility and more open-ended
  answers
• Unstructured a narrative prompted by one general
  question, perhaps

24
1-to-1 Interviews are useful when

• Detailed information is needed from respondents
• A smaller number of respondents is acceptable
• Attitudes or feelings are investigated
• Sensitive issues are explored (?)
• Key players are targeted
• Ideas for a questionnaire need to be explored or
  fine-tuned

25
Advantages of 1-to-1 interviews

• Depth of information
• Insights eg gained from key informants
• Flexibility of formats
• Validity direct contact means that data can be
  checked for accuracy and for relevance
• High response rate
• Opportunity for targeted individuals to make
  their voice heard element of advocacy

26
Interview problem areas

• The interviewer effect and power relationships
  face-to-face contact can directly influence
  answers
• The instant position effect respondents feel
  obliged to supply answers of some sort
• Investment of time and resources
• Complexity of data analysis
• Reliability issues for multiple interviews

27
Reliability issues

• How can we know if respondents consistently
  understand the question? Or if they have
  misinterpreted it?
• How can we know if respondents are consistently
  telling the truth? Or if they are misleading the
  interviewer deliberately or at a subconscious
  level?

28
Features of interview design

• How many questions?
• In what order?
• How long/short?
• Format? (open qqs, statements etc.)
• Follow-up probes and/or prompts?
• Wording of questions?
• Mode of recording/analysing answers?

29
Conducting an interview

• Establishing a relationship/trust
• Explaining aims procedures
• Low key presentation of self and involvement
• Active listening
• Clarifications, probes and prompts
• Recording, transcribing and analysing

30
Focus groups

• Consist of a small group of people who are
  brought together by a trained moderator
  (usually the researcher) to explore attitudes,
  feelings and ideas about a particular topic or
  set of issues

31
Main characteristics of FGs

• A question schedule is prepared by the moderator
  to focus the discussion
• Place value on interaction within the group as a
  means of eliciting information a social
  experience
• No requirement to reach consensus
• Less pressure to hold a fixed point of view in
  fact opinions can develop

32
FGs are useful when

• Topics or issues need to be explored in depth, or
  attitudes revealed
• Typical groups of key informants are to be
  targeted eg language teachers

33
Advantages of FGs

• A less directive interview format
• Interviewer bias can be greatly reduced
• Provide a more natural social forum for the
  exchange of ideas. Interviewees are able to take
  the initiative not just respond
• The right to silence

34
FG problem areas

• Less directive means less predictable
• Huge amounts of data can be collected, much of
  which may be unusable
• Perception that FGs are a market research tool
  (Tony Blair New Labour)