Steps in the process

1
Steps in the process

• Information needs
• Questionnaire
• Evaluation and revision (repeat before each test)
• Testing and revision (repeat if possible)

2
Example of broad objectives

• To find out
• who uses the facilities
• whether they are satisfied with them and
• whether they think they are value for money
• Information is needed by age, ratepayer status
  and ethnic group

3
Example of detailed information need

• For each facility
• what proportion of the population (defined below)
  have used the facility in the past 12 months,
  where use means to visit or to drop off or pick
  up household member(s) aged lt18 who are visiting
  the facility.
• Use of information If low use of facility (less
  than 50 of population) encourage use or
  re-examine need for facility.

4
Steps in the process

• Information needs
• Questionnaire
• Evaluation and revision (repeat before each test)
• Testing and revision (repeat if possible)

5
Elements needed

• Title and date
• Reason for survey
• What will happen to their data
• Instructions for returning return-by date
• Contact for queries (name, phone, email)
• Any elements needed for processing

6
Layout

• A3 fold is better than two-sided or stapled put
  questions inside fold
• Have very visible numbers for navigation
• Two-column is good but dont put 2-column above
  1-column
• Signal the type of answer by the shape of the
  response space

7
Evaluation

• Does every information need have the questions
  necessary to meet it?
• Is every question necessary for an information
  need?
• Use the Guide to Good Survey Design to check for
  pitfalls
• Look for any other faults

8
Pitfalls with language

• Hard to understand just badly written
• Overly educated, complex or technical
  words/sentences
• Abbreviations that respondents may not understand
• Undefined terms or concepts
• Source A Guide to Good Survey Design, SNZ, 1995

9
Pitfalls with questions (1)

• Ambiguity it is hard to be sure what is being
  asked
• Questions that are too long
• Double or triple questions
• Unreasonable recall period
• Double negatives
• Source A Guide to Good Survey Design, SNZ, 1995

10
Pitfalls with questions (2)

• Questions that assume a state of affairs exists
• Social desirability or other bias
• Leading, loaded, unbalanced
• Questions which require information or a level of
  skill which respondents may not have
• Source A Guide to Good Survey Design, SNZ, 1995

11
Pitfalls with questions (3)

• Response options that are not mutually exclusive
• Response options that are not mutually exhaustive
• Response options that do not fit the question
• Source A Guide to Good Survey Design, SNZ, 1995

12
Problems with questionnaire (1)

• Crowded or untidy appearance
• Type hard to read
• Inconsistent use of terms
• Inappropriate title for the questionnaire or for
  sections
• Routing instructions which are incorrect or hard
  to follow
• Source A Guide to Good Survey Design, SNZ, 1995

13
Problems with questionnaire (2)

• Inconsistent layout
• Layout which is likely to cause recording errors
• Layout which is difficult for respondents or
  interviewers
• Illogical grouping or flow of questions
• Mistakes in numbering
• Source A Guide to Good Survey Design, SNZ, 1995

14
Steps in the process

• Information needs
• Questionnaire
• Evaluation and revision (repeat before each test)
• Testing and revision (repeat if possible)

15
Cognitive testingHow

• Researcher present during administration
• Think-aloud when possible
• Follow-up questions DURING or AFTER
  administration (hotly debated)
• At the end, very general follow-up questions

16
Cognitive TestingWhy

• Allows access to response process without decay
  (with think-aloud)
• Good diagnostic information from small numbers
• Cheap and easy to organise
• Allows flexibility about size of samples and
  timing of tests

17
Suggested language question

• What languages can you speak?
• ? too young to speak
• ? English
• ? Maori
• etc

18
First tested language questionCensus 1996

• Tick as many circles as you need to answer this
  question.What language(s) can you have a
  conversation in?
• ? too young to speak
• ? English
• ? Maori
• etc

19
Second tested question

• Tick as many circles as you need to answer this
  question.In which languag(s) could you have a
  conversation about most everyday things?
• ? too young to speak
• ? English
• ? Maori
• etc

20
Third tested question

• Tick as many circles as you need to answer this
  question.In which language(s) could you have a
  conversation about a lot of everyday things?
• ? too young to speak
• ? English
• ? Maori
• etc