Developing Establishment Survey Questionnaire Design Guidelines at the U'S' Census Bureau

1
Developing Establishment Survey Questionnaire
Design Guidelines at the U.S. Census Bureau

• Rebecca L. Morrison

2
Outline

• Introduction Background
• Development of the Guidelines
• Preliminary Guidelines
• Application of Guidelines
• Conclusion Next Steps

3
Introduction Background
4
Introduction Background

• U.S. Census Bureau
• leading source of quality datanations people
  and economy
• Tries to minimize burden of data collection
• Burden time, level of effort
• Reduce cognitive burden through visual design,
  structure of data request

5
Introduction Background (contd)

• Paper does not focus on specific question wording
  and order issues.
• Paper presents preliminary guidelines in
  questionnaire design formatting, layout,
  navigation, instructions, and data request
  phrasing and style.

6
Development of the Guidelines
7
Development of the Guidelines Census Bureau

• Agency standards for various survey quality
  issues
• No standard for design of questionnaires
• Economic surveys moving toward consistent look
  and feel

8
Development of the Guidelines ESMS

• Establishment Survey Methods Staff
• Group of survey methodologists
• In-house consultants to economic programs
• Noticed differences across questionnaires
• Survey programs work with ESMS on questionnaire
  design and pretesting
• Catalyst towards consistency

9
Development of the Guidelines Methods

• Questionnaire design principles from standard
  texts
• Special attention to literature on visual design
• Based on pretesting a variety of questionnaires
  with respondents

10
Development of the Guidelines Considerations

• List of design elements to consider
• Used to assess tradeoffs within specific survey
  conditions
• Not a cookbook

11
Preliminary Guidelines
12
Guideline A Text styles

• Be aware of text styles, and how they are used
  for emphasis within a survey instrument.

13
Guideline A Example

• 2004 Annual Survey of Local Government Finances

14
Guideline A Example
15
Guideline A Text styles

• These are favored by respondents
• Print item numbers in reverse-print bubbles,
  e.g., ?, ?, ?
• Print questions in bold black text.
• Print instructions in plain text or italics.
• Use an 8-point font or larger.

16
Guideline A Text styles

• Answer spaces
• Open, non-delineated vs. delineated
• Respondents do not seem to have a strong
  preference
• Be consistent

17
Guideline A Text styles

• Key codes / Punch codes
• De-emphasize processing codes for respondents
• Print in a darker shade of background color
• Example

18
Guideline B Reduce visual clutter. Use natural
navigational paths.

• Reduce clutter on the page. Use navigational
  paths and layouts that are natural and readable
  for respondents.

19
Guideline B Reduce visual clutter. Use natural
navigational paths.
20
Guideline B Reduce visual clutter. Use natural
navigational paths.

• Format in 1997 Economic Census confusing
• Lines were speedbumps to navigation
• Not clear how respondent was to navigate two
  adjoined columns
• Format changed for 2002
• Single column of questions
• Generally, a single column of response options

21
Guideline B Reduce visual clutter. Use natural
navigational paths.

• Two columns of questions
• Occasionally done when questions
• Are shorter
• Do not involve extensive instructions
• Do not ask for numerical information
• Survey of Business Owners (SBO)
• Information collected is categorical or ordinal,
  not interval or ratio

22
Guideline B Reduce visual clutter. Use natural
navigational paths.

• Spread out the questions across more pages
• Use bulleted lists
• More open space more user-friendly

23
Guideline C Instructions

• Place instructions close to questions, or
  incorporate them into questions. Where possible,
  convert instructions into questions.

24
Guideline C Instructions

• Instructions convey specifications, intent of
  question
• Respondents tend not to pay attention to
  instructions, or only look when they think they
  need them

25
Guideline C Instructions

• 2 Goals for instructions
• Eliminate, or reduce, amount of instructions
  located separate from question
• Place instructions/information where it is most
  needed

26
Guideline C Instructions

• Convert instructions into questions so
  respondents attend to them
• When content critical to correct interpretation
  of later questions
• When it helps clarify/correct reported data
• Example (2004 Medical Expenditure Panel Survey)

27
Guideline D Avoid sentence fragments and key
words

• Phrase data requests as questions or imperative
  statements, not as sentence fragments or key
  words.

28
Guideline D Avoid sentence fragments and key
words

• Types of data requests
• Question (question word, question mark)
• Imperative statement (report, enter, add)
• Sentence fragment (key words, no verb)

29
Guideline D Avoid sentence fragments and key
words

• Example, 2002 Economic Census
• Is this establishment physically located inside
  the legal boundaries of the city, town, village,
  etc.?
• Versus
• Type of municipality where this establishment is
  physically located

30
Guideline D Avoid sentence fragments and key
words

• Cognitive interview study respondents prefer
  questions over sentence fragments
• Survey methodology grad students questions more
  effective, imply respondent has to do something

31
Guideline E Simple vs. complicated questions

• Ask additional, simple questions, rather than
  fewer, more complicated ones.

32
Guideline E Simple vs. complicated questions

• Length and complexity of question affects how
  long it takes for respondent to understand it
• More complicated questions might also be more
  likely to be double-barreled
• May be easier for respondents to answer series of
  shorter, simpler questions

33
Guideline E Simple vs. complicated questions

• Example 2002 Industrial Research Development
  Survey

34
Guideline E Simple vs. complicated questions

• 2006 RD Survey

35
Guideline F Use matrices judiciously

• Use matrices judiciously, and consider the likely
  respondents background when deciding whether or
  not to use them.

36
Guideline G Be consistent!

• Items should be numbered in the same way
• Bold and italicized print should always mean the
  same thing
• Navigation path should remain constant
• Data requests should be in the same form
• Variation can be confusing for respondents

37
Application of Guidelines
38
Guideline A Text Styles

• Work in progress among estab surveys at Census
  Bureau
• Economic Census automation required consistent
  design features
• Plain text questions
• Italics instructions, definitions,
  include/exclude lists
• Bold Separating items into categories, sums of
  added lines, emphasis within questions

39
Guideline A Text Styles

• Survey of Business
• Owners
• Questions plain
• Instructions bold
• and/or italics
• Emphasis within
• questions bold
• and/or italics

40
Guideline B Reduce visual clutter. Use natural
navigational paths.

• Survey of Residential Alterations and Repairs
  (SORAR)
• Two columns to one column
• Removed or lightened lines
• Use of bulleted lists
• More open space

41
Guideline B Reduce visual clutter. Use natural
navigational paths.

• Old
• SORAR

42
Guideline B Reduce visual clutter. Use natural
navigational paths.

• New SORAR

43
Guideline B Reduce visual clutter. Use natural
navigational paths.

• Two columns
• Survey of
• Business
• Owners

44
Guideline B Reduce visual clutter. Use natural
navigational paths.

• BEA quarterly foreign
• direct investment (FDI)
• Complex navigational
• path
• Crowded text

45
Guideline B Reduce visual clutter. Use
natural navigational paths.

• BEA form
• revised

46
Guideline C Instructions

• Economic census continues to use separate
  instruction sheets and booklets
• Two questionnaires moved many instructions to
  questionnaire
• 2007 Commodity Flow Survey
• BEA quarterly FDI form

47
Guideline C Instructions
48
Guideline D Avoid sentence fragments and key
words

• Mixed success in application across surveys

49
Guideline D Avoid sentence fragments and key
words

• Commodity Flow Survey uses mix of questions and
  imperative statements

50
Guideline E Simple vs. complicated questions

• Example
• 2002 Survey
• of Business
• Owners

51
Guideline E Simple vs. complicated questions

• Example 2007
• Survey of Business
• Owners
• (still in cognitive
• pretesting)

52
Guideline F Use matrices judiciously

• Example
• BEA
• quarterly
• FDI

53
Guideline F Use matrices judiciously
54
Conclusion Next Steps
55
Conclusion Next Steps

• Guidelines are preliminary, based on principles
  of questionnaire design
• Further refinement is necessary before they can
  be applied to establishment surveys at Census
  Bureau

56
Conclusion Next Steps

• Other agencies/organizations may develop
  guidelines that work better for their environment
• There is no cookbook for questionnaire design

57
Thank you!

• Rebecca L. Morrison
• 301-763-7595
• Rebecca.L.Morrison_at_census.gov

58
Preliminary Guidelines

• A Be aware of text styles, and how they are used
  for emphasis within a survey instrument.
• B Reduce clutter on the page. Use navigational
  paths and layouts that are natural and readable
  for respondents.
• C Place instructions close to questions, or
  incorporate them into questions. Where possible,
  convert instructions into questions.

59
Preliminary Guidelines

• D Phrase data requests as questions or
  imperative statements, not as sentence fragments
  or key words.
• E Ask additional, simple questions, rather than
  fewer, more complicated ones.
• F Use matrices judiciously, and consider the
  likely respondents background when deciding
  whether or not to use them.