How to Conduct Surveys to Better Understand IT Projects

1
How to Conduct Surveys to Better Understand IT
Projects

• Maureen Brown
• School of Government, UNC Chapel Hill
• March 23, 2004
• Reston, VA

2
Agenda

• Hour 1 Introduction to Survey Design
• Hour 2 Designing Questions to Assess SACWIS

3
Session Objectives
After completing this session, you should be
prepared to

• Identify the steps of the survey process
• Discuss the strengths and weaknesses of the four
  primary survey designs
• Identify the six primary cost factors of surveys
• Define the two primary types of questions
• Identify the six types of close-ended questions
• Describe the two ways to develop a sample
• Relate the four primary types of errors
• Discuss the two primary ethical issues associated
  with surveys
• Identify best practices for successful surveys

4
Steps of the Survey Process

• Define the Objectives of the Survey
• Determine the Sample Group
• Draft the Questions
• Test the Questions
• Administer the Questionnaire
• Analyze the Results

5
Define the Objectives

• Needs Assessment To solicit opinion about
  IT/data problems and solutions, requirements
  elicitation
• Marketing To evaluate nature and level of demand
  for a particular IT/data solution
• Evaluation To isolate the impact of IT/service
  Clinger-Cohen, ROI, C/BA, risk mitigation,
  performance

6
What information do you need?
Define the Objectives (cont.)

• What problem are you trying to solve?
• What information do you need to solve it?

7
Define the Objectives (cont.)
Precision is CRITICAL! Think in terms of
RESULTS! Preparing hypothetical results can help
to separate need to know from nice to know
8
Survey Designs

• Mail Out
• Online
• Telephone
• Face-to-Face

9
Survey Designs (cont.)

• 1. Mail out (including email)
• Typical procedure. . .
• First a personalized, advance notice letter
• Second 1 week later, a personalized letter, the
  survey, and a stamped self-addressed return
  envelop
• Third 4 to 8 days after the survey goes out, a
  follow-up postcard thanking those who responded
  and requesting a response from those who did not
• Fourth 3 weeks after the survey went out, to
  those who have not yet responded, a new
  personalized cover letter informing people, We
  have not yet heard from you with a replacement
  survey and stamped return envelop

10
Survey Designs (cont.)

• 2. Online
• Typical Procedure. . .
• First Build the survey
• Second Test the survey
• Third Post the survey
• Fourth Notify potential respondents of location
  and
• by-date
• Fifth Follow up with respondents

11
Survey Designs (cont.)

• Benefits Cost and Time
• Average cost of paper surveys 2.07USD per
  respondent
• Average cost of web-based surveys .88USD
• Problems Web page design, computer programming
• Source Schaefer, E. (2001). Web surveying
  how to collect important assessment data without
  any paper. (http//oiir.iit.edu/oiir/Presentations
  /WebSurveying/WebSurveying_20010424.pdf)
• Office of Information Institutional Research.
  Illinois Institute of Technology. Retrieved
  February 20, 2002. (http//oiir.iit.edu/oiir/Prese
  ntations/WebSurveying/WebSurveying_20010424.pdf)

12
Survey Designs (cont.)

• Provide instructions for the necessary computer
  actions, (i.e., erasing radio buttons, drop-down
  menus, and clearing open-ended questions).
• Provide instructions where the action is
  necessary, not at the beginning of the
  questionnaire.
• Do not make it necessary for respondents to
  answer each question before going on to a next
  one.
• Use a scrolling design that allows respondents to
  see all questions unless skip patterns are
  important.
• Make sure that all responses can be displayed on
  one screen, using double rows if necessary and
  navigational aids to achieve this.

13
Survey Designs (cont.)

• Use a welcome screen that is
• Motivating
• Emphasizes the ease of responding
• Shows respondents how to move to the next page
• Use a conventional format similar to a paper
  questionnaire.
• Consider brightness, fonts, navigation, and
  spacing.
• Limit line length.
• Respondents are less likely to skip words when
  lines are short.

14
Survey Designs (cont.)

• Use symbols or words to give notion of progress.
• Exercise caution with question structures that
  are known to have measurement problems in paper
  surveys, such as check-all-that-apply and
  open-ended questions.
• Remember
• Browser Difference
• Computer Experience Differences
• Data Security and Privacy
• Violates Laws of Randomization

15
Survey Designs (cont.)

• 3. Telephone
• 4. Face-to-Face
• Typical Procedure. . .
• First Send a notification letter indicating
  purpose and time length may also send a draft of
  questions that will be asked
• Second Schedule an appointment
• Third Be punctual!

16
Response Rates

• 7 44 for web surveys
• 6 - 68 for email
• Combination of mail and web has been most
  effective
• Cost benefits of web versus mail are not well
  known

17
Cost Factors

• How many people are available to work on the
  survey
• Experience level
• Time to produce results
• Consulting costs
• Facilities
• Budget

18
Types of Measures

• Subjective Versus Objective Questions
• Independent Variables
• Dependent Variables
• Control Variables (demographics)

19
Types of Questions

• Open-ended
• Breaking the ice in an interview when
  respondents' own words are important when the
  surveyor doesn't know all the possible answers
• Closed-ended
• Collecting rank ordered data when all response
  choices are known when quantitative statistical
  results are desired
• Likert-scale
• Multiple-choice
• Ordered
• Unordered
• Numerical
• Partially Close-ended

20
Types of Questions (cont.)

• Open-ended
• In your opinion, what software feature is needed
  most?
• Best for focus groups or other kinds of
  exploratory questioning
• Identifies range of answers that can be offered
  to respondents in more structured interviews
  later in the research process
• Rarely yields useful data for making reliable
  estimates about the percent of people with
  particular views or characteristics

21
Types of Questions (cont.)

• Likert-scale
• When you want to know respondents' feelings or
  attitudes about something  

22
Types of Questions (cont.)
Multiple-choice When you want respondents to pick
the best answer or answers from among all the
possible options
23
Types of Questions (cont.)

• Ordered responses
• listed below
• Asks respondents to evaluate problems/solutions
  independently of each other
• Measures respondents view of seriousness of each
  problem/solution
• Guides policy making by showing the extent to
  which one item is viewed as more important than
  another

24
Types of Questions (cont.)

• Unordered response
• Asks respondents to choose the single most
  important item from a predefined list of
  alternatives
• Provides insight on what is perceived as most
  important

25
Types of Questions (cont.)

• Partially Close-ended
• Same as close-ended with unordered response but
  allows respondents freedom to identify important
  items that are overlooked

26
Numerical For real numbers, like age, number of
months, etc.
27
Patterning and Sequencing

• Branching and skip options

28
Types of Samples

• Probability
• Purposive

29
Sample Size

• How much error can be tolerated?
• How large is the population?
• How varied is the population?
• How small is the smallest subgroup within the
  population that must be served?

30
Sample Size Rules of Thumb
31
Pretest Survey

• Is each question getting the information it is
  intended to get?
• Are all the words understood?
• Are the questions interpreted the same by all
  respondents
• Do all close-ended questions have an answer that
  applies to each respondent?
• Does the questionnaire create a positive
  impression that motivates people to respond?
• Are the questions answered correctly and in a way
  that can be understood
• Are skip patterns followed correctly?
• Does any part of the survey suggest bias on your
  part?

32
Types of Errors

• Coverage Error
• Sampling Error
• Measurement Error- Timing and Bias
• Nonresponse Error

33
Words of Wisdom

• Do Not Ask
• Open ended questions
• Questions when you can derive answers from other
  sources
• Something too precise
• Something embarrassing
• People directly about themselves
• Multiple concepts in one question (double
  barrelled)
• Leading questions

34
Words of Wisdom (cont.)

• Start with easy questions
• Go into the difficult ones
• End easy
• Group like concepts
• Alternate loading

35
Words of Wisdom (cont.)

• Group items on a questionnaire into logically
  coherent sections.
• Each question should follow comfortably from the
  previous question.
• Incentives have had mixed results.

36
Words of Wisdom (cont.)

• There are good and bad questions. The qualities
  of a good question are as follows
• Evokes the truth
• Asks for an answer on only one dimension
• Can accommodate all possible answers
• Has mutually exclusive options
• Produces variability of responses
• Follows comfortably from the previous question
• Does not presuppose a certain state of affairs

37
Words of Wisdom (cont.)

• One of the most common mistaken assumptions is
  that the respondent knows the correct answer to
  the question.
• It is important to look at each question and
  decide whether all respondents will be able to
  answer it.
• If there is any possibility that the respondent
  may not know the answer to your question, include
  a Don't Know" response category.
• Do not imply a desired answer.
• Do not ask the respondent to order or rank a
  series of more than five items.
• Do not use emotionally loaded or vaguely defined
  words.
• Do not use unfamiliar words or abbreviations.
• Use a glossary of key terms.

38
Words of Wisdom (cont.)

• Avoid A clear mandate, most, numerous, a
  substantial majority, a minority of, a large
  proportion of, a significant number of, many, a
  considerable number of, and several.
• Other adjectives produce less variability and
  generally have more shared meaning. These are
  Lots, almost all, virtually all, nearly all, a
  majority of, a consensus of, a small number of,
  not very many of, almost none, hardly any, a
  couple, and a few.

39
Words of Wisdom (cont.)

• Keep things simple, concise, and cleargeneral
  guidelines include
• Do not ask a question if the answer is obvious.
• For example, How would no change in the cost of
  raw materials affect your production?
• Avoid abbreviations and jargon.
• If you must use them, clearly define them.
• Ask yourself whether you need several questions
  or if you can get the information in one
  question.
• Do not try to cram too much into one question.

40
Words of Wisdom (cont.)

• Make your questions easy to understand.
• Make sure your sample population understands
  them.
• Consider whether respondents will have the
  information to answer your questions.
• Is it readily available?
• Will they know the answers?
• Will they have to research?
• Remember, if they have to look it up, they will
  probably skip the question or throw the survey
  out.

41
Words of Wisdom (cont.)

• Consider whether respondents will willingly
  provide the information.
• How personal is it?
• In cases where you need to collect very personal
  information, repeat your policy on anonymity.
• If you provide a list of answers, make sure all
  possible answers are present.
• Even with Yes and No questions, it may be
  necessary to include a neutral Undecided or Do
  not Know.
• Start a sequence with the question that is most
  comfortable to answer.
• This focuses the respondent.

42
Words of Wisdom (cont.)

• Do not mix I feel or I think questions with
  questions regarding facts.
• Keep factual and perception questions in separate
  groupings.
• Place sensitive demographic questions (such as
  age or income) at the end of the survey.

43
Considerations for Question Wording

• Is the question relevant? Is it consistent with
  survey goals?
• Does the question ask for need to know or nice
  to know information?
• What will be the value of a response? If 95
  percent say, Yes, would this affect decision
  making?
• Might the question elicit a vague answer? Make
  sure you ask directly for the information.
• Will respondents be able to answer the question?
  Will they have the information?
• Does the question lead to a particular response?
  (Is it a leading question?)

44
Considerations for Question Wording (cont.)

• If a set of answers is provided, are all possible
  answers listed? Is one side of the issue
  represented more than another?
• Does the question use negative phrases or words?
• Are positive adjectives or phrases used?
• If a scale is used for responses, is it balanced
  (for example, 1 to 5, with 3 being neutral)?
• Might the question antagonize the respondent? Are
  dead giveaway words used, such as all,
  every, or always?
• Are many demographic questions asked?
• Is potentially offensive language used (for
  example, sexist or racist wordings)?
• Is the question wordy?

45
Considerations for Question Wording (cont.)

• Were ambiguous words used words with more than
  one meaning?
• Is the question worded simply?
• Are abbreviations used?
• Does the question contain technical terms or
  jargon?
• Have adjectives been quantified and/or clearly
  defined?
• Does each question ask for one piece of
  information?
• Have multiple negatives been used?
• Does the question presume a previous situation or
  state of affairs?
• If responses are provided, are they mutually
  exclusive?

46
Ethical Considerations

• Politics, Politics, Politics!
• Incentives versus coercive and offensive
• Confidentiality
• Anonymity
• American Society for Information Systems

47
Best Practices

• Understand and avoid the four kinds of errors.
• Be specific about what new information you need
  and why.
• Choose the survey method that works best.
• Decide whether and how to sample.
• Write good questions that will provide useful,
  accurate information.
• Design and test a questionnaire that is easy and
  interesting to answer.

48
Best Practices (cont.)

• Put together the necessary mix of people,
  equipment, and supplies to carry out your survey
  in the necessary time frame.
• Code, computerize, and analyze the data from your
  questionnaires.
• Present your results in a way that informs your
  audience, verbally or in writing.
• Maintain perspective while putting your plans
  into action.

49
Best Practices (cont.)

• Pretest questions before they go online.
• Write an introduction for the survey which will
  bring cooperation from participants.
• Use filtering questions and have questionnaires
  appropriate for filtered groups.
• Divide long surveys into sections.
• Use open-ended questions sparingly.
• Use incentives to get people to respond.

50
For More Information

• Maureen Brown
• Phone [phone]
• Email brown_at_iogmail.iog.unc.edu
• http//www.surveymonkey.com/home.asp
• http//info.zoomerang.com/
• www.createsurvey.com
• www.knowledgenetworks.com
• www.harrisinteractive.com
• Local Universities
• Salant P. and Dillman D. How to Conduct Your Own
  Survey. New York, New York Wiley, 1994.
• http//www.joe.org/joe/2003august/tt6.shtml