Designing Questionnaires

1
Designing Questionnaires

• Chapter 11

2
Designing Questionnaires

• A questionnaire is a conversation
• Portrays to the respondent what is desired by the
  researcher
• Reports to the researcher information about the
  respondent
• Steps
• What information, type of data and type of
  questionnaire?
• Develop question content, format, phrasing
• Develop response format
• Sequence and layout questionnaire
• Pretest the questionnaire
• Revise the questionnaire

3
Steps in Constructing a Questionnaire
1. Review information requirements of problem,
opportunity, decision to be made, and so on. 2.
Develop and prioritize a list of potential
research questions to provide required information
. 3. Evaluate each potential research
question Can potential study participants
understand the question? Can potential study
participants answer the question? Will potential
study participants answer the question? 4.
Determine types of question to be
asked? Open-end questions Closed-end
questions 5. Decide on specific wording of each
question to be asked. 6. Determine questionnaire
structure. 7. Evaluate questionnaire.
4
Selecting Questions
5
Response Rate, Inaccuracy, Ambiguity

• Questionnaire Quality Affects Response Rate
• Response and Non-Response Errors
• Inaccuracy Errors in the Formulation stage
• Concurrent Inaccuracy (ex when the respondent
  says yes when in fact thinks no or vice
  versa)
• Predictive Inaccuracy (ex when the respondent
  says and means yes but later does the opposite)
  
• Ambiguity errors in the transmission stage.

6
Inaccuracy

• My result from
• Inability to respond memory error, ignorance
  error, misunderstanding.
• Unwillingness to respond
• time/cost
• invasion of privacy (use the randomized
  response)
• perceived losses to prestige

if the proportion of respondents who answered
Yes is .06, the proportion born in May
(determined from the Census of Population) is
.08, and the probability of answering each
question is .5, the estimated proportion who
answered Yes to the shoplifting question would
be
7
Ambiguity

• Occurs during the transmission through
• Communication
• Question as understood by respondent / answer as
  understood by interviewer
• Standardized interviewing (interpretation of
  questions is left up to respondent)
• Observation

8
Ambiguity

• Example question Government regulation is
  necessary to protect and improve the quality of
  life.
• Question Response
• Understanding Agree Uncertain Disagree Total
• MALE RESPONDENTS
• Good 447 10 332 839
• 53.28 1.19 39.57 100.00
• Fair/poor 128 11 33 172
• 74.42 6.40 19.19 100.00
• FEMALE RESPONDENTS
• Good 453 17 215 685
• 66.13 2.48 31.39 100.00
• Fair/poor 248 20 72 340
• 72.94 5.88 21.18 100.00
• TOTAL 1326 58 652 2036
• 65.13 2.85 32.02 100.00
• Several percentages can be calculated. For
  example, while 65.13 percent of the total sample
  agreed with the statement, 61.8 percent of the
  males and 68.4 percent of the females agreed. But
  the data reveals more specific interpretations
• Of the males whose question understanding was
  good, 53.2 percent agreed.

9
Basic Question Types
10
Forms of Questions and Answers

• Free-answer (open-ended text)
• - The shortest and least complex
• - The lowest probability of the question being
  ambiguous but the highest probability of the
  answers being ambiguous
• Multiple choice (select 1 of n or k of n)
• - The longest and most complex
• - First and last options tend to be more
  frequently used (systematic error )
• Dichotomous (select 1 of 2)
• - Between (1) and (2) as length and complexity
• 4. Rank order answers
• Constant sum answers

11
Forms of Questions and Answers

• Select 1/npick-1 The respondent is given a list
  of n options and is required to choose one option
  only.
• Select k/npick-k The respondent gets a set of n
  options to select from but this time chooses up
  to k options (k n).
• Select k1/n and Rank k2/k1pick and rank This
  question type is similar to pick-k, but in
  addition to selecting k1 options from a list of n
  options, the respondent is then asked to rank k2
  of those options selected.
• Select k1/k2/npick-and-pick Respondent is asked
  to select k1 options in Category 1 and k2 options
  in Category 2. Each option can be selected in
  only one of the two categories.
• Rank k/nrank In this question the respondent
  gets n options and is asked to rank the top k (k
  n).
• Integer Rating The respondent is asked to rate
  on a linear scale of 1 to n the description on
  the screen or accompanying prop card (for
  example, 1 for completely disagree to 5 for
  completely agree). Only integer responses are
  accepted.
• Continuous Rating This is similar to integer
  rating, except that the response can be any
  number (not necessarily an integer number) within
  the range (for example, 5.2 on a scale of 0 to
  10).
• Constant Sum The respondent is provided with a
  set of attributes (5, 10, etc.) and is asked to
  distribute a total of p points across those
  attributes.
• Yes/No This type of question entails a yes/no
  answer.
• Integerinteger- The respondent is asked for a
  fact that can be expressed in integer number
  form. A valid range can be provided for error
  checking. Example Age.
• Realreal- Similar to integer- except that the
  answer expected is in the form of a real (not
  necessarily an integer) number. Example Income.
  A valid range can be provided for error checking.
  
• Character The respondent types in a string of
  characters as a response. Example Name. No error
  checking is done on this type of input.
• Multiple Integer Ratings This question type is
  identical to integer-scale except that multiple
  questions (classified as options) can appear on
  a single screen. Each question is answered and
  recorded separately.
• Multiple Real Number Ratings This question type
  is identical to real-scale except that multiple
  questions (classified as options) can appear on
  a single screen. Each question is answered and
  recorded separately.

12
Forms of Questions and Answers

• Infinite Variations
• Single Choice
• Multiple Choice
• Likert Rating Scales
• Semantic Differential Scales
• Importance Performance Scales
• Rank Order Questions
• Constant Sum Questions
• Graphic Slider Scales http//www.quask.com/en
  /samples_opmeterspolls2.asp
• Graphic Choice Question http//www.surveyz.com/Tak
  eSurvey?id19966responseCheckfalse
• Audio Clip Questions http//www.surveyz.com/Ta
  keSurvey?id19682responseCheckfalse
• Video Clip Questions http//www.surveyz.com/Ta
  keSurvey?id13434
• Side by Side Questions
• Conjoint Analysis Questions

13
(No Transcript)
14
Ambiguity (continued)

• Question wording
• Framing questions
• Sources
• Question length
• Respondent unfamiliarity
• Ambiguity of one or more words in the context
• Two questions combined in one
• Lack of specificity
• Recognizing and reducing ambiguity
• Alternative question wording
• Pretesting
• Verification by observation

15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
Rules for Question Building

• Avoid loaded or leading words or questions. 
  Slight wording changes can produce great
  differences in results.  Could, Should, Might all
  sound almost the same, but may produce a 20
  difference in agreement to a question (The
  supreme court could.. should.. might.. have
  forced the breakup of Microsoft Corporation). 
  Strong words that represent control or action,
  such as prohibit produces similar results (Do you
  believe that congress should prohibit insurance
  companies from raising rates?)   Sometimes
  wording is just biased  You wouldn't want to go
  to Rudolpho's Restaurant for the company's annual
  party would you? 
• Misplaced questions.  Questions placed out of
  order or out of context should be avoided.  In
  general, a funnel approach is advised.  Broad and
  general questions at the beginning of the
  questionnaire as a warm-up (What kind of
  restraunts do you most often go to?).   Then more
  specific questions, followed by more general easy
  to answer questions (like demographics) at the
  end of the questionnaire. 
• Mutually non-exclusive response categories.
  Multiple choice response categories should be
  mutually exclusive so that clear choices can be
  made.  Non-exclusive answers frustrate the
  respondent and make interpretation difficult at
  best.
• Nonspecific questions.  Do you like orange
  juice?  This is very unclear...do I like what? 
  Taste, texture, nutritional content, Vitamin C, 
  the current price, concentrate, fresh squeezed? 
  Be specific in what you want to know about.   Do
  you watch TV regularly?  (what is regularly?).
• Confusing or unfamiliar words.  Asking about
  caloric content, bits, bytes, MBS, and other
  industry specific jargon and acronyms are
  confusing.  Make sure your audience understands
  your language level, terminology and above all,
  what you are asking. 
• Non-directed questions give respondents
  excessive latitude.  What suggestions do you
  have for improving tomato juice?  The question is
  about taste, but the respondent may offer
  suggestions about texture, the type of can or
  bottle, mixing juices, or something related to
  use as a mixer or in recipes.

19
Rules for Question Building

• Forcing answers.  Respondents may not want, or
  may not be able to provide the information
  requested.   Privacy is an important issue to
  most people.  Questions about income, occupation,
  finances, family life, personal hygiene and
  beliefs (personal, political, religious) can be
  too intrusive and rejected by the respondent.
• Non-exhaustive listings.  Do you have all of
  the options covered?  If you are unsure, conduct
  a pretest using the  "Other (please specify) 
  __________" option.   Then revise the question
  making sure that you cover at least 90 of the
  respondent answers.
• Unbalanced listings.  Unbalanced scales may be
  appropriate for some situations and biased in
  others.   When measuring alcohol consumption
  patterns,  One study used a quantity scale that
  made the heavy drinker appear in the middle of
  the scale with the polar ends reflecting no
  consumption and an impossible amount to consume. 
  However, we expect all hospitals to offer good
  care and may use a scale of excellent, very good,
  good, fair.  We do not expect poor care. 
• Double barreled questions.  What is the fastest
  and most convenient Internet service for you? The
  fastest is certainly not the most economical. 
  Two questions should be not asked.
• Dichotomous questions.  Make sure answers are
  independent.  For example the question "Do you
  think basketball players as being independent
  agents or as employees of their team?"  Some
  believe that yes, they are both. 
• Long questions .  Multiple choice questions are
  the longest and most complex.  Free text answers
  are the shortest and easiest to answer.  When you
  Increase the length of questions and surveys, you
  decrease the chance of receiving a completed
  response.
• Questions on future intentions.  Yogi Berra
  (Famous New York Yankees Baseball Player) once
  said that making predictions is difficult,
  especially when they are about the future. 
  Predictions are rarely accurate more than a few
  weeks or in some case months ahead. 

20
Other Considerations

• The question-order effects.
• Rotation of questions is recommended but with the
  caution to not destroy the logical sequencing of
  questions.
• The funnel approach
• From general to specific (easy to difficult)
• In a form of a contingency tree (minimizes the
  chance that a respondent is asked irrelevant
  questions)
• There is no pure optimum length but there is an
  operational optimum length.
• Comparability across different populations.
• Cross-cultural/National translate or not a
  questionnaire?

21
(No Transcript)
22
Summary

• Issues involved in asking questions to
  respondents
• Controlling for response error
• Difficult to separate questionnaire from mode of
  data collection
• Inaccuracy and Ambiguity
• Types of questions, questionnaire sequence

23
The questions listed below are being
considered for the questionnaire. Comment on
each, indicating whether you would leave the
question as it is or change it. If you think if
should be changed, rewrite it as you believe it
should be asked. (1) Do you or any of your
family drink fruit juices? Yes _____ No
_____ If yes (2) Is the juice drunk at a meal
or between meals or both At meal _____ Between
meals _____ Both _____ (3) Do you prefer frozen
or fresh juices? Frozen _____ Fresh
_____ (4) What advantages, if any, do you
believe using fresh juice has over using frozen
juice?   (5) What advantages, if any, do you
believe using frozen juice has over using fresh
juice?   (6) What brand or brands of juice do
you regularly buy? Dont know                    
       (7) On this card is a list of fruit
juices. Tell me which are your familys first,
second, and third choices. grape ____ tomato
____ lime ____ lemonade ____ orange ____ V-8 ___
other ____   (8) What is the last brand of juice
bought by your family? ____________ Dont know
24

• QUESTIONNAIRE
• 1. Under present law, families who run small
  businesses and farms are often forced to sell
  their holdings rather than pass them on to the
  next generation owing to the burden of estate
  taxes. Would you favor legislation to ease this
  burden?
• Yes _____ No _____
• 2. Of the following areas of federal spending,
  choose one in which you would prefer to make a
  budget cut
• a. Public welfare payments __________
• b. Public works projects __________
• c. Defense spending __________
• d. Foreign assistance programs __________
• e. Food stamps __________
• f. Education __________
• g. Other __________
• 3. Do you believe that charitable organization,
  such as churches and nonprofit hospitals
• should remain tax-exempt?Yes _____ No _____
• 4. Which one of the following would you choose as
  the most important in solving the energy shortage
  over the next 20 years?
• a. Solar/geothermal power development __________
  
• b. Nuclear power development __________
• c. Conservation of present sources of energy
  __________
• d. Expansion of domestic oil reserves
  __________

25
5. Which one of the following would you say is
the most important effort Congress could make to
prevent crime? a. Enact harsher penalties to
deter crimes __________ b. Reenact the death
penalty for certain crimes __________ c. Enact
restrictions on violence on television
__________ d. Increase funding for the
courts __________ e. Increase funding for law
enforcement agencies __________ f. Reform the
countrys prison system __________ 6. Most of
the economic indicators for the nation show
positive signs of a recovery. Unemployment is
down to 7.6 percent, personal incomes are up, and
the prime lending rate is down. a. Do you
feel that we are in a recovery? Yes _____ No
_____ b. Do you feel that the economy has
stabilized? Yes _____ No _____ c. Do you
expect inflation to increase? Yes _____ No
_____ d. Do you believe that unemployment
will stabilize Yes _____ No _____ Should
Congress finance more jobs producing programs
with tax revenues? Yes _____ No _____ 7. In
each of the following areas do you feel that
Congress efforts should be increased? a. Energy
research and development Yes _____ No _____ b.
Health care and insurance Yes _____ No _____ c.
Crime control Yes _____ No _____ d. Tax
reform Yes _____ No _____ e. Preservation of
the environment Yes _____ No _____ f.
Other Yes _____ No _____
26
(No Transcript)
27

• Time Scouthttp//www.time-scout.com/
• Time-Scout Monitor is an easy-to-use product
  that helps parents control the amount of screen
  time children spend with video games, TV and the
  computer. Plug a device, such as a television,
  into the Time-Scouts lock and let "the box" be
  the bad guy, not you.
• Time Scout is in need of a market segmentation
  study to determine
• Market Segments
• Likelihood of purchase
• Motivations for purchase
• Parenting style correlates for purchase
• Parent perceptions of need
• Reaction test for a video clip

http//new.qualtrics.com/SE?SIDSV_8AJb28FDCGdZWnO
SVIDProd
http//www.surveyz.com/TakeSurvey?id13434
Time Scout Monitor