Questionnaires as Instruments

1
Questionnaires as Instruments

• Questionnaires
• Most frequently used survey instrument
• Are scientific instrument just like an MRI
  machine
• Quality of the questionnaire determines accuracy
  and precision of the measurements
• Preferences and attitudes
• most often these are measured with self-report
  scales
• participants respond on rating scales
• Usually Likert scales from 1-7 0r 1-9
• For example, an assessment of emotional
  well-being might include the following items
• My mood is generally
  positive.
• Strongly disagree 1 ------ 2 ------ 3 ------ 4
  ------ 5 Strongly agree
• I am often
  sad.
• Strongly disagree 1 ------ 2 ------ 3 ------ 4
  ------ 5 Strongly agree

2
Questionnaires as Instruments

• Psychologists measure different types of
  variables
• Demographic (e.g., age, gender, race,
  socioeconomic status)
• To check for response bias
• To categorize data i.e. differences between young
  and older adults
• Use a well know template such as US Census
• FIGURE 5.5 Although ethnic background is an
  important demographic variable, accurately
  classifying people on this variable is not an
  easy task.

3
Reliability of Self-Report Measures

• Reliability refers to the consistency of
  measurement.
• Assessed by Test-retest reliability
• Administer measure two times to the same sample.
  Individuals scores should be consistent over
  time.
• A high correlation between the two scores
  indicates good test-retest reliability (r gt.80)
• r value ranges from 0 (no correlation) to 1
  (perfect correlation)

4
Reliability of Self-Report Measures

• How do we improve reliability?
• Many similar items on the same construct improves
  reliability
• Multiple questions that focus on the same
  construct (operational definition)
• For example several questions on quality of sleep
• When the sample is diverse relative to the
  construct quality of sleep
• Some participants have poor sleep while others
  have great sleep
• Testing situation is free of distractions and
  instructions are clear.

5
Reliability of Self-Report Measures

• Reliable measures make us more confident that we
  are consistently measuring a construct within a
  sample, but are reliable measures truthful?
• I could reliably measure your research methods
  knowledge by measuring your height. The taller
  you are, the better your score.
• Is this a truthful or accurate measure of
  research methods knowledge?
• Note We do not expect some measures to produce
  consistent scores over time.
• When people change on a particular variable over
  time
• we expect the measure to have low test-retest
  reliability
• Scores on a math exam before and after taking a
  math course

6
Validity of Self-Report Measures

• Validity refers to the truthfulness of a measure.
• A valid measure assesses what it is intended to
  measure.
• Construct Validity Does an instrument measure
  the theoretical construct (concept) it was
  designed to measure? Such as sleep quality

7
Validity of Self-Report Measures

• Construct Validity
• This seems like a straightforward question, but
  consider widely used measures of intelligence
  which include items such as
• comprehension Why would people use a secret
  ballot?
• vocabulary What does dilatory mean?
• similarities How are a telephone and a radio
  alike?
• Do these items (and others like them) assess
  intelligence in a valid manner?
• The construct validity of intelligence measures
  is a matter of heated debate.

8
Validity of Self-Report Measures

• Establishing the construct validity of a measure
  depends on
• convergent validity and
• discriminant validity.
• Convergent validity refers to the extent to which
  two measures of the same construct are correlated
  (go together).
• Discriminant validity refers to the extent to
  which two measures of different constructs are
  not correlated (do not go together).

9
Example of Construct Validity

• Suppose you have developed a new measure of
  self-esteem (i.e., a persons sense of
  self-worth).
• Which constructs listed below would you expect to
  show convergent validity ?
• Which would have discriminant validity?
• Measures of
• Depression
• Well-being
• Social anxiety
• Life satisfaction
• Grade point average

10
Example of Construct Validity Table 5.1

• Life Satisfaction construct
• Satisfaction with Life Scale (SWLS)
• Life Satisfaction (LS-5)
• Positive Affect (PA)
• Is life satisfaction the same or different from
  being happy?

11
Example of Construct Validity Table 5.1

• Check correlation between each of the measures
• Compare in a correlation matrix
• Each value is a correlation coefficient
• SWLS LS5 correlation of .77 evidence of
  convergent validity
• SWLS PA correlation of .42 evidence of
  discriminant validity
• LS5 PA correlation of .47 evidence of
  discriminant validity

12
Constructing a Questionnaire

• The best choice for selecting a questionnaire is
  to use one that already has been established as
  reliable and valid.
• If a suitable measure cannot be found,
  researchers choose to create their own
  questionnaire.
• It may seem easy, but a lot goes into developing
  a reliable and valid questionnaire.

13
Constructing a Questionnaire

• Important steps for preparing a questionnaire
• 1. Decide what information should be sought,
  demographics, constructs,
• 2. Decide how to administer the questionnaire
• 3. Write a first draft of the questionnaire.
  (borrow from other surveys)
• Reexamine and revise the questionnaire
• Pretest the questionnaire using a sample of
  respondents under conditions similar to the
  planned administration of the survey.
• 6. Edit the questionnaire, and specify the
  procedures for its use.

14
Guidelines for Effective Wording of Questions

• Choose how participants will respond
• free-response Open-ended (fill in the blank)
  questions allow greater flexibility in responses
  but are difficult to code.
• closed-response
• are quicker to respond to and easier to score
• may not accurately describe individuals
  responses
• For example Multiple choice, True-false, Likert
  Scale
• Use simple, direct, and familiar vocabulary
• keep questions short (20 or fewer words)
• Respondents will interpret the meaning of words
• Sensitivity to cultural and linguistics
  differences in word usage

15
Guidelines for Effective Wording of Questions

• Write clear and specific questions
• Avoid double-barreled questions (e.g., Do you
  support capital punishment and abortion?).
• Place any conditional phrases at the beginning of
  the question (e.g., If you were forced to leave
  your current city, where would you live?
• Avoid leading questions (e.g., Most people favor
  gun control what do you think?).
• Avoid loaded (emotion-laden) questions (e.g.,
  People who discriminate are racist pigs T or
  F).
• Avoid response bias with Likert Scale
  questions
• 1 is Strongly disagree 7 is Strongly
  disagree
• Original I get plenty of sleep
• Reversed I do not get enough sleep

16
Guidelines for Ordering of Questions

• For self-administered questionnaires, place the
  most interesting questions first to capture
  respondents attention.
• For personal and telephone interviews, place
  demographic questions first to establish rapport
  with the respondent.
• Use funnel questions Start with the most general
  questions, and move to more specific questions
  for a given topic.
• Use filter questions These questions direct
  respondents to the survey questions that apply
  directly to them.

17
MARGIN OF ERROR IN SURVEY RESULTS

• Due to sampling error between the actual
  population values and the estimates from our
  sample.
• Margin of error presents a range of values that
  are likely to contain the true population value
• Use confidence intervals to estimate error
• For example in elections
• 63 of those sampled favor the incumbent
• A 95 confidence has a margin of error of 5
• So between 58 and 68 favor the incumbent

18
Thinking Critically About Survey Research

• Correspondence Between Reported and Actual
  Behavior
• Peoples responses on surveys may not be
  truthful.
• Reactivity People sometimes dont report
  truthful responses, because they know the
  information is being recorded.
• Social Desirability occurs when people respond to
  surveys as they think they should, rather than
  how they actually feel or believe.

19
Thinking Critically About Survey Research

• Generally, researchers accept peoples responses
  as truthful, unless theres reason to suspect
  otherwise
• For example, responses arent consistent or
  visual pattern of responses forms a picture.
• Because behavior doesnt always match verbal
  reports of behavior, the multimethod approach to
  answering questions in psychology is best.

FIGURE 5.6 How people say they would respond to
this type of situation does not always match what
they actually do.
20
Thinking Critically About Survey Research

• Correlation and Causality
• Correlation does not imply causation.
• Example Correlation between being socially
  active (outgoing) and life satisfaction
• Three possible causal relationships
• A causes B (being outgoing causes people to be
  more satisfied with their life)
• B causes A (being more satisfied with life causes
  people to be more outgoing)
• C causes A and B

21
Thinking Critically About Survey Research

• Correlation and Causality
• Some third variable may be responsible for the
  relationship between social activity and life
  satisfaction.
• For example, having more friends (a third
  variable) may cause people to be more outgoing
  and to be more satisfied with their life.
• A correlation that can be explained by a third
  variable is called a spurious relationship.

22
Thinking Critically About Survey Research,
continued

• Diagram of direct and indirect relationships
• Chaos
• path b path c
• Poverty Psychological
• path a (direct) distress
• Chaos mediates the relationship between
  poverty and psychological distress among
  children.

23
Thinking Critically About Survey Research,
continued

• Path analysis example
• A moderator variable may affect the direction and
  strength of these relationships.
• Possible moderators
• Sex of the child
• Population density (e.g., rural, urban)
• Personality features of children (e.g.,
  resilience)

24
Thinking Critically About Survey Research,
continued

• Path analysis
• Helps us to understand relationships among
  variables
• But these relationships are still correlational
• Cannot make definitive causal statements
• Other untested variables may be important