Kein Folientitel

1
Response Scale Format and Measurement
Quality Dagmar Krebs Bettina Langfeldt Universi
ty of Giessen Germany European Conference on
Quality in Survey Statistics Cardiff, April
2006

2
Objective of the Study Attitude measurement in
national (German General Social Survey, Allbus)
as well as international surveys )ISSP, EVS) is
usually accomplished by response scales running
from left (positive) to right (negative)
assigning a low number to the positive and a high
number to the negative pole of the scale.
18 pn


extremely not at all important important 1 2
3 4 5 6 7 8
3
1. Question of the Study How is response
behavior affected by attaching different numbers
to scale endpoints 18 np 18 pn

not at all extremely important important 1
2 3 4 5 6 7 8
extremely not at all important important 1 2
3 4 5 6 7 8

4
2. Questions of the Study How is response
behavior affected by reversing the direction
of response scales? 18 np 81 pn

not at all extremely important
important 1 2 3 4 5 6 7 8
extremely not at all important important 8
7 6 5 4 3 2 1
5

Content 1. Hypotheses 2. Design of the
Study 3. Results 3.1 Direction Effects versus
Number Effects of Response Format 3.2 Difference
s in Variations and Correlations
3.3 Differences of Response Format and
Attitude Strength (Means) 4. Conclusion
6

• Hypotheses
• From a study of Schwarz Hippler, 1995
  empirical evidence is
• available about the effect of numerical labeling
  of scale points.
• Therefore effects of numbers attached to scale
  points can be
• expected especially in those designs where
  respondents know the
• numbers attached to response alternatives. This
  is the case in this
• experimental study.
• Hypothesis 1

If numbers attached to labels of response scales
have identical meaning, direction of the response
format has no effect on response behavior.
Therefore, comparison of response formats
18np - 81pn
yields more similar measures than comparison of
18np - 18pn
response formats.
Similarity can be shown in lower measures of
dispersion of responses to items as well as in
higher correlation coefficients.
7
Hypothesis 2 If respondents have a strong
attitude toward the topic in question here the
importance of job characteristics H2a and
the response scale starts with the negative pole
(18 np) respondents check middle or
slightly positive categories. H2b and the
response scale starts with the positive pole
(81 pn) and a high value, response patterns are
similar to those in the 18np-scale. H2c and
the response scale starts with the positive
pole but a low number, (18 pn) responses are
scattered over all categories of the scale
regardless of attitude strength. Attitude
strength is assessed in the first measurement.
8
2. Description of the Study 2.1 Design of the
Study
9

2.2 Number of Respondents participating in Wave
1 and Wave 2
10

• 2.3 Items
• refer to Job Motivation operationalised as
  judgements of the importance of future job
  characteristics (Irene Hanson Frieze).
• Items for three dimensions of job motivation
• Status Extrinsic Job Motivation
• Responsibility Intrinsic Job Motivation
• Flexibility

11

• Status Extrinsic Job Motivation
• - a good career right after finishing university
• - a job with high income
• - a job offering good chances for upward
  mobility
• - being respected by superiors
• - becoming one of the best in my field
• Responsibility Intrinsic Job Motivation
• - a job where I can decide independently how to
  do my work
• - a job where I can use my talents and
  capabilities
• - a job with tasks requiring a sense of
  responsibility
• - a job where I have the chance of realizing my
  ideas

• Flexibility Motivation
• - a job where I have a high degree of autonomy
  of decicision
• - a job with high flexibility of working hours
• - a job that leaves much time for the family
• - a job where I have much leisure time

12
2.4 Questionnaire Design Regarding your future
job how important are the following job
characteristics for you? Please give for each of
the following characteristics the number that
best describes how important that characteristic
is for you. For me it is ...

(for the 18 np condition)
13

• 2.5 Respondents
• Data collection was done in two waves with three
  response
• formats
• format 18 np for all participants in wave one
  and
• a split into 81 pn as well 18 pn in wave two.
• Randomization of split versions by mode of
  distribution -
• every second student got split 81 pn (or split
  18 pn respectively).
• Respondents students of (mostly) paedagogics in
  an
• introductory lecture.
• No systematic or random sampling. Questionnaires
  were
• distributed and collected during regular
  lessons.
• Items were identical in both waves and split
  versions.

14
3. Results
15
(No Transcript)
16

17

• Hypothesis 1
• If numbers attached to labels of response scales
  have identical
• meaning, direction of the response format has no
  effect on
• response behavior.
• Differences between response formats

18np 81pn
18np 18pn
are smaller than between
low and similar standard deviations expected and
shown by the data.
high standard deviations expected and shown by
the data.
Also correlations are in the expected direction
and strength.
18
 
Convergent correlations
19
Hypothesis 2 Attitude strength, scale format and
response behavior a) strong attitude high item
scores in the middle or slightly positive

20
b) strong attitude in 1st measurement item
scores in 2nd measurement

21
c) strong attitude in 1st measurement item
scores are scattered over all response categories
in 2nd measurement

22

• 4. Conclusion
• Scales 18np and scale 81pn yield more similar
  results
• - smaller differences in means than 18pn
  response format
• - smaller differences in variation than 18pn
  response format
• - comparable response patterns regarding
  attitude strength
• - higher convergent correlations between
  measurements.
• The higher amount of response variation in
  response format 18pn compared to response format
  18np or 81pn can be
• interpreted as a hint, that this format violates
  conversational
• norms in assigning the high value to the
  negative and the low
• value to the positive response alternative. The
  18pn response
• format is different from the other two formats
  and therefore it
• makes a big difference which one is used. Which
  format is the
• better one in terms of data quality can not
  yet be decided.

23

• Therefore,
• further research is necessary, comparing scales
  internationally to
• get more information about respondents
  understanding of
• response formats and to confirm or reject the
  international use of
• the 18pn response format.

24

Thank you for your attention