Theory Building

1
Theory Building

• Reliability
• Validity
• Levels of Measurement
• Scales

2
Reliability

• Measurement directly affects the quality of
  conclusions
• Care is needed to make sure that results are not
  corrupted by improper measurement
• The operational definition of a concept should
  have a precise meaning
• The terms by which you measure a concept should
  be explicit

3
Reliability

• Reliability and validity are the biggest threats
  to proper measurement
• Reliability is the extent to which an experiment,
  test, or any measuring procedure yields the same
  results on repeated trials
• Do you get the same result every time?

4
Reliability

• Three tests of reliability
• Test-retest method
• Applying the same test to the same observations
  after a period of time and then comparing the
  results of the different measurements
• Alternative form method
• Two different measures of the same concept
  administered to the same respondents at different
  times before the scores are compared
• Split-halves method
• Divide a multi-item measure into two measures
  with both of the new measures applied at the same
  time

5
Validity

• A valid measure is one that measures what it is
  supposed to measure, in other words, the degree
  of correspondence between the measure and the
  concept it is thought to measure
• Four tests of validity

6
Validity

• Face validity
• Can be asserted, but not demonstrated, when the
  measurement instrument appears to measure the
  concept as intended
• Content validity
• Entails determining the full meaning of a concept
  and then making sure that the full concept is
  included in the measure

7
Validity

• Construct validity
• A measure of a concept is related to a measure of
  another concept with which the original concept
  is thought to be related
• Inter-item association
• Similarity of outcomes of multiple measures of a
  concept to demonstrate the validity of the entire
  measurement scheme

8
Validity

• Tests of validity are not as good as tests of
  reliability
• Reliability is easy to demonstrate through some
  form of repeated trials
• Validity is more difficult because we can never
  be sure about the true value of a concept
• Especially true with abstract concepts

9
Validity

• Whereas a valid measure is reliable (because if
  truly valid, it will measure the concept
  correctly every time), a reliable measure is not
  necessarily valid
• The measure could be measuring the concept
  incorrectly in a consistent way

10
Levels of Measurement

• The level of measurement of a variable describes
• The amount of precision associated with a
  variable
• The mathematical properties of the variable
• Both precision and mathematical properties
  increase as you increase the level of measurement
  from nominal to ratio

11
Levels of MeasurementNominal

• The nominal level describes variables that
  indicate only a difference between categories
• Example Which school did you attend?
• Very little precision and cannot be used with
  many statistical tools

12
Levels of MeasurementOrdinal

• At the ordinal level, categories may be ranked in
  order in addition to indicating a difference
  between categories
• Example Please indicate the highest level of
  education you reached (elem., high, college,
  more)
• Precision A little more precision and can be
  used with more statistical tools

13
Levels of MeasurementInterval

• The interval level includes all of the
  information of the preceding levels and adds
  meaningful intervals between values of the
  variable but does not use a meaningful zero
• Example What did you score on the SAT?
• Precision More precision and can be used with
  most statistical tools

14
Levels of MeasurementRatio

• The ratio level adds a meaningful zero to the
  interval level
• Example How many years of education?
• Precision Most precision and can be used with
  most statistical tools

15
Scales

• Some concepts can be captured with a single
  question
• More complex concepts may require a multi-item
  measure consisting of several questions that
  capture different components of the concept and
  increase validity

16
Scales

• Summation index
• Combines the scores on multiple questions to
  create one single measure of a concept
• Likert scale
• Uses only select questions from an index that
  differentiate between different respondents to
  create a single score for each respondent

17
Scales

• Guttman scale
• Has answer choices arranged in an ordinal manner
  respondents will agree with each of the
  lower-ranked answers if they agree with a
  higher-ranked answer
• Factor analysis
• Allows researchers to uncover patterns across
  related measures to create summary variables that
  represent different dimensions of the same concept