14. Validity and Reliability

1
14. Validity and Reliability

• Pelham Blanton
• Ch 3 4

2
Validity

• The relative accuracy or correctness of a
  psychological statement.
• Types
• Internal
• External
• Construct
• Conceptual

3
1. Internal Validity

• The extent to which a set of research findings
  provides evidence about causality
• If a study is high in internal validity, then we
  can say that changes in the dependent variable
  were due to our manipulations of the independent
  variable
• Example?

4

• Lab experiments are typically high in internal
  validity because of
• high levels of control over extraneous variables
• random assignment

5
John Stuart Mills Requirements to Establish
Causality

• Covariation
• If Variable A causes Variable B then changes in A
  will result in changes in B
• This alone is NOT enough to determine causality
• Temporal Sequence
• If Variable A causes Variable B, then A must
  precede B
• Longitudinal studies
• Again NOT enough to determine causality on its own

6
John Stuart Mill (contd)

• Eliminating Confounds
• i.e., conducting a true experiment to rule out
  confounding (third) variables
• It is not always possible to conduct true
  experiments however
• Lab experiments tend to be high in internal
  validity but low in external validity

7
2. External Validity

• The extent to which we may generalize from our
  study to the real world.
• Example?
• If there are many boundary conditions then
  external validity is low

8
Two Forms of Generalizabilty

• Generalizability wrt people
• e.g., subjects all from same social group,
  culture, age, point in history
• Generalizability wrt situations
• Do behaviours in a lab resemble behaviours in
  other situations?
• e.g., hallway aggression, distraction in
  cognitive tasks driving
• Observational studies are high in external
  validity but low in internal validity. Why?

9
3. Construct Validity

• Do the variables that you measure really reflect
  the abstract, hypothetical variables you are
  interested in?
• Quality of operational definitions
• Stealing is immoral
• Lester loves Viola
• Sometimes there is debate about what is the best
  O.D. (e.g. self-esteem)
• If the operational definitions do a good job of
  converting the abstract to the observable then
  construct validity is high

10
4. Conceptual Validity

• How well a specific research hypothesis maps onto
  the broader theory.
• Example?
• Should the researcher have been interested in her
  specific hypothesis? Did it logically follow from
  the theory? Or did it come out of nowhere?
• Construct specific hypothesis
• Conceptual broader theory

11
Reliability

• The consistency or repeatability of a measure or
  observation
• Three kinds
• Inter-rater reliability
• Internal consistency
• Test-retest reliability

12
1. Inter-rater Reliability

• a.k.a. inter-observer agreement
• The degree to which different judges (raters)
  independently agree on an observation
• Judges need to be trained and independent
• e.g.,
• Panel of judges in sports competitions
• Clinical trials
• Cell counting
• Your midterm
• During training the raters may discuss how they
  are scoring, but at the start of the study the
  ratings must be independent.

13
2. Internal Consistency

• The degree to which all the items (observations)
  in a multiple-item measure behave the same way
• E.g., all the questions on a self-esteem scale
  should make the same conclusions about
  self-esteem
• E.g., your midterm
• If an item is not consistent, you should remove
  it from your measure

14
3. Test-retest Reliability

• a.k.a. temporal consistency
• The degree to which a measure gives the same
  results across repeated uses
• Test-retest reliability of adult IQ on WAIS is .96

15

• State vs trait
• More reliable measures are usually more valid
• Reliability is a necessary but not a sufficient
  condition for validity

16
Measurement Scales

• Nominal
• Ordinal
• Interval
• Ratio

17
1. Nominal Scales

• a.k.a. categorical scales
• Naming or labeling variables
• Mutually exclusive exhaustive
• e.g.,
• Gender (traditional)
• Clinical diagnoses

18
2. Ordinal Scales

• Order or ranking
• Mutually exclusive exhaustive
• e.g.,
• 1st, 2nd, 3rd
• Top 10 movies
• Life stress scales (number of stressful events in
  last 6 months). We do not assume anything about
  the differences (i.e., the diff btw 1015 is not
  the same as the diff btw 20 25)
• Know the order but not the difference

19
3. Interval Scales

• Provide information about differences between
  stimuli (but not ratios)
• The diff between 10oF and 20oF is the same as the
  diff between 80oF and 90oF
• BUT 40oF is not half as hot as 80oF
• Can have negative values (e.g., -12oF)
• Can accommodate ties
• e.g.,
• GRE scores
• Likert scales
• Almost all psychological scales

20
4. Ratio Scales

• Single unit differences always have exactly the
  same value along all possible points on the scale
• Have a true zero point (none of the quantity is
  present)
• Must be true not arbitrary such as 0oF
• Ratio scales cannot have negative values
• e.g.,
• Length
• Volume
• Time

21
Which Scale to Use?

• Use the most sensitive one possible