External validity

1
External validity Sampling

• Generalization

2
External Validity

• the approximate truth of conclusions that involve
  generalizations
• the degree to which the conclusions in a study
  would hold for other persons in other places and
  at other times

Key Question How reasonable is it to believe that
the result found in this study would be the same
with other people, in other places, at other
times?
3
External Validity and Sampling

• Sampling
• process of selecting units (e.g., people,
  organizations, occasions) from a population of
  interest so that by studying the sample you can
  fairly or reasonably generalize your results to
  the population from which the units were chosen
• primary approach is referred to as the sampling
  model

4
Sampling Model
Generalize back
Population
Draw sample
Sample
5
Sampling Model

• The goal is to claim representative sampling
• potential problems
• knowing the population you want to generalize to
  in the first place
• ability to draw a representative sample
• ability to sample across all times
• We can never generalize with absolute certainty,
  rather we have to decide how reasonable is it to
  assume similar results give different
  circumstances

6
Threats to External Validity
Interaction of selection and treatment
Maybe it is just these people.
Interaction of setting and treatment
Maybe it is just these places.
Interaction of history and treatment
Maybe it is just these times.
7
Improving External Validity
Replicate, Replicate, Replicate
8
Types of Sampling
Probability sampling
Uses some form of random selection, and requires
that each unit have a known (often equal)
probability of being selected
Non-probability sampling
selection is systematic or haphazard, but not
random
9
Practice Question

• To study exercisers in Bloomington/Normal, a
  researcher drew 2 health clubs at random (both of
  which happened to be large clubs open to the
  general public with hundreds of members) to
  recruit participants from. Do you think this will
  be a representative sample? Why or why not?

10
Practice Question

• A poll was conducted by a magazine editor by
  printing a questionnaire in an issue of the
  magazine for readers to mail back. Thousands of
  readers returned completed questionnaires.
  Suppose a friend reads the results and is
  convinced that it reflects the view of all
  adults. What would you say to convince your
  friend that he/she might be wrong?

11
Guiding Questions for Critiquing the External
Validity of Research

1. What are the main results of the study (e.g.,
   positive or negative relationship, group
   differences, effectiveness of the intervention or
   treatment)?
2. Do the researchers explicitly state or imply that
   similar results would hold for other (a) people,
   (b) places or situations, and/or (c) times? If
   so, what is the population/place/time they are
   attempting to generalize to?
3. If the researchers are generalizing their
   results, how reasonable are these conclusions
   given the sample, sampling procedures, and
   settings used? This is the key External Validity
   question
4. What specifically might lead you to question
   these conclusions? In other words, if they did
   suggest the results were generalizable, why might
   you think otherwise? The more convincing of a
   rationale you can generate, the more you should
   question the external validity

12
Use the guiding questions to evaluate the
external validity of the following study

• Prior research has found that (a) intercollegiate
  athletes are especially at-risk for excessive
  alcohol consumption (e.g., Nelson Wechsler,
  2001), and (b) sport-type differences exist among
  college athletes in terms of yearly drinking
  prevalence rates (National Collegiate Athletic
  Association, 2001). No studies, however, have
  examined sport-type differences on more specific
  measures of alcohol consumption (i.e., drinks per
  week). In the present study, data were analyzed
  on 298 intercollegiate athletes from two
  different NCAA Division III universities. Results
  indicated significant sport type differences on
  alcohol consumption variables, with athletes from
  the sports of swimming and diving and wrestling
  reporting the highest levels of alcohol
  consumption (M 5.20, SD 4.00) and soccer and
  football reporting the lowest (M 4.02, SD
  3.25). Results suggest college athletes
  participating in individual sports are at-risk
  for future alcohol abuse.