THREATS TO INTERNAL VALIDITY

1
THREATS TO INTERNAL VALIDITY
2

• HISTORY
• Is a threat when an observed effect might be due
  to an event which takes place between the pretest
  and the post test, when this event is not the
  treatment of research interest.

3

• MATURATION
• Is a threat when an observed effect might be due
  to a change in the respondent between pretest and
  post test.

4

• TESTING
• Is a threat when an effect might be due to the
  number of times particular responses are measured.

5

• INSTRUMENTATION
• Is a threat when an effect might be due to a
  change in the measuring instrument between
  pretest and post test and not to the treatments
  differential input at each time interval

6

• STATISTICAL REGRESSION
• Is a threat when an effect
• might be due to respondents being classified
  into groups on the basis of unreliable pretest
  scores. . . .

7

• When this happens, high pretest scorers will
  score relatively lower at the post test and low
  pretest scorers will score relatively higher at
  the post test.

8

• SELECTION
• Is a threat when an effect may be due to the
  difference between the kinds of people in one
  experimental group as opposed to another.

9

• MORTALITY (ATTRITION)
• Is a threat when an effect may be due to the
  different kinds of persons who dropped out of a
  particular threatment group during the course of
  an experiment.

10

• INTERACTIONS WITH SELECTION
• Many of the forgoing threats to internal
  validity can interact with selection to produce
  forces that might spuriously appear as
  threatment effects.

11

• EXAMPLES
• Selection--Maturation results when experimental
  groups are maturing at different speeds.

12

• Selection--History results from various
  treatment groups coming from different settings
  so that each group could experience a unique
  local history that might effect outcome variables.

13

• DIFFUSION OR IMITATION OF TREATMENTS

14

• Is a threat when treatments involve
  informational groups that can communicate with
  each other. Respondents in one treatment group
  may learn the information intended for others.

15

• COMPENSATORY EQUILIZATION OF TREATMENTS

16

• Is a threat when the experimental treatment
  provides goods or services that are generally
  believed to be desirable. There may then emerge
  administrative and constituency reluctance to
  tolerate the focused inequality that results.

17

• COMPENSATORY RIVALRY by RESPONDENTS
  RECEIVING LESS DESIRABLE TREATMENTS

18

• Is a threat when the assignment of persons or
  organizational units to experimental and control
  conditions is made public and conditions of
  social competition are generated. The control
  group, as the natural underdog, may be motivated
  to reduce or reverse the expected difference.

19

• RESENTFUL DEMORALIZATION of RESPONDENTS
  RECEIVING LESS DESIRABLE TREATMENTS

20

• When an experiment is obstrusive, the reaction of
  a no-treatment control group or groups receiving
  less desirable treatments can be associated with
  resentment and demoralization, as well as with
  compensatory rivalry.

21
THREATS TO STATISTICAL CONCLUSTION VALIDITY
22

• LOW STATISTICAL POWER
• The likelihood of making an incorrect
  no-difference conclusion increases when sample
  sizes are small, and significance level is set
  low.

23

• VIOLATED ASSUMPTIONS OF STATISTICAL TESTS
• Most tests of the null hypothesis require that
  certain assumptions be met if the results are to
  be meaningfully interpreted.

24

• FISHING AND ERROR RATE PROBLEM

25

• The likelihood of falsely concluding that
  covariation exists when it does not increases
  when multiple comparisons of mean differences are
  possible and there is no recognition that a
  certain proportion of the comparisons will be
  significantly different by chance.

26

• THE RELIABILITY OF MEASURES
• Unreliable measures inflate standard errors of
  estimate and these standard errors play a crucial
  role in inferring differences between statistics,
  such as the means of different treatment groups.

27

• THE RELIABILITY OF TREATMENT IMPLEMENATION

28

• The way a treatment is implemented may differ
  from one person to another if different persons
  are responsible for implementing the treatment. .
  .

29

• This lack of standardization, both within and
  between persons, will inflate error variance and
  decrease the chance of obtaining true differences.

30

• RANDOM IRRELEVANCIES IN THE EXPERIMENTAL
  SETTING

31

• Some features of an experimental setting other
  than the treatment will undoubtedly affect scores
  on the dependent variable and will inflate error
  variance.

32
THREATS TO EXTERNAL VALIDITY
33

• INTERACTION OF SELECTION AND TREATMENT

34

• If subjects in a study are of one category or
  type, treatment results may not be generalizable
  outside of that category of person. Results may
  in fact be due to characteristics of that
  category of person.

35

• INTERACTION OF SETTING AND TREATMENT

36

• If a study is conducted in only one setting,
  treatment results may be specific to that setting
  and not generalizable to other settings.

37

• SPECIAL DAY
• If an experiment takes place on a special day,
  it is questionable whether a discovered
  cause-effect relationship would also be found
  under more mundane circumstances.

38
(No Transcript)
39
(No Transcript)