QuasiExperimental Design

1
Quasi-Experimental Design
2
Quasi-experiments provide a good model for the
kind of detective work and creative problem
solving that should be used in any kind of
research.
3

• One Group Post-Test Only Design
• X O
• The simplest and the weakest possible design
• Lack of a pretest prevents assessment of change
• Lack of a control group prevents threats from
  being ruled out.

4
One Group Post-Test Only Design X O This design
highlights the importance of contextual knowledge
when interpreting results. Some research with
weak designs can be convincing because of the
background expectations against which the results
are judged. Example Why was Milgrams obedience
research so convincing?
5
One Group Post-Test Only Design X O Without
changing the basic nature of this design, it can
be improved considerably by adding additional
outcome measures O1 X O2
O3Compared to norms or expectations, only O2
should be unusual.
6
Post-Test Only Design with Nonequivalent
Groups X O O This design is an
improvement, but the lack of a pretest means that
any posttest differences could be due to
selection. How do we know if the groups are
equivalent?
7
Post-Test Only Design with Nonequivalent
Groups X1 OX2 OX3 OX4
O O
Why is this a stronger version of the same basic
design?
8

• One-Group Pretest Post-Test Design
• O X O
• This very common applied design is susceptible to
  all threats to within-groups comparisons
• History
• Maturation
• Testing
• Regression
• Instrumentation

9

• One-Group Pretest Post-Test Design
• O X O
• Threats in this design might be ruled out with
  logic or common sense (e.g., regression).
• Some maturation threats might be ruled out by
  correlating the pretest with other available
  maturation variables (e.g., age).
• Attrition after the pretest can create additional
  selection problems.

10
One-Group Pretest Post-Test Design O X
O One powerful modification is to add pretests O
O O O O X O Maturation threats
can now be examined and their influence separated
from treatment effects.
11
O O O O O O O O O X O
12
Untreated Control Group Design with Pretest and
Posttest O1 X O2 O1 O2
This design is stronger than the previous three
because it provides both kinds of comparisons
within-groups and between-groups. The major
threat is selectionthe possibility that any
outcome differences are due to pre-existing
differences between the groups.
13
Untreated Control Group Design with Pretest and
Posttest O1 X O2 O1 O2
This design highlights the importance of outcome
patterns. Some patterns of outcome are
implausible as arising from some common threats.
Others are quite plausible and require design
modification to eliminate.
14
Treatment
Outcome
Control
Pretest
Posttest
This pattern might be interpreted as evidence for
the effectiveness of treatment, but some
plausible threat-based explanations would first
have to be ruled out.
15
This pattern could arise from differential
maturation in the two groups. Internal analyses
might be able to rule that out. In an internal
analysis, data from the study are used to
diagnose problems that threaten the main purpose
of the study.
16
Treatment
Outcome
Control
Pretest
Posttest
17
The pattern might also arise from instrumentation
problems that affect the two groups differently.
Outcome
Treatment
Construct
Control
18
The pattern might also arise from differential
statistical regression.
Control
Treatment

The treatment group is selected from the extreme
end of the distribution to make participants as
similar as possible to the average control group
participant.
19
Treatment
Outcome
Control
Pretest
Posttest
This design and outcome might be typical when the
purpose of research is to reduce some problematic
behavior to the level of a comparison group.
20
Two alternative explanations are plausible. This
pattern might occur because of instrumentation
problems (a floor effect that prevents the
control group from decreasing) and differential
maturation. Perhaps the treated group is simply
composed of late bloomers who would shift their
behavior on their own in the absence of
treatment. How would either be ruled out?
21
Control
Outcome
Treatment
Pretest
Posttest
Compensatory designs suffer from the same
problems. These designs, with large pretest
differences, are also susceptible to regression
problems. How can those be ruled out?
22
Treatment
Control
Outcome
Pretest
Posttest
This outcome pattern effectively rules out
instrumentation problems. Why?
23
Control
Treatment
Outcome
Control
Treatment
Construct
24
This outcome pattern is unlikely to arise from
regression, unless the treatment group was
selected from the lower end of that
groups distribution and the mean is higher than
the distribution from which the control group was
selected.
Control
Treatment

25
This pattern would not likely arise from a simple
maturation process.
Treatment
Outcome
Control
Age
26
Untreated Control Group Design with Proxy
Pretest Measures OA1 X OB2 OA1
OB2
In this design, the pretest and posttest are
different measures. This may be necessary if the
posttest cannot be given as a pretest or if there
is concern that it might sensitize participants
if it was given as a pretest.
27
Untreated Control Group Design with Proxy
Pretest Measures OA1 X OB2 OA1
OB2
Example The study might investigate the impact
of a new curriculum on learning of algebra. It
might make no sense to give an algebra test
before the introduction of the treatment because
there might not be any variability.
28
Untreated Control Group Design with Proxy
Pretest Measures OA1 X OB2 OA1
OB2
Instead, a measure that correlates with the
ability to learn algebra (e.g., math aptitude)
could be given. The hope is that the proxy
pretest will allow the two advantages of
covariance analysis.
29
Unadjusted difference
Adjusted difference
Algebra Achievement
Error is also reduced
Control
Treatment
Math Aptitude
30
The major problem with the proxy pretest design
is that the proxy is an imperfect measure of the
construct that should be controlled.
Consequently, the covariance adjustment may not
be adequate.
31

• Inadequate covariance adjustment due to a faulty
  or unreliable pretest can
• Produce a spurious treatment effect

Pretest Posttest
Pretest Posttest
Without adjustment or with poor adjustment of
pretest differences
With adjustment of pretest differences
32
Inadequate covariance adjustment due to a faulty
or unreliable pretest can (b) Fail to uncover a
real treatment effect
Pretest Posttest
Pretest Posttest
Without adjustment or with poor adjustment of
pretest differences
With adjustment of pretest differences
33
The Big Picture Threats can arise in any design
in many and sometimes subtle ways. Controlling
those threats requires using a variety of tactics
to render threats less plausible, if they cannot
be eliminated entirely. These tactics include
considering the patterns that might indicate the
presence of threats, design modifications, and
statistical controls.
34

• The ability to recognize and solve threats to
  internal validity is critical to conducting good
  research, for everyone
• An intended experiment fails to accomplish one of
  the key defining features
• An experiment cannot be conducted for practical
  or ethical reasons
• A consumer of research needs to know what
  conclusions can be trusted

35
(No Transcript)