QuasiExperimental Designs That Use Both Control Groups and Pretest

1
Quasi-Experimental Designs That Use Both Control
Groups and Pretest

• Chapter 5 EQD

2
Designs That Use Both Control Groups and Pretests

• The use of carefully selected comparison groups
  facilitates causal inference.
• Of more advantage if accompanied by pretests
• Tell us how groups being compared initially
  differ and so alert us to the higher probability
  that some internal threats rather that others
  might be operating
• Tell us something about the magnitude of initial
  group differences on the variable that is usually
  most highly correlated to the outcome
• Help us with the statistical analysis, especially
  if the reliability of these measures is known

3
Untreated Control Group Design with Dependent
Pretest and Posttest Samples

• Nonequivalent Comparison Group Design
• NR O1 X O2
• NR O1 O2
• The joint use of a pretest and a comparison group
  makes it easier to examine certain threats to
  validity.
• Because the groups are nonequivalent by
  definition, the selection bias is assumed to be
  present
• Pretests allows exploration of the possible size
  and direction of that bias.

4
Validity Threats

• When pretests differences exist, the possibility
  increases that the selection will combine with
  other threats.
• Selection-Maturation Threat
• results from differential rates of normal growth
  between pretest and posttest for the groups
• Selection-Instrumentation Threat
• test changes differently for the two groups
• Selection-Regression Threat
• different rates of regression to the mean in the
  two groups (if one is more extreme on the pretest
  than the other)
• Selection-History Threat
• an event occurs between pretest and posttest that
  groups experience differently

5
Outcome 1

• Both groups grow apart in the same direction.
• Possible threats
• Selection-maturation
• Selection-history

6
Outcome 2

• No change in the control group.
• Possible threats
• Selection-history
• Selection-maturation
• Selection-instrumentation

7
Outcome 3

• Initial pretest differences favoring the
  treatment group that diminish over time.
• Possible threats
• Selection-regression
• Selection-history
• Selection-instrumentation

8
Outcome 4

• Initial pretest differences favoring the control
  group that diminish over time
• Possible threats
• Selection-regression
• Selection-history
• Selection-instrumentation

9
Outcome 5

• Outcomes that cross over in the direction of
  relationships.

10
Ways to improve the untreated control group
design with dependent pretest and posttest samples

• Using a Double Pretest
• NR O1 O2 X O3
• NR O1 O2 O3
• Same pretest is administered at two different
  time points, preferably with the same time delay
  as between the second pretest and the posttest.
• Using Switching Replications
• NR O1 X O2 O3
• NR O1 O2 X O3
• The researcher administers treatment at a later
  date to the group that initially served as a
  no-treatment control
• Using a Reversed-Treatment Control Group
• NR O1 X O2
• NR O1 X- O2
• The treatment in one group is expected to produce
  an effect in one direction and the treatment on
  the other group (opposite treatment) is expected
  to reverse the effect

11
Cohorts

• Useful as control groups if
• One cohort experiences a given treatment and
  earlier or later cohorts do not
• Cohorts differ in only minor ways from other
  cohorts
• Organizations insist that a treatment be given to
  everybody, thus making impossible simultaneous
  controls and making possible only historical
  controls
• An organizations archival records can be used
  for constructing and then comparing cohorts

12
Designs that Combine many Design Elements

• Untreated matched controls with multiple pretests
  and posttests, Nonequivalent dependent variables,
  and Removed and repeated treatment.
• Ask for the Sale campaign to sell lottery
  tickets.
• Combining switching replications with a
  nonequivalent group design.
• Study of health education program.
• An untreated control group with a double pretest
  and both independent and dependent samples.
• Community-level interventions designed to reduce
  cardiovascular risk factors.

13
Elements of Design

• Assignment
• Measure
• Comparison Groups
• Treatment

14
Assignment

• Not always controlled by the researcher.
  Sometimes participants self-select into
  conditions or someone else makes the assignment
  decision.
• Assignments can be improved by
• Matching
• Masking (blinding)
• Not allowing self-selection

15
Measurement

• Posttest observation
• Nonequivalent dependent variable
• Multiple substantive posttests
• Pretest observation
• Repeated pretests
• Retrospective pretests
• Proxy pretests
• Pretests on independent samples
• Moderator variable with predicted interaction
• Measuring threats to validity

16
Comparison Groups

• Single nonequivalent groups
• Multiple nonequivalent groups
• Cohorts
• Internal vs. external contrast
• Constructed contrast
• Regression extrapolation contrasts actual and
  projected posttests scores are compared.
• Normed contrasts treatment group scores are
  compared with normed samples from test manuals
  and the like.
• Secondary data contrasts treatment respondents
  are compared with samples drawn from other
  studies.

17
Treatment

• Switching replications
• Replicates the treatment effect at a later date
  in a group that originally served as a control.
• Reversed treatments
• Treatment expected to reverse the outcome when
  compared with the expected outcome in the
  treatment condition.
• Removed treatments
• First presents and then removes treatment to
  demonstrate that the pattern of outcomes follows
  the pattern of treatment application.
• Repeated treatments
• Reintroduces the treatment after its removal

18
Conclusion

• This chapter emphasizes building stronger designs
  through adding design features that reduce the
  plausibility of validity threats in the context
  under study.