Quasi-Experimental And N=1 Designs OF Research

1
Chapter 22

• Quasi-Experimental And N1 Designs OF Research

2

• In earlier chapters we stated and emphasized that
  one of the major goals of science is to find
  causal relations. In the behavioral sciences, the
  true experiment is the strongest approach used to
  meet this goal.
• However, there are research problems in the
  behavioral sciences and especially educational
  research that cannot be studied using a true
  experimental design. We will examine two research
  designs where one or more of the components of
  the true experiment have been compromised. The
  first is called quasi experimental designs and
  the second is called single subject or N1
  designs.

3
Compromise Designs a.k.a. Quasi-Experimental
Designs

• Recall that true experimentation requires at
  least two groups, one receiving an experimental
  treatment and one not receiving the treatment, or
  receiving it in different form. The true
  experiment requires the manipulation of at least
  one independent variable, the random assignment
  of participants to groups, and the random
  assignment of treatments to groups.

4
Compromise Designs a.k.a. Quasi-Experimental
Designs

• Cook and Campbell (1979) present two major
  classifications of quasi-experimental design.
• The first is called the nonequivalent control
  group designs, the second is the interrupted
  time series designs.

5
Nonequivalent Control Group Designs

• No-treatment control group designs
• Nonequivalent dependent variables designs
• Removed treatment group designs
• Repeated treatment designs
• Reversed treatment nonequivalent control group
  designs
• Posttest only designs
• Regression continuity designs

6
No-Treatment Control Group Designs

• Design 22.1
• An effort should be made to at least use samples
  from the same population, or samples that are as
  alike as possible. The experimental treatments
  should be assigned at random. Then the similarity
  of the groups should be checked using any
  information available (sex, age, social class,
  and so on). The equivalence of the groups could
  be verified using the means and standard
  deviations of the pretests t-test and F-test
  will do.

7
No-Treatment Control Group Designs

• There are still difficulties, all of which are
  subordinate to one main difficultyselection.
  When participants are selected into groups on
  bases extraneous to the research purposes, we
  call this selection or alternatively,
  self-selection
• Note that if we had used only volunteers and had
  assigned them to experimental and control groups
  at random, the selection difficulty is lessened.
  External validity or representativeness, however,
  would be decreased.

8
No-Treatment Control Group Designs

• Without the benefit of random assignment,
  attempts should be made through other means to
  eliminate rival hypotheses. We consider only the
  design that uses the pretest because the pretest
  could provide useful information concerning the
  effectiveness of the independent variable on the
  dependent variable.

9
No-Treatment Control Group Designs

• Another more frequent example in educational
  research is to take some school, classes for the
  experimental group and others for the control
  group. If a fairly large number of classes are
  selected and assigned at random to experimental
  and control groups, there is no great problem.
• But if they are not assigned at random, certain
  ones may select themselves into the experimental
  groups, and these classes may have
  characteristics that predispose them to have
  higher mean Y scores than the other classes.

10
No-Treatment Control Group Designs

• In other words, something that influences the
  selection process (e.g., volunteer participants),
  also influences the dependent variable measures.
  This occurs even though the pretest may show the
  groups to be the same (alike) on the dependent
  variable. The X manipulation is effective
  because of selection, or self-selection, but it
  is not effective in and of itself.

11
No-Treatment Control Group Designs

• Possible outcomes from this design are given in
  Figure 22.1. There is the possibility of a
  different interpretation on causality depending
  on which outcome the researcher obtains. In
  almost all of the cases the most likely threat to
  internal validity would be the selection-maturatio
  n interaction.

12
No-Treatment Control Group Designs

• You might recall that this interaction occurs
  when (1) two groups are different to begin with
  as measured by the pretest then (2) one of the
  groups experience greater differential changes,
  such as getting more experienced, more accurate,
  more tired, and so on, than the other group. The
  after-treatment difference, as observed in the
  posttest, can not exactly be attributed to the
  treatment itself.

13
No-Treatment Control Group Designs

• There are four alternative explanations to the
  outcome in Figure 22.1(a)
• The first is selection-maturation interaction.
  Group Es increase may be due to their higher
  level of intelligence. With a higher level of
  intelligence, these participants can process
  more, or grow faster, than Group C.

14
No-Treatment Control Group Designs

• A second explanation is one of instrumentation.
  The scale used to measure the dependent variable
  may be more sensitive at certain levels than
  others. In a normal distribution, changes in raw
  scores near the center of the distribution
  reflect bigger percentile changes than at the
  tails.

15
No-Treatment Control Group Designs

• The third explanation is statistical regression.
  The increase in scores by Group E would be due to
  their selection on the basis of extreme scores.
  On the posttest, their scores would go up because
  they would be approaching the population
  baseline.
• The fourth explanation centers on the interaction
  between history and selection.

16
No-Treatment Control Group Designs

• All of the threats mentioned for Figure 22.1(a)
  are also true for Figure 22.1(b). To determine if
  selection-maturation is playing a main role in
  the results, Cook and Campbell (1979) recommend
  two methods. The first method involves looking
  only at the data for the experimental group
  (Group E). If the within-group variance for the
  posttest is considerably greater than the
  within-group variance of the pretest, then there
  is evidence of a selection-maturation
  explanation.

17
No-Treatment Control Group Designs

• The second method is to develop two plots and the
  regression line associated with each plot. Figure
  22.2
• If the regression line slopes for each plot
  differ from each other, then there is evidence of
  a differential average growth rate, meaning that
  there is the likelihood of a selection-maturation
  interaction.

18
No-Treatment Control Group Designs

• The outcome shown in Figure 22.1(c) is more
  commonly found in clinical psychology studies.
  The treatment is intended to lead to a decline of
  an undesired behavior.
• This outcome is also susceptible to
  selection-maturation interaction and three others.

19
No-Treatment Control Group Designs

• The fourth outcome is shown in Figure 22.1(d).
  The selection-maturation threat can be ruled out
  since this effect usually results in a slower
  growth rate for low scores and a faster growth
  rate for high scores.
• The final outcome is shown in Figure 22.1(e). The
  four threats can be ruled out. Hence, the outcome
  in Figure 22.1(e) seems to be the strongest one
  and should enable the researcher to make a causal
  statement concerning treatment.

20
Research Examples

• Nelson, Hall, and Walsh-Bowers (1997)
  Nonequivalent Control Group Design.
• They were unable to assign participants to
  different housing settings randomly.
• They state that the difference they found between
  these three groups on posttest measures could
  have been due to the selection problem, and not
  the type of care facility.

21
Research Examples

• Chapman and McCauley (1993) Quasi-Experiment
• Although one can perhaps think of this study as a
  nonexperimental one, Chapman and McCauley felt
  that it came under the classification of
  quasi-experimental.
• Awards were given to approximately half of a
  homogeneous group of applicants in a procedure
  that Chapman and McCauley say approximates random
  assignment to either fellowship or honorable
  mention.

22
Time Designs

• Design 22.2 A longitudinal Time Design (a.k.a.
  Interrupted Time Series Design)
• The reactive effect should show itself by
  comparing Y3 to Y4 this can be contrasted with
  Y5. If there is an increase at Y5 over and above
  the increase at Y4 from Y3, it can be attributed
  to X. A similar argument applies for maturation
  and history.

23
Time Designs

• One difficulty with longitudinal or time studies,
  especially with children, is the growth or
  learning that occurs naturally over time
  Children do not stop growing and learning for
  research convenience. The longer the time period,
  the greater the problem. In other words, time
  itself is a variable.

24
Time Designs

• The most widely used statistical test is ARIMA
  (autoregressive, integrated, moving average)
  developed by Box and Jenkins (1970). The use of
  such a statistical analysis requires the
  availability of many data points.
• The usual tests of significance applied to time
  measures can yield spurious results. One reason
  is that such data tend to be highly variable, and
  it is as easy to misinterpret changes not due to
  X as due to X.

25
Multiple Time Series Design

• Design 22.3
• This design has the advantage of eliminating the
  history effect by including a control group
  comprised of an equivalentor at least
  comparablegroup of participants who do not
  receives the treatment condition. Consequently,
  the design offers a greater degree of control
  over sources of alternative explanations or rival
  hypotheses.

26
Single Subject Experimental Designs

• The majority of todays behavioral research
  involves using groups of participants. However,
  there are other approaches.
• The single-subject designs are sometimes referred
  to as the N1 design. They are an extension of
  the interrupted time series design. Where the
  interrupted time series generally looks at a
  group of individuals over time.

27
Single Subject Experimental Designs

• Common characteristics
• Only one or a few participants are used in the
  study.
• Each subject participants in a number of trials
  (repeated measures).
• Randomization procedures are hardly ever used.

28
Single Subject Experimental Designs

• These design observe the organisms behavior
  before the experimental treatment and use the
  observations as a baseline measure. Observations
  taken after the treatment are then compared to
  the baseline observations. The participant serves
  as his or her own control.

29
Single Subject Experimental Designs

• Behavioral scientists doing research before the
  development of modern statistics attempted to
  solve the problem of reliability and validity by
  making extensive observations and frequent
  replication of results. This is a traditional
  procedure used by researchers doing
  single-subject experiments.
• The assumption is that individual participants
  are essentially equivalent and that one should
  study additional participants only to make
  certain that the original subject was within the
  norm.

30
Single Subject Experimental Designs

• The single-subject approach assumes that the
  variance in the subjects behavior is dictated by
  the situation. As a result, this variance can be
  removed through careful experimental control.
• The group difference research attitude assumes
  that the bulk of the variability is inherent and
  can be controlled and analyzed statistically.

31
Some Advantages of Doing Single-Subject Studies

• In Figure 22.3, if group-oriented research is
  employed, two groups have the same means and
  measures of variability. But visual inspection
  for the data shows a trend pattern vs. a random
  pattern. The single-subject approach does not
  have this problem, because a participant is
  studied extensively over time. The cumulative
  record for that participant shows the actual
  performance of the participant.

32
Some Advantages of Doing Single-Subject Studies

• Statistical significance and practical
  significance are two different things. The
  experiment may have little practical significance
  even if it has plenty of statistical
  significance.
• Simon (1987) advocates using well-constructed
  designs with the number of participants necessary
  to find the strongest effects. Single-subject
  researcher, on the other hand, favor increasing
  the size of the effect rather than attempting to
  lower error variance. They feel that this can be
  done through tighter control over the experiment.

33
Some Advantages of Doing Single-Subject Studies

• With single-subject studies, the researcher can
  avoid some of the ethical problems that face
  group-oriented researchers. One such ethical
  problem concerns the control group, which does
  not receive any real treatment.
• If there are not enough participants of a certain
  characteristic available for study, the
  researcher can consider single-subject designs
  instead of abandoning the study.

34
Some Disadvantages of Doing Single-Subject Studies

• One of the more general problems with the
  single-subject paradigm is external validity.
  Some find it difficult to believe that the
  findings from one study using one subject can be
  generalized to an entire population.
• With repeated trials on one participant, one can
  question whether the treatment would be equally
  effective for a participant who has not
  experienced previous treatments.

35
Some Disadvantages of Doing Single-Subject Studies

• Single-subject studies are perhaps even more
  sensitive to aberrations on the part of the
  experimenter and participant. These studies are
  effective only if the researcher can avoid biases
  and the participant is motivated and cooperative.
• A researcher doing single-subject research could
  be affected more so than the group-oriented
  researcher and needs to develop a system of
  checks and balances to avoid this pitfall.

36
Some Single-Subject Research Paradigms

• The Stable Baseline An Important Goal
• The behavior before the treatment intervention
  must be measured over a long enough time period
  so that a stable baseline can be obtained. This
  baseline, or operant level, is important because
  it is compared to later behavior.
• If the baseline varies considerably, it could be
  more difficult to assess any reliable change in
  behavior following intervention.

37
Designs that Use the Withdrawal of Treatment

• The ABA Design
• The ABA design involves three major steps. The
  first step is to establish a stable baseline (A).
  The experimental intervention is applied to the
  participant in the second step (B). If the
  treatment is effective, there will be a response
  difference from the baseline. In order to
  determine if the treatment intervention caused
  the change in behavior, the researcher exercises
  step three a return to baseline (A).

38
Designs that Use the Withdrawal of Treatment

• There are also some ethical concerns about
  reverting the organism back to the original state
  if that state was an undesirable behavior.
  Experiments in behavior modification seldom
  return the participant back to baseline. To
  benefit the participant, the treatment is
  reintroduced. The ABAB design does this.

39
Designs that Use the Withdrawal of Treatment

• Repeating Treatments (ABAB Designs)
• There are two versions of the ABAB design. The
  first was briefly described in the above section.
  Repeating the treatment also provides the
  experimenter with additional information about
  the strength of the treatment intervention.
• The ABAB design essentially produces the
  experimental effect twice.

40
Designs that Use the Withdrawal of Treatment

• The second variation of the ABAB design is called
  the alternating treatments design. In this
  variation there is no baseline taken. The A and B
  in this design are two different treatments that
  are alternated at random. The goal of this design
  is to evaluate the relative effectiveness of the
  two treatment interventions.
• The advantage this design has over the first ABAB
  design is that there is no baseline to be taken,
  and the participant is not subjected to
  withdrawal procedures. Since this method involves
  comparing two sets of series of data, some have
  called it the between-series design.

41
Designs that Use the Withdrawal of Treatment

• There are some other interesting variations of
  the ABAB design where withdrawal of the treatment
  is not done. McGuigan (1996) calls it the ABCB
  design where in the third phase, the organism is
  given a placebo condition. This placebo
  condition is essentially a different method.

42
A Research Example

• Powell and Nelson (1997) Example of an ABAB
  Design
• The treatment intervention was letting Evan
  choose the class assignment he wanted to work on.
  There are two conditions choice and no-choice.
  Baseline data were collected during the no-choice
  phase.

43
Using Multiple Baselines

• There is a form of single-subject research that
  uses more than one baseline. Several different
  baselines are established before treatment is
  given to the participant. These types of studies
  are called multiple baseline studies.
• There are three classes of multiple baseline
  research designs across behaviors, across
  participants, and across environments.

44
Using Multiple Baselines

• There is a common pattern for implementing all
  three classes of this design. That pattern is
  shown in Figure 22.4.

45
Using Multiple Baselines

• With the multiple baselines across behaviors, the
  treatment intervention for each different
  behavior is introduced at different times. If one
  of the behavior changes, while the other
  behaviors remain constant or stable at the
  baseline, the researcher could state that the
  treatment was effective for specific behavior.
• After a certain period of time has passed, the
  same treatment is applied to the second
  undesirable behavior (Baseline 2).

46
Using Multiple Baselines

• An important consideration with this particular
  class of multiple baseline design is that one
  assumes the responses for each behavior are
  independent of the responses for other behaviors.
  
• The intervention can be considered effective if
  this independence exists. If the responses are in
  some way correlated, then the interpretation of
  the results becomes more difficult.

47
Using Multiple Baselines

• In the multiple baseline design across
  participants, the same treatment is applied in
  series to the same behavior of different
  individuals in the same environment.
• In the multiple baseline design across
  environments, the same treatment is given to
  different participants who are in different
  environment.