RESEARCH DESIGN PHC 6700/RCS 6740 March 7, 2006

1
RESEARCH DESIGNPHC 6700/RCS 6740March 7, 2006

•  

2
RESEARCH DESIGN

• Experimental Designs
•  
• The specific research designs available to
  investigators can be divided into two basic
  types
• group designs, and
• single-subject designs.
•  

3
RESEARCH DESIGN

• A Typical Experimental Design
•  
• Pretest-Posttest Control Group Design
•  
• R O1 X O2
• R O3 O4
•  

4
RESEARCH DESIGN

• Group Designs
•  
• The group (multi-subject) designs all include
  one or more groups of subjects and are classified
  as either
• between-groups,
• within-subjects,
• or mixed.

5
RESEARCH DESIGN

• Between-Groups Design
• Between-groups design is used to assess the
  effects of different levels of an independent
  variable by administering each level to a
  different group of subjects and then comparing
  the status or performance of the group on the
  dependent variable.
• The simplest between-groups designs include a
  single independent variable with two levels. When
  using the design, the study includes two groups
  that each receives a different level of the IV.

6
RESEARCH DESIGN

• Example
• A psychologist assesses the effects of a
  self-control procedure by comparing the
  achievement of children who have been trained in
  the procedure (experimental group) with that of
  children who have not been trained in the
  procedure (control group).

7
RESEARCH DESIGN

• The simple two-group design can be expanded in
  two ways. One way is to include more than two
  levels of a single independent variable.
• The psychologist in this study could compare
  three levels of the control procedure
• a procedure that includes self-instruction only,
• a procedure that includes self-instruction,
  self-monitoring, and self-reinforcement, and
• no procedure.
• In this situation, the study would involve
  comparing the average academic achievement test
  scores of subjects in the three groups.

8
RESEARCH DESIGN

• Another way to expand a two-group design is to
  include two or more independent variables.
• Whenever a study includes two or more independent
  variables, it is called a factorial design.
• The major advantage of a factorial design is that
  it provides more thorough information about the
  relationships among variables by allowing an
  investigator to analyze the main effects of each
  independent variable as well as the interaction
  between independent variables.

9
RESEARCH DESIGN

• If the psychologist in the self-control study
  included initial symptom severity (mild,
  moderate, and severe) as a second independent
  variable, s/he would be able to determine if
  there are
• MAIN effects of the self-control procedure,
• MAIN effects of initial symptom severity, and/or
• an INTERACTION between self-control procedure and
  initial symptom severity.

10
RESEARCH DESIGN

• Main Effect
• A main effect is the effect of one independent
  variable on the dependent variable, disregarding
  the effects of all other independent variables.
• An interaction refers to the effects of two or
  more independent variables considered together.
• An interaction occurs when the effects of an
  independent variable differ at different levels
  of another independent variable.

11
RESEARCH DESIGN

• Illustration
• Assume the psychologist in the self-control
  study obtains a sample of 60 children and divides
  them into three groups on the basis of their
  initial symptom severity (mild, moderate, or
  severe). S/he then randomly assigns subjects in
  each group to either the experimental
  (self-control procedure) or control (no
  procedure) group so that there are 10 children in
  each of the study's now six groups (see table in
  next slide)
• Although the data collected by the psychologist
  would have to be analyzed with an inferential
  statistical test to determine if there are
  significant main and/or interaction effects,
  tentative conclusions can be drawn by examining
  the data.

12
RESEARCH DESIGN

• As an example, assume that the psychologist
  obtains the following mean achievement test
  scores for the six groups of children
•  
•  
•  

13
RESEARCH DESIGN

• To determine if there are main effects of each
  IV, it is necessary to calculate the marginal
  means. For self-control procedure, the marginal
  means are 42 and 34. These means were obtained by
  adding the means in each column and dividing by 3
  (the number of means) (52 40 34)/3 42 and
  (36 30 36)/3 34.
• Because the marginal means are different, it is
  possible to tentatively conclude that there are
  main effects for the self-control procedure.
• Overall, the self-control procedure seems to have
  beneficial effects on academic achievement test
  scores.

14
RESEARCH DESIGN

• For initial symptom severity the marginal means
  are 44, 35, and 35. These means were obtained by
  adding the means in each row and dividing by 2
  (the number of means) (52 36)/2 44 (40
  30)/2 35 and (34 36)/2 35.
• The marginal means indicate that there are also
  main effects for initial symptom severity.
• Although children with moderate and severe
  symptoms obtained the same mean achievement test
  score (35), children with mild symptoms obtained
  a higher mean score (44). (If all three means
  were identical, there would be no main effects of
  symptom severity.)
• This indicates that, overall, mild symptoms are
  associated with the highest achievement test
  scores.

15
RESEARCH DESIGN
44
35
35
42 34
16
RESEARCH DESIGN

• To determine if there is an interaction, the
  cell means are inspected. If there is an
  interaction, the effects of the self-control
  procedure will differ at different levels of
  symptom severity.
• As can be seen in the previous table, for
  children with mild symptoms, the self-control
  procedure had a very positive effect
• Children who were trained in the procedure
  obtained a mean of 52,
• while children who received no training obtained
  a mean of 36.
• For children with moderate symptoms, the
  self-control procedure also had positive effects,
  but the gap between the two groups is somewhat
  smaller
• Children who received training obtained a mean of
  40,
• while children who received no training obtained
  a mean of 30.
• Finally, for children with severe symptoms, the
  self-control procedure did not have a beneficial
  effect.
• Children with severe symptoms obtained a higher
  achievement test score if they received no
  training (34 with training versus 36 with no
  training).

17
RESEARCH DESIGN

• These results suggest that there is an
  interaction between the two IVs. The effects of
  the self-control procedure differ for different
  levels of initial symptom severity. (If the
  effects of the self-control procedure were the
  identical for all levels of symptom severity,
  there would be no interaction.)

18
RESEARCH DESIGN

• An inspection of the data suggests that there
  are main effects for both variables as well as an
  interaction effect.
• Note that the presence of the interaction
  invalidates the conclusion that was drawn on the
  basis of the main effects alone
• Overall, the self-control procedure seems to be
  beneficial (main effect), but a closer inspection
  of the data shows that this is true only for
  children with mild and moderate symptoms
  (interaction).
• This illustrates the importance of interpreting
  the main effects (or lack of main effects) with
  caution whenever there is an interaction
• An interaction is likely to modify or invalidate
  the conclusions made on the basis of the main
  effects alone.
• Finally, note that a study can have any
  combination of main and interaction effects. It
  is possible to have an interaction, for example,
  without any main effects (or vice versa).

19
RESEARCH DESIGN

• Study Tip Be sure you understand what main and
  interaction effects are on a conceptual level and
  be able to determine, from a table of data like
  the one given, whether it looks like there are
  main and/or interaction effects. Keep in mind
  that there can't be an interaction unless the
  study has at least two IVs and that, when there
  is an interaction, the main effects must be
  interpreted in light of the interaction.

20
RESEARCH DESIGN

• Within-Subjects Designs
• When using a within-subjects (AKA repeated
  measures) design, all levels of the independent
  variable are administered sequentially to all
  subjects.
• Consequently, comparisons of different levels of
  the independent variable are made within subjects
  rather than between groups of subjects.
• Like between-groups designs, within-subjects
  designs can include only two levels of a single
  independent variable or can be expanded to
  include three or more levels of a single IV
  and/or two or more IVs.

21
RESEARCH DESIGN

• The single-group time-series design (AKA the
  simple interrupted time-series design) is one
  type of within-subjects design.
• When using this design, the effects of a
  treatment are evaluated by measuring the
  dependent variable several times at regular
  intervals both before and after the treatment is
  applied.
• This procedure allows subjects to act as their
  own no-treatment controls.

22
RESEARCH DESIGN

• Example A psychologist in a study assesses the
  effects of a low dose of phenothiazines on Brief
  Psychiatric Rating Scale (BPRS) scores by
  administering the BPRS to a single group of
  patients at one week intervals for two months
  before and two months after patients begin
  receiving a low dose of the drug.

23
RESEARCH DESIGN

• A shortcoming of the single-group time-series
  design is that its internal validity can be
  threatened by history
• It's possible that an external event could occur
  at about the same time the independent variable
  is applied and account for any observed
  difference in pre- and posttest scores.
• Note, however, that this design does help control
  maturation since maturational effects tend to
  occur gradually over time and can usually be
  detected in the overall pattern of pre- and
  posttest scores.

24
RESEARCH DESIGN

• In another type of within-subjects design, two
  or more levels of an independent variable are
  applied sequentially to each subject, and the
  dependent variable is measured after each level
  has been applied.

Example A psychologist compares the effects of a
low and high dose of phenothiazines on BPRS
scores by giving the low dose of the drug to a
group of patients for two months and
administering the BPRS and then giving the same
patients the high dose of phenothiazines for two
months and administering the BPRS again. The
effects of the phenothiazines will be analyzed by
comparing the BPRS scores obtained by the
patients after each dose was administered.
25
RESEARCH DESIGN

• A problem with this design is that it is
  susceptible to carryover effects (multiple
  treatment interference)
• If the psychologist this study finds that the
  high dose of phenothiazines is more effective
  than the low dose for improving symptoms, this
  could be because the high dose followed a period
  of time during which patients received the low
  dose.
• Counterbalancing can be used to control carryover
  effects.
• Carryover effects could be controlled in the
  above study by dividing the patients into two
  groups and administering the two levels of
  phenothiazines in a different order to each group
  (i.e., low dose, high dose to patients in Group
  l and high dose, low dose to patients in Group
  2). If the high dose is more effective for both
  groups of patients, its effects would not be
  attributable to carryover effects.

26
RESEARCH DESIGN

• Within-subjects designs have at least two
  advantages over between groups-designs.
• First, they require a fewer number of subjects
  and, consequently, are more "economical."
• Second, these designs help control individual
  differences that can threaten a study's internal
  validity since subjects are compared with
  themselves rather than with other subjects.
• As a result, within-subjects designs can actually
  be more powerful than between groups-designs
  (i.e., better able to detect the actual effects
  of the IV).

27
RESEARCH DESIGN

• A disadvantage of the time-series and other
  within-subjects designs is that the analysis of
  the data can be confounded by autocorrelation
  (also known as serial dependency).
• In other words, subjects' performance on the
  post-tests is likely to correlate with their
  performance on the pretests.
• Autocorrelation can inflate the value of the
  inferential statistic (e.g., the t or F), thereby
  resulting in an increased probability of a Type I
  error.
• For this reason, a number of experts recommend
  that special statistical techniques be used to
  analyze data collected in a study using this type
  of design.

28
RESEARCH DESIGN

• Study Tip Be sure you have this design linked
  with the concept of autocorrelation and know that
  autocorrelation can decrease power and inflate
  the chance of making a Type I error.

29
RESEARCH DESIGN

• Mixed Designs
•  
• A mixed ("split-plot") design combines
  between-groups and within-subjects methodologies.
  
• Counterbalanced designs can be considered a type
  of mixed design because they permit comparisons
  both between groups and within subjects.
• A design is also a mixed design when it includes
  two or more independent variables and at least
  one variable is a between-groups variable and
  another is a within-subjects variable.

30
RESEARCH DESIGN

• Example In the example study, the psychologist
  would be using a mixed design if therapy approach
  is treated as a between-groups variable (patients
  receive only one type of therapy), while
  phenothiazines is treated as a within-subjects
  variable (the placebo, low dose, and high dose
  are administered sequentially to each patient).

31
RESEARCH DESIGN

• Mixed designs are common in research studies
  that involve measuring the dependent variable
  over time or across trials.
• In this type of study, time or trials is an
  additional IV and is considered a within-subjects
  variable because comparisons on the dependent
  variable will be made within subjects across time
  or across trials.

32
RESEARCH DESIGN

• Example In our example study, the psychologist
  decides to compare the effects of four levels of
  therapy (family therapy, individual therapy, a
  combination of the two, and no therapy) by
  assigning patients to one of the levels and
  measuring the short- and long-term effects of
  therapy by administering the BPRS at two-month
  intervals for 24 months after therapy begins.
  Because the study includes a between-groups
  variable (therapy) and a within-subjects variable
  (time), it is utilizing a mixed design.

33
RESEARCH DESIGN

• Single-Subject Designs
•  
• The single-subject designs were derived
  primarily from the work of behavioral
  psychologists, especially those engaged in
  applied behavioral analysis, which combines
  behavioral principles with the techniques of
  experimental psychology to solve
  socially-relevant problems.
• While the single-subject designs are often used
  to investigate the effects of an independent
  variable on the behavior of one subject or a
  small number of subjects, they can also be used
  with groups of subjects.

34
RESEARCH DESIGN

• Several characteristics distinguish the
  single-subject designs from the group designs
• Each single-subject design includes at least one
  baseline (no treatment) phase and one treatment
  phase. As a result, each subject acts as his/her
  own no-treatment control.
• The dependent variable is measured repeatedly at
  regular intervals throughout the baseline and
  treatment phases. Repeated measurement of the DV
  helps control any maturational effects that might
  otherwise threaten the study's internal validity
  by enabling an investigator to detect those
  effects in the pattern of performance on the DV
  measure.

35
RESEARCH DESIGN

• There are a large number of single-subject
  designs, but the most commonly used are the AB
  design (and its extensions) and the multiple
  baseline design.
• AB Design
• Reversal (Withdrawal) Designs
• Multiple Baseline Designs

36
RESEARCH DESIGN

• AB Design
•  
• The simplest single-subject design is the AB
  design, which includes a single baseline (A)
  phase and a single treatment (B) phase. As in all
  single-subject designs, the dependent variable is
  measured at regular intervals during both phases.

37
RESEARCH DESIGN

• Example A psychologist decides to assess the
  effects of a low dose of phenothiazines on the
  BPRS scores of one patient using the AB design.
  She administers the BPRS to the patient at
  two-week intervals for three months during the
  baseline (A) phase of the study and for three
  months during the treatment (B) phase. The design
  of this study and its possible results are
  illustrated below

38
RESEARCH DESIGN

• Because the patient's symptoms decreased only
  after he began receiving the phenothiazines,
  these results suggest that a low dose of
  phenothiazines has a positive effect on symptoms.

39
RESEARCH DESIGN

• Reversal Designs (ABA, ABAB, Etc.)
•  
• The AB design can be expanded to include more
  than one baseline phase or more than one baseline
  and more than one treatment phase. Because any
  expansion requires the withdrawal of the
  treatment during the second and subsequent
  baseline phases, the extensions of the AB design
  are called reversal (withdrawal) designs.

40
RESEARCH DESIGN

• Advantage of the reversal designs over the
  simple AB design they provide additional control
  over potential threats to a study's internal
  validity.
• When an ABAB design is used, if status on the DV
  returns to the initial baseline (no treatment)
  level during the second A phase and then to its
  previous treatment levels during the second B
  phase, an investigator can be more certain that
  any observed change in the dependent variable is
  actually due to the IV rather than to an
  historical event or other extraneous factor.
• Repeating the study by adding another A and B
  phase is referred to as intrasubject replication.

41
RESEARCH DESIGN

• Example To confirm that the observed
  improvement in symptoms was due to the
  phenothiazines, the psychologist extends her
  study to include an additional baseline and
  treatment phase. This ABAB design and its
  possible results are illustrated below

42
RESEARCH DESIGN

• Reversal designs are considered inappropriate
  when withdrawal of a treatment during the course
  of a research study would be unethical (e.g.,
  when the treatment has successfully eliminated a
  self-injurious behavior).
• Reversal design does not provide conclusive
  information if the effects of an independent
  variable persist rather than "reverse" (return to
  baseline levels) when it is withdrawn.
• When this occurs, an investigator can't be
  certain whether an observed effect on the
  dependent variable is due to the independent
  variable or to other factors.
• If a reversal design is inappropriate for
  ethical or practical reasons, an investigator
  might use a multiple baseline design.

43
RESEARCH DESIGN

• Multiple Baseline Design
•  
• Multiple baseline design does not require
  withdrawal of a treatment during the course of
  the study but, instead, involves sequentially
  applying the treatment either to different
  behaviors of the same subject (multiple baseline
  across behaviors) to the same subject in
  different settings (multiple baseline across
  settings) or to the same behavior of different
  subjects (multiple baseline across subjects).
• Once the treatment has been applied to a
  "baseline" (behavior, setting, or subject), it is
  not withdrawn from that baseline during the
  course of the study.

44
RESEARCH DESIGN

• Example To test the effects of the
  self-instructional component of the self-control
  procedure on attention span, a psychologist uses
  a multiple baseline design. S/he trains a child
  in self-instruction and then tells the child to
  use the technique when working on arithmetic
  homework in three different settings first when
  working alone in a quiet room then when working
  in the library and then when working in the
  classroom. The psychologist measures the child's
  attention span in all three settings at regular
  intervals during the baseline and treatment
  phases.

45
RESEARCH DESIGN

• This design and its possible results are
  illustrated below

46
RESEARCH DESIGN

• Because the child's attention span in each
  setting did not increase until self-instruction
  was applied to it, these results confirm that
  self-instruction is useful for improving
  attention span.
• As can be seen in the above figure, sequentially
  applying an intervention to different settings
  helps determine if an intervention is actually
  responsible for any observed changes in the
  target behavior

47
RESEARCH DESIGN

• If the behavior changes in a particular setting
  only after the intervention has been applied in
  that setting, an investigator can be more certain
  that the change is due to the intervention rather
  than to history or other factors.
• To be effective, the setting, behaviors, or
  subjects chosen for inclusion in the study must
  be relatively independent. If they are not, it
  may not be possible to evaluate the effects of
  the IV with the multiple baseline design.

48
HAPPY SPRING BREAK! ?