Experimental Research Sample Designs

1
Experimental Research Sample Designs

• Trista Perez
• Katie Byington

2
Subject Selection

• Random Selection
• The equal probability that subjects within the
  population can be selected (no bias)
• Increases external validity results more likely
  to be generalizable to population

3
Sample of Subjects

• Diversity of Sample
• Much research uses males
• Extreme reliance on college students (PY101)
• Samples of Convenience
• Set of subjects studied because they are
  available or it is convenient
• Should always provide rationale as to why certain
  sample was selected

4
Random Assignment

• Involves assigning participants to groups in an
  unbiased manner
• Addresses the need for internal validity
• Selectionhistory
• Selectionmaturation
• Unintended influences

5
Group Equivalence

• Random assignment only ensures that participants
  have equal chance to be in each group in the
  study
• Nuisance variables - characteristics that are not
  of interest in study but may influence results
• Random assignment helps but does not ensure that
  groups are equal
• Should aim to have groups of more than 40
  subjects

6
Matching

• When specific characteristic relates to scores on
  dependent variable, investigator may choose to
  match subjects
• Matching grouping subjects together on basis of
  similar characteristic(s)
• Identical pre scores rank order male/female
• Subjects matched on variables assumed to be
  related to performance on dependent measure

7
Selected Group Designs

• Defining characteristic of group experiments
  Subjects assigned to groups in an unbiased
  fashion
• Types
• Pretest-Posttest Control Group Design
• Posttest-Only Control Group Design
• Solomon Four-Group Design
• Factorial Designs
• Quasi-Experimental Designs

8
Pretest-Posttest Control Group Design

• Minimum of 2 groups
• 1 receives treatment 1 does not
• Participants tested before and after intervention
• Random assignment to groups either before or
  after completion of pretest
• R O1 x O2
• R O3 O4

9
Pretest-Posttest Control Group Design

• Strengths
• Controls for threats to internal validity
• Attrition not a big problem
• Advantages of using pretest
• Participants can be matched on pretest variables
• Allows evaluation of that matched variable in the
  results
• Increases statistical power of the test
• Allows examination of which participants changed
  and what proportion changed in a certain way
• Allows evaluation of attrition

10
Pretest-Posttest Control Group Design

• Weaknesses
• Main weakness influence of administering
  pretest
• Intervening events and processes (e.g. history,
  maturation) between pretest and posttest
• So
• Strengths outweigh weaknesses for this design

11
Posttest only Control Group Design

• Minimum of 2 groups
• 1 control group 1 experimental group
• Participants tested only after they have received
  the intervention
• Random assignment to groups must occur before the
  intervention begins
• R X O1
• R O2

12
Strengths

• Controls for threats to internal validity
• No threat of sensitization to procedure as with
  Pre-post test design
• Pretest might not be necessary
• Large N
• Budget issues
• Time
• Ethical issues

13
Weaknesses

• Absence of pretest not good for clinical studies
• Need to know pre-intervention functioning
• Group differences after intervention could be due
  to differences in groups prior to intervention
• Compare pretest performance to post intervention
• Loss of statistical power
• In summary only a good idea if you have a large
  N or can justify the absence of a pretest

14
Solomon Four Group Design

• Minimum of 4 groups
• 2 groups from the pretest-posttest design, 2
  groups from the posttest only design
• Purpose is to test effects of having a pretest on
  the post intervention results
• R O1 X O2
• R O3 O4
• R X O5
• R O6
• Treat as 2 X 2 factorial design
• (Treatment Group X Time)

15
Strengths

• Controls for threats to internal validity
• Interaction between pretest and intervention can
  be tested to see if pretesting has an effect on
  post performance
• There are several replications of treatment and
  control
• Within group
• pre vs. post
• Between group
• Experimental vs. Control with pretest
• Experimental vs. Control without pretest

16
Weaknesses

• Requires 4 groups
• Larger N
• Larger budget
• More complicated design

17
Factorial Designs

• Allows the simultaneous investigation of 2 or
  more variables (or factors) in an experiment
• 2 or more conditions within each variable
• Simplest factorial design 2 variables with 2
  different levels
• 2x2 design therapist (inexperienced vs.
  experienced) and type of treatment (A vs. B)

18
Factorial Designs

• A family of designs in which vary by the number
  and types of variables and number of levels
  within each variable
• If pretest is used, testing can become one of the
  variables with 2 or more levels (e.g. pretest vs.
  posttest)
• Multiple variables are included to look at
  separate and combined effects of different
  variables

19
Factorial Designs

• Strengths
• Can assess effects of separate variables in a
  single experiment
• Different variables can be studied with fewer
  participants and observations
• Provides unique information about the combined
  effects of the independent variables

20
Factorial Designs

• Weaknesses
• Number of groups in investigation multiplies
  quickly as new factors or new levels are added
• Interpreting results can be tricky
• So
• Useful for evaluating the separate and combined
  effects of variables of interest when these
  variables are conceptually related

21
Quasi-Experimental Designs

• Research designs in which the investigator cannot
  exert control required of true experiments
• Must take care in selecting controls and
  analyzing data
• Pretest-Posttest Design
• Control group not necessarily equivalent to
  experimental group (non-randomly assigned)

22
Quasi-Experimental Designs

• Posttest-Only Design
• Quasi-experimental design in which a pretest is
  not used
• Problems with this design
• The equivalence of groups prior to intervention
  cannot be assessed
• In the posttest only experimental design, absence
  of a pretest not bad because of random assignment
  
• Although there are problems with this design,
  sometimes it is the only option

23
Variations in Selected Group Designs

• Groups receive pre- and post-treatment assessment
  s at different time points
• The bad pretest equivalence of groups cannot
  actually be determined
• The good in institutional settings, staggering
  assessments can keep patients in different
  conditions from interacting
• All possible quasi-experimental designs cannot be
  described

24
Multiple-Treatment Designs

• Each of the different treatments under
  investigation is presented to each subject
• Evaluation of treatments is within subjects but
  there are usually separate groups of subjects
  present in the design
• The version of multiple-treatment design used
  depends on the number of treatments and the way
  in which they are presented
• All can be called counterbalanced designs because
  they try to balance the order of treatment across
  subjects

25
Crossover Design

• Type of multiple-treatment design
• Minimum of 2 groups
• Only differ in the order of treatments
• Used frequently in treatment studies, halfway
  through the intervention groups switch
  experimental conditions
• Random assignment before the intervention begins
• Subjects assessed after each intervention
• R O1 XA O2 XB O3
• R O4 XB O5 XA O6

26
Crossover Design

• Strengths
• Allows for comparison of two different treatments
  in the same person
• Each subject acts as his/her own control
• Medications can be studied independently since
  there is a washout period between them

27
Crossover Design

• Weaknesses
• Cannot track long term effects of treatment
• Treatments that cure illnesses cannot be tested
  after one another or before placebo
• There could be carryover effects even with a
  washout period

28
Multiple-Treatment Counterbalanced Design

• Simple design typically two treatments
• Each client receives all treatments, but in a
  different order (i.e. treatments counterbalanced)
• Becomes more complex with more treatments
• Example comparing 4 treatments (A, B, C, D)

29
Multiple-Treatment Counterbalanced Design

• Weaknesses
• Not all sequences of treatments are represented
  (in the table rows) and not every treatment is
  preceded and followed by every other treatment
• Not possible to rule out the influence of
  different sequences as a contributor

30
Consideration in Using Multiple Treatment Designs

• Order and sequence effects
• Independent and Dependent variables
• Ceiling and floor effects

31
Order and Sequence Effects

• With multiple treatments single group designs
  require counterbalancing
• Order effect - point in time in which the
  treatment occurred might be responsible for
  pattern in results
• Sequence effect arrangement of treatments
  contribute to their effects

32
Independent and Dependent Variables

• Must consider variables when determining which to
  include in multiple treatment designs

33
Ceiling and Floor Effects

• Change in dependent variable may have an upper
  and lower limit

34
Summary and Conclusions

• Selecting subjects and assigning them to
  conditions
• Selecting an appropriate design for your study
• ______________________________________
• Which designs do you think are most effective?
• Which do you think are generally ineffective?
• Which designs have you had success with in the
  past?