Quasi-Experimental Designs

1
Quasi-Experimental Designs

• Manipulated Treatment VariablebutGroups Not
  Equated

2
Pretest-Posttest Nonequivalent Groups Design

• N  O  X  O
• N  O     O
• Cannot assume that the populations are equivalent
  prior to treatment.
• Selection and Selection Interactions are threats
  to internal validity.
• Can try to select subjects or intact groups in
  ways that make it likely that the groups are
  similar, but what about unknown variables on
  which the groups may differ.

3
Double-Pretest Nonequivalent Groups Design

• N  O  O  X  O
• N  O  O     O
• Some control for Selection x Maturation.
• If groups are maturing at different rates, that
  may be shown in the two pretests.

4
Regression-Discontinuity Design

• C  O  X  O
• C  O     O
• C indicates subjects are assigned to groups
  based on score on covariate.
• Groups are deliberately nonequivalent.
• I shall illustrate with a hypothetical example

5
Evaluating an Online Tutorial

• IV Student completes online tutorial or not.
• DV Students score on statistics course exam.
• Pretest/Covariate Students score on a test of
  statistics aptitude.
• How do I assign students to groups?

6
Evaluating an Online Tutorial

• Let the students self-select into groups.
• This gives me a pretest-posttest nonequivalent
  groups design.
• I use pretest scores as covariate in ANCOV.
• This does not, however, remove all possible
  confounds.
• How might the groups have differed other than on
  statistics aptitude???

7
Evaluating an Online Tutorial

• Randomly assign students to groups.
• This would be an experimentally sound, randomized
  pretest-posttest control group design.
• And you would live to regret trying it.
• Students complain.
• Their parents complain.
• The Chair of the Department intervenes.
• The IRB revokes your authority to do research.

8
Evaluating an Online Tutorial

• Try a switching-replications design.
• Those who have to wait until the second half of
  the class would be disadvantaged
• if you dont learn the beginning material well,
  the later material will very hard to learn.
• Those who have it taken away at mid-semester will
  complain.

9
Evaluating an Online Tutorial

• Apply the treatment only to those most in need
  of it, those lowest in statistics aptitude.
• Those not selected may complain that they could
  benefit from it too.
• Tough, there is an American tradition of favoring
  the underdog.

10
Evaluating an Online Tutorial

• Those selected may complain about having to do
  extra work.
• You cant please everybody every time. Convince
  them they need to do it.
• May be cases when you want to give the tutorial
  only to those with highest aptitude
• purpose of tutorial is to allow brightest
  students to finish course early, allowing prof
  more time to spend in class with others.

11
Evaluating an Online Tutorial

• Suppose I pick a cutoff on the covariate so that
  ½ get the treatment, ½ dont.
• Simulated data are in file RegD0.txt .
• C control group, T Treatment group.
• 2nd score is posttest score.
• 3rd score is pretest score.
• I defined the treatment effect to be zero in the
  population.

12
Evaluating an Online Tutorial

• In the population, Post  7  1.35?Pre  error,
  ? .9,
• In the sample, ignoring group, Post  7.58  1.27?
  Pre  error, r .85, and MSE 2.13
• Look at this plot of the data

13
(No Transcript)
14
Evaluating an Online Tutorial

• Now I compute two separate regressions, one for
  each group.
• T Post  8.09  1.17?Pre  error, r .62, and
  MSE 2.13.
• C Post  6.33  1.43?Pre  error, r .72, and
  MSE 2.29.
• The plot shows how little the two lines differ

15
(No Transcript)
16
Evaluating an Online Tutorial

• I re-simulated the data, with a 3 point treatment
  effect built in.
• The data are at RegD1.txt.
• T Post  11.27  1.07?Pre  error, r .82, and
  MSE 1.35.
• C Post  7.90  1.18?Pre  error, r .82, and
  MSE 1.25
• The plot shows a clear regression discontinuity

17
(No Transcript)
18
Evaluating an Online Tutorial

• The dotted line shows the expected regression for
  the treatment group if there were no treatment
  effect.
• Hard to imagine how any threat to internal
  validity would create the observed regression
  discontinuity.
• Caution This analysis assumes the regression is
  linear, not curvilinear.

19
Proxy-Pretest Design

• N  O1  X  O2
• N  O1     O2
• You have a nonequivalent groups posttest only
  control group design.
• The treatment has already been administered.
• Now you decide you want a pretest too.
• Cant warp time, can find an archival proxy
  pretest.

20
PSYC 2210 and Understanding Stats

• Mid-semester, I ask myself does taking 2210
  improve students understanding of stats?
• Ill compare students in current 2210 class with
  those in another class (excluding any who have
  already taken 2210).
• I want a pretest too, but the treatment is
  already in progress.

21
PSYC 2210 and Understanding Stats

• I use, as a proxy pretest, students final
  averages from PSYC 2101.
• Conduct an ANCOV
• IV took 2210 or not
• DV end of course stats achievement test
• COV the proxy pretest

22
Separate Pre-Post Samples Design

• Pretest subjects different than posttest
  subjects.
• I want to evaluate online tutorial in stats.
• Both I and my friend Linda taught stats this
  semester and last semester.
• Both of us gave our students a end of course
  standardized exam.

• N  O
• N     X  O
• N  O
• N        O

23
Separate Pre-Post Samples Design

• Row 1 My students last semester, no tutorial.
• Row 2 My students this semester, with tutorial.
• Row 3 Lindas students last semester, no
  tutorial
• Row 4 Lindas students last semester, no
  tutorial.
• Selection problems likely.

• N  O
• N     X  O
• N  O
• N        O

24
Nonequivalent Groups Switching Replications Design

• N  O  X  O     O
• N  O     O  X  O
• I am teaching two sections of stats.
• I make the experimental tutorial available the
  first half semester to one class
• and the second half semester to the other.
• Might reduce complaints, until students from the
  two classes meet each other.

25
Nonequivalent Dependent Variables Design

• Only one group of subject, but two DVs.
• One DV you expect to be affected by X.
• The other you expect not to be affected by X.
• The second DV serves as a control variable.
• Should be similar enough to 1st DV that it will
  be effected in same way by history, maturation,
  etc.

26
Nonequivalent Dependent Variables Design

• I want to evaluate effect of stats remedial
  tutorial given to all PSYC 2210 students.
• DV1 stats given at start and end of semester.
• DV2 Vocabulary test given at start and end of
  semester.
• More impressive if have multiple control
  variables and an a priori prediction of extent to
  which each will be effected.

27
Nonequivalent Dependent Variables Design

• Stats Knowledge (DV1) most affected
• Logical Thinking next most
• Verbal Reasoning same as LT
• Arithmetic Skills next most
• Vocabulary next most
• Artistic Expression least affected by treatment

28
Regression Point Displacement Design

• N(n 1)  O  X  O
• N         O     O
• Only one subject in the treatment group
• Several or many in the control group.
• X novel economic development plan.
• Treatment unit your hometown, in which the plan
  was just initiated.

29
Regression Point Displacement Design

• You consult state economic database.
• Pick 20 cities comparable to your city, these
  serve as the control group.
• Pretest value of criterion variable (such as
  unemployment rate) last year.
• Posttest value of same variable two years
  later.

30
Regression Point Displacement Design

• Plot Post x Pre for the Control Group.
• Draw in regression for predicting Post from Pre.
• Plot experimental unit data point.
• If it is displaced well away from regression
  line, you have evidence of a treatment effect.

31
(No Transcript)