Quasi and NonExperimental Designs

1
Quasi- and Non-Experimental Designs

• James R. Foreit, Ph.D.
• Operations Research Proposal
• Development Workshop
• May 3, 2006

2
Characteristics of an Experimental Design

• Manipulation of intervention (time order)
• Comparison of experimental and control groups
• Control of threats to validity
• - Random Assignment

3
Characteristics of a Quasi-Experimental Design

• Manipulation of an independent variable
• Comparison between groups, time periods
• No random assignment

4
Difference Between Quasi- and True Experimental
Designs

• A true experimental design uses random assignment
  to protect against sources of invalidity
• A quasi-experimental design does not use random
  assignment and cannot protect against many types
  of invalidity
• A true experimental design demonstrates
  causality a quasi-experimental design does not

5
You Cannot Always Use an Experimental Design

• Units cannot be randomly assigned to organismic
  variables
• You may have a very small sample
• Political, ethical and administrative reasons No
  one will randomly assign a public health program
• Fear of contamination may prevent random
  assignment

6
Reliability

• Reliability refers to the consistency and
  dependability of the data.
• If I ask the same person the same question twice
  will I get the same answer?
• A reliable measure is one that if repeated a
  second time will give the same results as it did
  the first time
• Types of reliability Test-retest inter-rater,
  consistency

7
Validity

• Validity refers to measurements that are not only
  reliable but also true and accurate
• A valid measurement measures what it is supposed
  to measure
• A valid measure is also reliable
• A reliable measure is not always valid

8
Validity Concerns

• Internal validity Did the experimental
  treatment make a difference in this specific
  study?
• External validity To what programs, settings
  and populations can the results of the study be
  generalized?

9
Factors Commonly Jeopardizing Internal Validity
in OR Studies

• Selection Bias
• History
• Testing
• Differential Mortality
• Instrumentation

10
Selection Bias

• Selection bias occurs whenever the people
  selected for the control group differ
  systematically from the experimental group
• Self-selection into groups is a common problem in
  operations research studies

11
History

• Some things happen to one group that do not
  happen to the comparison group
• Strikes
• New procedures
• A presidential address

12
Testing

• Testing bias occurs when earlier measurements
  affect the results of later measurements
• Giving identical pre-tests and post-tests to
  trainees

13
Instrumentation

• Whenever a measurement instrument is changed
  between a pre-test and a post-test

14
Differential Mortality

• If the people/units who drop out of one study
  group differ systematically from drop outs of
  other group, we do not know if results due to
  intervention or differential mortality.

15
Quasi-Experimental Designs

• Uses of Different Quasi-Experimental Designs and
  Validity Threats

16
Time Series Design

• Repeated measures on the same group over time
• No control or comparison group
• O1 O2 O3 X O4 O5 O6

17
Use of Time Series Designs

• Evaluate a mass media campaign
• Whenever you cannot use a separate control group
  (e.g., only one facility in the study)

18
Validity Threats in a Time Series Design

• A time series design does not control for
• History
• Instrumentation
• Testing
• A time series does control for
• Selection

19
Pre-test Post-testNon-equivalent Control Group
Design

• Intervention and comparison groups
• No random assignment
• O X O
• O O

20
Use of Non-equivalent Control Group Design

• When you have no more than two units to assign
  (e.g., two hospitals, two districts)
• When random assignment is not possible
• Study units should always be matched with a
  non-equivalent control group design

21
Validity Threats in a Non-equivalent Design

• A non-equivalent design does not control for
• Selection
• A non-equivalent design does control for
• History
• Testing
• Instrumentation

22
Non-experimental Designs

• Case Study
• X O
• One Group Pre-test Post-test
• X O X

23
Strengthen the Case for Your Design with Evidence

• No random assignment Any evidence of systematic
  bias in the selection?
• Time series study? Any historical event that may
  have influenced the results?

24
Operational Definitions

• Terms and variables should be defined in a way
  that permits measurement and monitoring.
• No The independent variable is group
  counseling
• Yes Groups lt 8 persons meet 2 hrs/day for 3
  consecutive days. Topics include What is HIV? (45
  minutes).

25
Monitor the Intervention

• Did the groups meet for 2 hours?
• Were all subjects covered?
• Where there fewer than 8 persons?
• Without being able to say that the intervention
  was conducted as planned, you cannot say that the
  results are due to the intervention

26
Other Monitoring Issues

• Is the intervention being conducted equally in
  all units?
• How much variation is there in the independent
  variable?
• Do groups remain equivalent?
• Are observations collected?