Quasi-Experimental Designs 101: What Works?

1
Quasi-Experimental Designs 101What Works?

• The Need To Know Team
• January 31 February 1, 2005
• Patricia J. Martens PhD

2
Outline

• Reviewing Xs and Os
• Quasi-experimental time series designs with
  comparison groups
• The Population Health Research Data Repository
  what data do we have?
• Brainstorming ideas

3
Key features of study designs

• Artificial manipulation?
• (experimental or observational)
• Experimental
• Are the groups randomly assigned to receive or
  not receive the intervention? (randomized
  controlled trial)
• Are the groups selected to be as similar as
  possible, not randomly? (quasi-experimental
  comparison groups)

4
Research Design Schema
5
Key Features of Study Designs

• Observational
• Information collected concurrently or over a time
  period? (cross-sectional or longitudinal)
• If over a time period, i.e. longitudinal, do you
  go from exposure to disease (cohort) or from
  disease back in time to examine exposures
  (case-control)?
• Do you start now and go forward (prospective), or
  do you have a cohort somewhere in the past and
  you follow them forward (historical prospective)?

6
Research Design Schema
7
Study design observational

• Cross-sectional studies
• studying all factors at once - both the
  hypothesized explanatory and outcome variables
• Prospective studies
• going forward in time, following a cohort and
  observing the effect of exposure to a future
  outcome
• Case-control studies
• going backwards in time from the cases/controls
  to look at differential exposures

8
Research Design Schema
9
Study design What Works proposal

• Randomized Controlled (Clinical) Trial
• designing a specific intervention and randomly
  assigning people to receive it or not to receive
  it
• Quasi-experimental
• using a comparison group which is not randomly
  assigned
• Each RHA is a comparison group
• A quasi-experimental time series with many
  comparison groups (all other RHAs in the
  province)
• Diagrammed and described by Campbell Stanley
  (1963)

10
Lets play Xs and Os

• X is an intervention
• O is an outcome measure
• X O

11
Lets play Xs and Os

• O X O

12
Lets play Xs and Os

• O X O
• O O

13
Lets play Xs and Os

• R means randomly assigned
• R O X O
• R O O
• (pretest-posttest control group design)

14
Lets play Xs and Os

• _ _ _ _ means not randomly assigned
  (quasi-experimental comparison)
• O X O
• - - - - - - - -
• O O

15
Lets play Xs and Os

• O X O
• - - - - - - - -
• O O
• quasi-experimental pretest- posttest design
• (non-randomized control group)
• (non-equivalent pretest-posttest comparison group
  design)

16
Examples of a quasi-experimental
pretest- posttest comparison group study to
determine effectiveness of hospital
policy/education program
17
Lets play Xs and Os

• O O X O O
• Time series (quasi experiments)

18
Example of a quasi-experimental time series to
determine effectiveness of a community-based
breastfeeding strategy
19
Lets play Xs and Os

• Time series (quasi experiment with comparison
  group)
• O O X O O
• - - - - - - - - - - - - - - -
• O O O O

20
Example of a quasi-experimental time series with
comparison groups to determine effectiveness of
a regional teen pregnancy reduction program
From CIHR proposal submission September 2004
21
Additions of small amounts of phosphorus to one
section of ELA Lake 226 caused surface blooms of
blue-green algae, and vividly demonstrated the
importance of phosphate as a cause of excessive
algal growth or eutrophication. This experiment
spurred legislation controlling the input of
phosphorus to many water bodies.
A demonstration of the work of Dr. David
Schindler and the Experimental Lakes project in
NW Ontario
http//www.umanitoba.ca/institutes/fisheries/eutro
.html
22
Study design Low internal validity

• Anecdote/case study
• Pre-experimental
• just doing a pretest and posttest on one group
  and seeing its effect
• Cross-sectional
• a snapshot in time cant tell which comes
  first, but only that they are associated

23
Study design medium internal validity

• Time series Time series with qualitative layer
• looking over time to see change, with information
  about when interventions occurred in the time
  frame
• Case-control
• going backwards in time from the cases/controls
  to look at different exposures to possible risk
  factors
• Observational (prospective)
• going forward in time, observing the effect of
  exposure on a cohort to a future outcome

24
Study design high internal validity

• Randomized Controlled (clinical) Trials, RCT
• designing a specific intervention and randomly
  assigning people to receive it or not to receive
  it
• following people to observe the outcome of
  interest
• Quasi-experimental comparison group studies
• using a comparison group which is not randomly
  assigned, but very similar at onset

25
High
Randomized Controlled Trials RCT Quasi-experimenta
l comparison group studies
Time series with comparison Observational
(prospective) Case-control Time series with
qualitative layer
Internal validity
Cross-sectional Pre-experimental Anecdote/case
study
Low
26
There is nothing so useless as doing
efficiently that which should not be done in the
first place.

• Peter Drucker

27
MCHPs paperclipsPopulation Health Research
Data Repository
Population-Based Health Registry
Census Data EA/DA level
National surveys
28
Brainstorming What Works proposal

• Pick (a) a policy and (b) a program
• Think of something that your region has done in
  the past, somewhere between 1997 and the present
  (hopefully, with a few years of data AFTER the
  onset of this)
• What OUTCOME measures would you think this would
  impact?
• Think of what you would expect to see if this
  intervention was working
• Are there specific target groups to which this
  intervention applies? (e.g. teens, people living
  in a certain district of your region?)
• What measures of this intervention would be
  available through the Repository data?
• Brainstorm and report! (see sheet for recording)

29
Policy or Program Outcome Measure(s) Target Group Outcome available in Repository? Other comments
Teen pregnancy reduction Teen pregnancy rate 12-19 year olds? Certain district? pregnancies or live births? Maybe birth control pill use in Rx data?