Poor Research Designs in Policy Impact Studies: Lies, Damn Lies, and Statistics

1
Poor Research Designs in Policy Impact Studies
Lies, Damn Lies, and Statistics

• AHRQ 2007 Annual Conference
• Improving Healthcare, Improving Lives
• September 26, 2007
• Stephen Soumerai, ScD
• Department of Ambulatory Care and Prevention
• Harvard Medical School and Harvard Pilgrim
  Health Care

2
Weak Designs No Causal InferenceWhat Threats to
Validity?
Posttest-Only Design with Nonequivalent
Groups (cross-sectional)
Treatment Group Comparison Group
Pretest-Posttest Design
Treatment Group
3
Strong Quasi-Experimental Designs
4
Reported Effectiveness of Printed Education
Materials Alone in Well-Designed vs. Inadequately
Controlled Studies
Adequately Controlled Studies
Inadequately Controlled Studies
5
Example of an Inadequate Cross-sectional Design
Problems with Small Samples and Outliers

• Effect Attributed by MCOP

6
Post-only Evaluations of Several Drug
Cost-Containment Policies on Medication Use and
Health Outcomes

• Conclusions Mixed effects on outcomes and costs
• Design Post-policy comparisons of several groups
  (e.g., with and without employer insurance)
• No data on baseline comparability
• Statistical adjustment for group differences
• Problems
• Study groups were already different with respect
  to SES and health status
• Instrumental variables, propensity scores, etc.
  cant fully control for bias

7
Use Longitudinal Models

• Increases statistical power in quasi-experimental
  studies
• Uses information on trends
• Multiple pre- and post-measurements of outcomes
• Provide graphical evidence visible versus
  statistical

8
Figure 1 Reductions in benzodiazepine use after
Triplicate Prescription Policy among patients
living in neighborhoods with different racial
compositions
Triplicate Prescription Policy
9
Benzodiazepine Use and Incidence of Hip Fracture
among Women in Medicaid Before and After NY
Regulatory Surveillance
Bz Use among Female Users before Policy,
Cumulative Incidence of Hip Fracture per 100000
Female Users before Policy
10
Summary Points

• Longitudinal data allow for strong
  quasi-experimental designs
• Provide more valid results
• Visible effects almost always significant
• Creative use of comparison series
• Unexposed comparison population
• High risk subgroups
• Unintended outcomes