Validation citations

1
Validation / citations
2
Validation

• Expert review of model structure
• Expert review of basic code implementation
• Reproduce original inputs
• Correctly respond to changes in parameters
• Consistent with other analyses
• Consistent across versions

3
Documentation specifications

• Validation requires adequate documentation to
• Describe clinical model
• Describe programming algorithms
• Describe inputs (sources / values / data quality)
• Describe validation strategy

4
Citations

• Basic rationale and structure -- Matchar et al.,
  Annals of Internal Medicine (1997)
• Bootstrapped sensitivity analysis -- Parmigiani
  et al., Medical Decision Making (1997)
• Application to acute stroke trial -- Samsa et
  al., Journal of Clinical Epidemiology (in press)

5
Outline

• Rationale for modeling
• Stroke model described
• Applying the SPM to a randomized trial ()
• Extensions

6
Applying the SPM to an RCT

• Basic idea After the short follow-up period
  typical of an RCT, we are likely to observe that
  the intervention induces small shifts in the
  distribution of morality / disability (measured
  at the conclusion of follow-up).
• Modeling can be used to account for the effects
  of these shifts on long-term outcomes.

7
Disability measure

• Disability could be measured by the Rankin Scale
  (RS), Barthel Index (BI), or a number of other
  instruments.
• For concreteness, we use the RS with 5
  categories. The analysis easily extends to other
  scales and/or cut-points.

8
Risk (hazard) ratios for disability on subsequent
outcomes

• RS Utility h(IS) h(MI) h(DT) Cost
• 0-1 .80 1.00 1.00 1.00 1.00
• 2 .65 1.05 1.07 1.11 1.27
• 3 .50 1.12 1.14 1.27 1.94
• 4 .35 1.24 1.26 1.71 3.98
• 5 .20 1.30 1.31 2.37 6.01
• As derived by an expert panel, supported by
  published and unpublished literature

9
Rationale

• The utilities for RS of 0-1 and 5 were obtained
  from the PORTs patient survey.
• Other values obtained by interpolation
• Consistent with unpublished RCT data
• Increased hazard of death results from sequelae
  of inactivity such as aspiration pneumonia, etc.
• Daily costs greatly increase when disability
  level implies high likelihood of
  institutionalization.

10
SPM results by Rankin

• RS Survival QALY Cost
• 0-1 10.60 6.07 143,820
• 2 9.99 4.70 167,602
• 3 9.28 3.37 217,039
• 4 8.22 2.13 328,895
• 5 7.09 1.09 403,911
• Beginning 6 months after IS, for patients with
  mean age 70 years

11
Thought experiment -- typical RCT results (6
month follow-up)

• Intervention Placebo
• QALY 0.25 0.23
• Cost 27,000 24,000
• That is, little difference in QALY in absolute
  terms. For cost, we assume equivalent
  utilization plus the cost of the drug.

12
Typical RCT results (contd)

• RS Intervention Placebo
• 0-1 31 25
• 2 21 15
• 3 11 10
• 4 6 10
• 5 11 15
• Died 20 25

13
Typical results (contd)

• The previous results describe a break-through
  drug, but the ICER based upon 6 months of
  follow-up is only (27,000-24,000) / (0.25-0.23)
  150,000 /QALY.

14
Combining short- and long-term outcomes

• For each patient, we assign long-term outcomes
  (cost and QALYs) based upon RS at 6 months. For
  example, total costs then become 6-month costs
  (observed) expected long-term costs (simulated).

15
Long-term QALYs by group

• Intervention (.31)(6.07) (.21)(4.70)
  (.11)(3.37) (.06)(2.13) (.11)(1.09)
  (.20)(0) 3.49 QALY
• Placebo (.25)(6.07) (.15)(4.70) (.10)(3.37)
  (.10)(2.13) (.15)(1.09) (.25)(0) 2.94
  QALY
• Identical multipliers -- groups only differ in
  proportion of patients in each RS category
• Same weighted average calculations for cost

16
Comprehensive CEA

• Intervention QALY 0.253.49 3.74
• Intervention cost 27,000167,818 194,818
• Placebo QALY 0.232.94 3.17
• Placebo cost 24,000 176,275 200,275
• ICER (194,818-200,275)/(3.74-3.17) -9,573
  /QALY

17
Conclusion

• By considering its long-term effects, the
  intervention moves from not cost effective to
  cost saving, even if the price of the drug is
  increased substantially. This result holds
  across a wide range of sensitivity analyses.

18
Outline

• Rationale for modeling
• Stroke model described
• Applying the SPM to a randomized trial
• Extensions ()

19
Extensions

• International applications
• Planning trials

20
International applications

• Basic idea Use the same SPM but change the
  inputs.
• In theory, every component of the input data
  (e.g., natural history, effect of covariates,
  effect of interventions, QOL, cost) could differ
  by nation.
• In practice, the biggest differences will involve
  utilization patterns (and thus cost).

21
Comment

• The size of the task depends upon the difficulty
  in estimating the relevant parameters.
• This, in turn, depends what we are willing to
  assume versus what will require additional data
  collection.
• Once new estimates have been derived, inserting
  them into the SPM is
• straightforward.

22
International costs

• Costs are affected by
• Different utilization patterns
• Different unit prices
• Different payment mechanisms / perspectives
  (affecting which components of cost to include in
  the calculation)

23
Recommendation

• Begin with US data on utilization and unit
  prices.
• Then use expert judgement (supported by the
  literature as available) to posit any changes in
  utilization patterns.
• Attach country-specific unit prices.
• Limit the analysis to the cost categories and
  time periods of interest to decision-makers in
  that country.

24
Planning trials

• The SPM can help to determine the (minimum)
  clinically important difference in short-term
  disability.
• This, in turn, would be the basis for the trials
  sample size calculations.

25
Observation

• Many stroke trials have been under-powered.
• That is, the size of the effect which is
  significant from the perspective of public health
  / CEA is much smaller than that suggested by
  clinical intuition (Samsa et al, Journal of
  Clinical Epidemiology, in press).

26
Observation

• Many RCTs of acute stroke treatments have had
  other sources of statistical inefficiency -- for
  example, pertaining to the choice of target
  population, the choice of measure, and so forth.

27
Summation

• Because most benefits of stroke treatments accrue
  in the long-term, modeling is necessary.
• The SPM is a well-validated model which can be
  helpful in both the planning and analysis of
  trials.
• Planned extensions include international practice
  patterns and user-friendly software.