Economic evaluations alongside clinical trials: what they contribute, how they are performed and their limitations

1
Economic evaluations alongside clinical trials
what they contribute, how they are performed and
their limitations
Dr. Stavros Petrou Presentation to Nottingham
CTU 19th June 2008
2
What is economic evaluation?

• Premise scarce (health care) resources
• Aim to maximise health gain with the available
  resources
• Method compare cost and consequences of
  interventions
• Balance about costs and consequences, inputs and
  outputs
• Economic evaluation explicit criteria for making
  choices.

3
Definition of economic evaluation

• Definition of economic evaluation
• The comparative analysis of alternative courses
  of action in terms of both their costs and their
  consequences (Drummond et al, 1997)
• Requires
• a comparison of two or more alternatives
• examination of both costs and consequences

4
Types of economic evaluation
Is there good evidence on effectiveness of
interventions being compared?
NO
YES
Is effectiveness of interventions equal?
Costing study
YES
NO
Cost minimization study
YES
Can all outcomes be valued in monetary terms (
e.g. willingness to pay)?
Cost benefit analysis
NO
NO
Cost-effectiveness analysis
Can outcomes be measured as quality adjusted
life years?
YES
Cost-utility analysis
5
Economic evaluation alongside trials
6
Economic evaluation alongside trials
Two independent groups
Control group
Treatment group
Mean ( CT, ET )
Mean ( CC, EC )
Incremental cost-effectiveness ratio
CT - CC
ET - EC
7
The cost-effectiveness plane
Existing treatmentdominates
New treatment more effectivebut more costly
New treatment dominates
New treatment less costly but less effective
8
The cost-effectiveness plane
9
NICE method of operation

• Preferred measure of cost-effectiveness
• Quality-adjusted life year (QALY)
• Alternatives - e.g. cost per life year gained -
  acceptable
• No absolute threshold for level of acceptability
• no empirical basis for setting a value
• may in some circumstances want to ignore
  threshold
• A set threshold implies efficiency has absolute
  priority over other objectives (e.g. fairness)
• many technology suppliers are monopolies a
  threshold would discourage price competition

10
Cost per QALY results from NICE
Those in bold rejected Source N Devlin, D
Parkin. Does NICE have a cost-effectiveness
threshold and what other factors influence its
decisions? A binary choice analysis. Health
Economics 2004 13(5)437-52.
11
Why QALYs as a measure of outcome?

• To use cost-effectiveness as a guide to
  decision-making, we need to compare the c-e of
  different uses of resources
• Therefore we need an effectiveness measure that
  can be used in a wide range of settings
• Life-years gained
• but only where survival is main outcome
• Quality adjusted life years (QALYs)
• Composite of survival and quality of life

12
1.0
QALYs gained
Quality of life valuation u
Health profile with intervention
Health profile without intervention
0
1
2
3
4
5
6
Time in years t
13
The EuroQol EQ-5D The following questions are
designed to tell us about your state of health
today. Please look at each group of questions,
and then tick the statement which best describes
you own health state today. You should make five
ticks in all, one for each group.
Mobility I have no problems walking
about ____ I have some problems walking
about ____ I am confined to bed ____
Self-care I have no problems with
self-care ____ I have some problems washing
or dressing myself ____ I am unable to wash
or dress myself ____ Usual activities I
have no problems performing my usual
activities ____ I have some problems with
performing my usual activities ____ I am
unable to perform my usual activities ____ Pai
n/discomfort I have no pain or
discomfort ____ I have moderate pain or
discomfort ____ I have extreme pain or
discomfort ____ Anxiety/depression I am
not anxious or depressed ____ I am
moderately anxious or depressed ____ I am
extremely anxious or depressed ____
14
Tariffs for the EuroQol EQ-5D
Coefficient
Constant 0.081
Mobility
- Some problems 0.069
- Confined to bed 0.314
Self care
- Some problems 0.104
- Unable to wash/dress 0.214
Usual activities
- Some problems 0.036
- Unable to perform 0.094
Pain/discomfort
- Moderate 0.123
Extreme 0.386
Anxiety/depression
- Moderate 0.071
Extreme 0.236
N3 (Level 3 at least once) 0.269
15
Why randomised trials?

• Most treatments do not have large effects
  reliably detecting moderate effects requires
  studies that simultaneously avoid
• moderate bias
• proper randomisation
• intention-to-treat analysis
• avoidance of inappropriate sub-group analysis
• moderate random error
• adequate size

16
Why economic assessment in clinical trials?

• Many health economists advocate models using lots
  of data sources trial, non-trial, summary data
  etc
• But...issues of bias and random error also affect
  incremental resource use and health outcomes
• And, trials provide patient-level data, useful
  for
• Dealing with patient heterogeneity
• Examining covariance, e.g. between costs and
  outcomes
• Building and validating models, eg to extrapolate
  
• Trials allow prospective measurement of resources
  outcomes of interest
• Incremental cost of economic evaluation alongside
  trials is low

17
UK Collaborative ECMO Trial

• Pragmatic RCT
• 185 mature (?35 weeks, ?2kgs) infants with severe
  respiratory failure (ox. index ?40)
• Infants recruited from 55 centres in 1993-5
• Randomisation to ECMO 1 of 5 specialist centres,
  cannulated, ECMO support
• Randomisation to CM conventional care
• Outcomes survival without severe disability up
  to 7 years of age

18
(No Transcript)
19
Design and analytical issues

• Costs measurement and valuation
• Consequences measurement and valuation
• Analytical issues within and beyond RCTs

20
Perspectives and types of costs

• Direct costs
• Health care system
• Other care inputs, e.g. social services
• Patient, family, carer expenses
• Informal care costs
• Opportunity cost of unpaid informal care
• Indirect costs
• Time off work, reduced productivity
• Early retirement
• Premature mortality
• Transfer payments
• Payments such as social security benefits that
  redistribute output with no exchange of goods or
  services

21
Three elements of cost

• Resource use (cost generating event)
• a day in hospital
• a GP consultation
• Unit cost
• cost per in-patient day / per hospitalisation
• cost per GP consultation / per GP minute
• Cost
• the product of resource use and unit costs

22
Measurement of resource use

• All significant resource inputs during first 7
  years of life
• Trial data collection forms
• transport (mode, distances)
• duration and intensity of neonatal care
• Observational research for infant death
• Postal questionnaires, validated by information
  from hospital records
• post-discharge hospital and social service
  utilisation
• GP records
• community service utilisation

23
Valuation of resource use

• Unit costs employed to value resource use
• Neonatal care top down methodology
• Readmissions - Reference cost schedules
• Community service utilisation - Published cost
  data (previous studies, PSSRU, etc.)
• Drugs BNF
• , 2002-3 prices

24
Time horizon and discounting

• Should extend far enough into the future to
  capture all costs and consequences of
  interventions being evaluated
• ECMO Study time horizon initially reflected
  that of RCT
• Costs (and consequences) occurring beyond the
  first year of life must be reduced to present
  values
• Rationale for discounting- Time preference-
  Opportunity cost market basis

25
Mean costs and mean cost differences
Cost category ECMO (n93) ECMO (n93) CM (n92) CM (n92)
Mean (SD) Mean (SD) Mean difference P value
Death 434 (665) 846 (714) -412 lt0.0001
Transport 2147 (3080) 296 (777) 1851 lt0.0001
Initial hospitalisation 22996 (21618) 6143 (13144) 16853 lt0.0001
Hospital readmissions 1518 (3868) 1127 (3063) 391 0.447
Outpatient hospital care 970 (1592) 496 (1008) 474 0.017
Community care 1976 (2882) 1247 (2647) 792 0.075
Other costs 229 (1255) 74 (182) 155 0.241
Total costs 30270 (24380) 10229 (18356) 20041 lt0.0001
Source Petrou S, Bischof M, Bennett C, Elbourne
D, Field D, McNally H. Cost-effectiveness of
neonatal ECMO based on seven year results from
the UK Collaborative ECMO Trial. Pediatrics 2006
117(5) 1640-1649.
26
Bootstrap mean cost differences
Cost category Mean cost difference (ECMO CM) Bootstrap mean cost difference (95 CI)
Death -412 -406 (-614, -209)
Transport 1851 1849 (1251, 2497)
Initial hospitalisation 16853 16,826 (11,775, 22,044)
Hospital readmissions 391 371 (-603, 1334)
Outpatient hospital care 474 481 (97, 875)
Community care 792 730 (-62, 1549)
Other costs 155 156 (-13, 443)
Total costs 20041 20057 (13690, 26318)
Source Petrou S, Bischof M, Bennett C, Elbourne
D, Field D, McNally H. Cost-effectiveness of
neonatal ECMO based on seven year results from
the UK Collaborative ECMO Trial. Pediatrics 2006
117(5) 1640-1649.
27
Measurement of outcomes at 7 years

• Survival period ? life years gained
• Neurodevelopmental assessments performed by
  developmental psychologist across 6 domains
• cognitive ability - behaviour
• neuromotor skills - hearing
• general health - vision
• Each domain defined as normal, impaired or mild,
  moderate or severely disabled
• Overall status defined by highest degree of
  impairment or disability in any domain
• ? disability-free life years gained
• Limitations of QALYs in childhood context

28
ECMO (n93) CM (n92)
Deaths
Before discharge 28 (30.1) 54 (58.7)
Between discharge and one year 2 (2.2) 0 (0.0)
Between one years and four years 1 (1.1) 0 (0.0)
Between four years and seven years 0 (0.0) 0 (0.0)
Total 31 (33.3) 54 (58.7)
Loss to follow-up
Between discharge and one year 1 (1.1) 1 (1.1)
Between one year and four years 2 (2.2) 2 (2.2)
Between four years and seven years 3 (3.2) 1 (1.1)
Total 6 (6.5) 4 (4.3)
Final assessment at seven years of age
Severe disability 3 (3.2) 0 (0.0)
Moderate disability 9 (9.7) 6 (6.5)
Mild disability 13 (14.0) 11 (12.0)
Impairment only 21 (22.6) 15 (16.3)
No abnormal signs or disability 10 (10.8) 2 (2.2)
Known survivors with no disability 31/56 (55.4) 17/34 (50.0)
29
Cost-effectiveness of neonatal ECMO

• Incremental cost-effectiveness ratio ?C / ?E
• 13,385 per life year gained
• or
• 23,566 per disability-free life year gained
• NB Natural or physical unit of outcome, which
  ignores full range of consequences.

30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
Mean net benefits of neonatal ECMO

• Mean net benefits Rc.?E - ?C

Rc Life year gained Life year gained Disability-free life year gained Disability-free life year gained
Mean net benefit 95 CI Mean net benefit 95 CI
0 -20,130 (-26,027, -13,522) -19,949 (-25,997, -13,375)
10,000 -5,299 (-14,475, 3,750) -11,387 (-20,394, -244)
20,000 9,532 (-6,691, 25,351) -2,825 (-18,377, 13,804)
30,000 24,362 (1,474, 47,314) 5,737 (-16,245, 29,393)
40,000 39,193 (8,926, 69,513) 14,299 (-14,863, 44,781)
50,000 54,024 (15,496, 91,567) 22,861 (-13,246, 60,319)
34
Are trial-based economic evaluations sufficient?

• Under the right circumstances, Yes
• Like clinical evidence, economic evidence can
  stand alone or be synthesised
• Requirements reasonable comparators, adopts
  final outcomes, collects data on a sufficiently
  broad set of services, adequately powered,
  sufficiently long follow-up, representative
  patient population
• Problems Use of inappropriate statistical tests,
  lack of power, failure to handle missing data,
  lack of intention-to-treat analysis, follow-up
  too short, lack of transferability

35
Was the ECMO trial-based economic evaluation
sufficient?

• Reasonable comparator ?
• Representative patient population ?
• Intention to treat analysis ?
• Adequately powered ?
• Appropriate perspective ?
• Sufficiently long follow-up x
• Adopts final outcomes ?
• Appropriate statistical tests ?
• Handled missing data ?

36
Longer-term cost-effectiveness

• Simple decision-analytic model
• Assumptions
• restricted to first 18 years of life
• survivors to 7 years survive to 18 years
• disability status at age 7 does not vary
• excess annual costs during years 4-7 continue
  during years 8-18
• ? ICER 7344 per life year gained
• 11802 per disability-free life year
  gained

37
Vehicles for economic evaluation

• Prospective collection of data alongside
  randomised controlled trials
• Least subject to bias, control over instruments,
  low incremental cost. May need to supplement.
• Prospective collection of data alongside
  non-randomised studies
• More subject to possible bias, control over
  instruments, low cost.
• Retrospective analysis of available data
• Low control over design and data, subject to
  bias.
• Modelling study
• Data from different sources, combined in
  decision-analytic models. Hard to validate.
  Useful and often unavoidable adjunct to trials.

38
Limitations of trials as a vehicle for economic
evaluation
Trial limitations Inappropriate or partial
comparisons More than one trial Partial
measurement Unrepresentative practice Intermedia
te outcomes Limited follow-up No trials
NICE Examples Temozolomide (recurrent malignant
glioma) Drugs for Alzheimers Riluzole
(resource use) Glycoproteins Beta interferon
(MS) Implantable cardioverter defibrillators
Liquid-based cytology
39
Modelling

• Hence modelling aims to address all these
  questions and can be used instead of or as a
  complement to trial evidence
• Structure the economic question
• Extrapolate beyond observed data
• Links intermediate and final endpoints
• Generalizes results to other settings/patient
  groups
• Synthesises evidence and can create head-to-head
  comparisons where RCTs dont exist
• Can indicate the need for further research

40
Models should

• Represent a simplification of the real world
• Encourage decision-makers to be explicit
• Reflect current clinical practice and use
  appropriate comparators
• Be based on the best quality data available
• Cover the appropriate time period
• Include sensitivity analysis to explore
  uncertainty of data inputs and model structure
• Be transparent and reproducible
• Have internal and external validity

41
The realities of research funding

• A large proportion of funding for economic
  evaluation is attached to trials
• Three options available to analysts
• Satisfy expectation of standard trial-based
  economic evaluation and risk misleading results
• Refuse to collaborate
• Pragmatic collaboration
• Seek opportunities to use modelling to inform
  trial design
• Ensure sufficient budget for analysis which
  includes synthesis and modelling

42
A pragmatic way forward?
43
Suggested checklist for assessing economic
evaluations
1) Was question well-defined? 2) Were
alternatives clearly described? 3) What evidence
on effectiveness was used? 4) Were resources
measured and valued fully appropriately?
5) Was discounting necessary and was it
performed? 6) Were incremental costs and outcomes
analysed? 7) Was an adequate sensitivity
analysis performed? 8) Are the results adequate
to inform purchasing? 9) Are the conclusions
justified? 10) Are the results applicable to the
local population? Source Drummond MF, OBrien,
B, Stoddart GL, Torrance GW. Methods for the
economic evaluation of health care programmes.
2nd edition. Oxford Oxford University Press,
1997. Drummond MF, Jefferson T, et al.
Guidelines for authors and peer reviewers of
economic submissions to the BMJ. BMJ 1996
313275-83.