Multiplicity issues in HTA funded RCTs

1
Multiplicity issues in HTA funded RCTs

• RM Pickering
• and
• CK Tracey
• Public Health Sciences and Medical Statistics
• University of Southampton

2
Health Technology Assessment (HTA)

• Funding stream from British National Health
  Service
• Funds pragmatic trials that wouldnt be funded by
  industry (eg - drug to non-drug comparison)
• HTA publishes reports on trials they fund
• Reports now form the Health Technology Assessment
  journal with 2006 impact factor 5.29

3
Review of HTA funded trials

• AIM - Assess the extent of multiplicity in
  non-industry trials
• Randomised trials published in HTA series,
  volumes 3-9 (1999-2005)
• Parallel group crossovers excluded
• If a report presented 2 trials the larger/main
  trial included

4
Number of trials included in review
5
Features of the trials (n41)
6
Primary outcomes/parameters

• A primary outcome might be stated vaguely
• eg 1 it may have component parts the SF36 has
  8 sub-scales
• eg 2 a continuous variable may be analysed as a
  mean but also as the percentage exhibiting
  caseness
• Parameters refer to the number of individual
  variables resulting in a separate P values
• eg 1 - parameters 8
• eg 2 - parameters 2

7
Time-points

• Where an outcome is measured at several follow-up
  assessments the number of tests can multiply up

Potential primary primary
parameters x time points parameters

• Only calculated if primary outcomes stated
• If multiplicity of time points explicitly
  addressed this would be incorporated
• Effectively the number of P values that could
  result from the stated primary outcomes

8
Number of primary outcomes /parameters
9
How many parameters were presented in Results for
the 28 trials with 1 primary outcome variable?
10
Multiple time points(n41)
11
Number of groups multiple comparisons (n41)
12
Number of P values presented in Results

• P values in Results relating to outcome
  comparisons across intervention groups
• Excludes P values comparing characteristics at
  baseline, sensitivity analyses or health
  economics
• Includes P values relating to subgroup analyses
• If analysis presented unadjusted and adjusted for
  baseline factors counts as 2

Number of P values/CIs presented

• Same as above but includes contrasts presented
  with CIs
• If a P value and CI for same contrast counts as 1

13
Distribution of number of P values (n41)mean
(min to max) 66.4 (1 to 194)
14
Distribution of number of P values/CIs
(n41)mean (min to max) 102.7 (3 to 379)
15
Funding applications (n38)

• Protocols not routinely submitted to HTA
• Didnt approach researchers for protocols
• Original funding application available for most
  trials

16
Changes in primary variables in report compared
to the funding application (n21)
17
How multiplicity was dealt with depending on
whether or not statistician named on funding
application
18
Number of significance tests/CIs reported
according to whether a statistician named on
funding application (n35)
19
Conclusions

• ICH guidance on multiplicity not closely adhered
  to
• Trials in review predate registration of all
  trials circa 2005
• Discrepancy in primary outcomes between protocols
  and published reports by Chan et al (JAMA 2004)
  in a review of 102 trials approved by Ethics
  Committee in Copenhagen Fredericksberg
• HTA reports interesting because authors arent
  limited in space
• Possible to aim for complete coverage
• Journal papers presentation of primary
  outcomes selection of most interesting findings
  from rest?

20
Thanks

• Prof James Raftery permission to access funding
  applications
• Alison Price arranging for copies of reports
• Sally Bailey help in accessing funding
  applications