Measuring Effectiveness in Trials of Acute Bacterial Sinusitis

1
Measuring Effectiveness in Trials of Acute
Bacterial Sinusitis

• John H. Powers, MD
• Lead Medical Officer
• Antimicrobial Drug Development and Resistance
  Initiatives
• Office of Medical Policy
• Center for Drug Evaluation and Research
• U.S. Food and Drug Administration

2
Introduction

• Regulatory and scientific history related to
  noninferiority trials and demonstration of
  effectiveness
• History of discussions regarding NI trials in
  antimicrobials and acute bacterial sinusitis
  trials
• Evaluation of historical evidence of magnitude of
  effects of antimicrobials in placebo controlled
  trials in acute bacterial sinusitis
• Conclusions regarding current knowledge of
  evaluation of effectiveness and in vitro
  resistance on outcomes

3
Regulatory/Scientific History

• 1938 pre-market requirement for safety only
  (based on Elixir of sulfanilamide deaths)
• 1962 requirement for demonstration of
  effectiveness to balance any potential harms of
  therapy (thalidomide)
• President Kennedy sent recommendations to Senate
  Committee an undefined standard of substantial
  evidence of effectiveness was inadequate in
  terms of assuring that drugs that reach the
  market have been shown to be effective for the
  claims made for them.
• Congress defines that only source of substantial
  evidence of effectiveness is adequate and
  well-controlled trials
• 1970 FDA publishes regulations that provide
  criteria for defining adequate and
  well-controlled trials

4
Adequate and Well-Controlled

• Clear statement of objectives
• Study design permits valid quantitative
  comparison with a control
• Select patients with disease (treatment) or at
  risk of disease (prevention)
• Baseline comparability (randomization)
• Minimize bias (blinding, etc.)
• Appropriate methods of assessment of outcomes
• Appropriate methods of analysis
• 21 CFR 314.126

5
Regulatory/Scientific History

• 1985 recognition of issues with trials designed
  to show similarity rather than superiority of
  drugs
• If the intent of the trial is to show similarity
  of the test and control drugs, the report of the
  study should assess the ability of the study to
  have detected a difference between treatments.
  Similarity of test drug and active control can
  mean either that both drugs were effective or
  that neither was effective. The analysis of the
  study should explain why the drugs should be
  considered effective in the study, for example,
  by reference to results in previous
  placebo-controlled studies of the active control
  drug.
• 21 CFR 314.126 (b)(2)(iv)

6
Regulatory/Scientific History

• Noninferiority trials attempt to
• rule out how much inferior a new treatment might
  be compared to an already proven effective
  treatment
• AND ensure control drug effect relative to
  placebo is consistent under the conditions of the
  trial
• 2000 ICH-E10 Guidance on Choice of Control
  Group and Related Issues in Clinical Trials
• Section 1.5 describes information necessary to
  select noninferiority margin determined by
  analysis of historical evidence of sensitivity to
  drug effects (HESDE)
• magnitude by which control drug may be reliably
  and reproducibly shown superior to placebo in
  previous superiority trials
• Effect of control relative to placebo should be
  well-characterized and consistent from trial to
  trials so that effect in current trial consistent

7
Regulatory/Scientific History

• Well-designed experiments have positive and
  negative controls to determine results causally
  related to intervention and not conditions of
  experiment (internal validity)
• Lack of negative controls in noninferiority
  trials means they lack intrinsic measure of
  internal validity
• Data on effect of control relative to placebo
  external to NI trial
• Similar potential biases to historical controlled
  trials
• Conditions of experiment (e.g. enrolling patients
  who do not have disease, timing of outcomes
  beyond natural history of disease, etc.) may
  change effect of control and affect conclusions
  regardless of margin chosen
• Demonstration of noninferiority does not
  necessarily mean drug has demonstrated
  effectiveness relative to placebo

8
Regulatory/Scientific History

• Selecting of appropriate margin of inferiority
  integral to design of noninferiority trials
• In practice, the noninferiority margin chosen
  usually will be smaller than that suggested by
  the smallest effect size of the active control
  because of interest in ensuring that some
  clinically acceptable effect size (or fraction of
  the control drug effect) was maintained.
• ICH E-10, section 1.5.1.1., page 11
• Effect size refers to magnitude of benefit of
  active control drug relative to placebo from
  previous placebo controlled trials

9
Quantitative Comparison with Control

• Three criteria before one can perform
  noninferiority trial
• Quantitative assessment of effect of control drug
  relative to placebo based on data from previous
  trials
• Reliable, well-characterized and reproducible
  effect from trial to trial
• Based on trials that are adequate and well
  controlled
• Evaluation of all previous superiority trials
• Takes into account variability of the data (
  confidence intervals rather than point estimates)
• THEN
• 2. Selection of margin that is less than effect
  of control relative to placebo to preserve some
  benefit of control based on clinical judgment
• AND
• 3. Maintenance constancy of the effect of the
  control from trial to trial
• Similar definition of disease, endpoints, timing
  of endpoints
• Changes in medical practice, adjunctive
  therapies, antimicrobial resistance

10
Definitions and Determining Margin

• M(1) defined as magnitude of benefit of active
  control compared to placebo as measured in
  current trial but determined from data in
  previous superiority trials
• M(2) defined loss of effect of test drug compared
  to active control where drug is still considered
  clinically useful based on clinical judgment
• Results of trial demonstrate noninferiority if
  success rates with test drug minus success rates
  with active control exclude M(2) T C lt M(2)
• BUT this only shows that test drug is effective
  relative to placebo when M(2) is less then M(1)
  M(2) lt M(1)

11
Non-Inferiority MarginsQuantifying the Effects
of a Control Drug
Difference in Point Estimates
12
Non-Inferiority MarginsQuantifying the Effects
of a Control Drug
Difference in Point Estimates
13
Non-Inferiority MarginsDetermining the Effect of
a Test Drug
Difference in Point Estimates
14
Non-Inferiority MarginsDetermining the Effect of
a Test Drug
Difference in Point Estimates
15
Non-Inferiority MarginsDetermining the Effect of
a Test Drug
control
100
control
test
80
success rates
0
past trials
current trial
16
History with Antimicrobials/ABS

• Several meetings addressing issues in
  noninferiority trials in study of antimicrobials
• Need to evaluate data on each indication to
  determine margins or whether data to support
  noninferiority trials
• Issues with selection of patients with disease
• Issues with defining outcomes and timing of
  outcomes
• Issues with analysis (ITT and subgroup analyses)
• October 2003 AIDAC meeting on clinical trial
  design in ABS
• No constellation of signs/symptoms predicts
  bacterial etiology so sinus puncture necessary at
  to define patients with disease
• No studies correlating gtten days of symptoms with
  sinus punctures
• Lack of evidence of specificity of radiographic
  findings with positive culture on sinus puncture
• No evidence to support presumed eradication of
  organisms
• Timing of outcome important in relation to
  ability of trial to evaluate effectiveness and
  time to resolution of symptoms may be most
  sensitive measure of outcome
• Lack of evidence from previous placebo controlled
  trials to base any noninferiority margin trials
  should be superiority trials

17
Evaluating Data from Literature

• Criteria for submission of published literature
  reports alone as evidence of effectiveness
• Multiple studies conducted by different
  investigators where each clearly has adequate
  design and findings consistent across studies
• High level of detail in published reports
  including clear and adequate descriptions of
  statistical analysis plans, prospectively
  determined analytical methods and study
  endpoints, and full accounting of all enrolled
  patients
• Clearly appropriate endpoints objectively
  assessed and not dependent on investigator
  judgment
• Robust results achieved by protocol-specified
  analyses that yield consistent conclusions of
  efficacy and do not require selected post-hoc
  analyses, subsetting or reduced datasets (e.g.
  analysis of only responders)
• Conduct of studies by groups with properly
  documented operating procedures

Guidance for Industry Providing Clinical
Evidence of Effectiveness for Human Drugs and
Biological Products p. 19.
18
What One Would Need to Justify a Margin of 10
in Acute Bacterial Sinusitis

• Similar disease definition
• Similar endpoint definition
• Similar timing of endpoint
• Constancy of effect of control

0
10
20
30
40
50
50
40
30
20
10
Favors placebo
Favors drug
19
ABS Placebo Controlled Trials

• Description of trials
• 21 trials compared antimicrobial to placebo in
  all languages but 4 not prospective, not
  randomized and/or no direct patient outcomes
• 17 randomized, prospective trials evaluated
• 2787 total patients (1132 evaluable on placebo
  and 1331 evaluable) on various
  drugs/doses/durations
• No quinolones
• One trial in children with cefuroxime in children
  (no evidence of benefit)
• Dates from 1964 to 2005 (41 year span)
• 8 trials published since 2000, 2 in 2005,
  showing that placebo controlled trials can and
  are being performed
• 3 published since last AIDAC meeting in 2003
  including one in US, one in Europe and one in
  pediatric population
• Demographics
• Average age per trial was 37 years with age range
  18-93 years
• Average gender distribution per trial was 60
  female in trials with adults or mixed populations

20
Analysis of Efficacy in Placebo Controlled Trials
in Acute Bacterial Sinusitis
cefuroxime d14
Kristo et al. 2005 n82
pivampicillin d8
Norrelund et al. 1978 1978 n135
doxycycline d10
Stalman et al. 1997 n186
amox or amoxicillin-clav d14
Garbutt et al. 2001 n161
Lindbaek et al. 1998 n70
amoxicillin or penicillin d10
amoxicillin-clavulanate d14
Bucher et al. 2003 n251
amoxicillin d14
Merenstein et al. 2005 n135
amoxicillin d14
van Buchem et al. 1997 n206
amoxicillin d10
deSutter et al. 2003 n135
pencillin or lincomycin d10
Axelsson et al. 1970 n142
amox or doxy or penicillin d 14
Varonen et al. 2003 n146
amoxicillin or amox-clav d10
Wald et al. 1986 n93
azithromycin d8
b
Kaiser et al. 2001 n265a (77)b
a
penicillin d7
Hansen et al. 2000 n127
azithromycin d14
Haye et al. 1998 n168
amoxicillin or penicillin d10
Lindbaek et al. 1996 n127
cyclacillin (not specified)
Ganaca et al. 1973 n50
0
10
20
30
40
50
50
40
30
20
10
Favors placebo
Favors study drug
21
ABS Placebo Controlled Trials

• No reliable, consistent magnitude of benefit of
  antimicrobials compared to placebo M(1), so no
  evidence upon which to base any noninferiority
  margin
• Majority of trials do not provide evidence of
  benefit of antimicrobials compared to placebo
  most trials powered to rule out differences of
  15 to 35 which is amount needed to justify 10
  to 15 margin
• Trial with largest point estimate of benefit had
  lower bound of 95 confidence interval 11.3
  most of point estimates of treatment difference
  lt10
• Point estimates success rates in placebo groups
  range from 29 to 95 and point estimate success
  with various drugs range from 35 to 93
• No evidence of decreasing complications or
  prevention of chronic sinusitis
• Cannot select loss of effect of test drug
  relative to control M(2) since magnitude of
  benefit of control relative to placebo M(1) not
  clear
• No constancy of effect of control
• Different definitions used for enrollment
  criteria
• None of trials used receipt of additional
  antimicrobial as an endpoint
• Timing of fixed endpoints vary and not longer
  than 14 days for primary fixed endpoint effects
  of drug compared to placebo disappear with time

22
Analysis of Safety in Placebo Controlled Trials
in Acute Bacterial Sinusitis
Axelson et al. 1970 n142
van Buchem et al. 1997 n206
Stalman et al. 1997 n186
Hansen et al. 2000 n127
Kaiser et al. 2001 n265 (77)
deSutter et al. 2003 n135
Lindbaek et al. 1996 n127
odds ratio 3.89 (2.09, 7.25)
Garbutt et al. 2001 n161
p-value 0.017
Bucher et al. 2003 n251
Lindbaek et al. 1998 n70
GI AEs 3 drug, 0 placebo
6 excluded from analysis on drug, 2 on placebo
Wald et al. 1986 n93
Norrelund et al. 1978 1978 n135
Varonen et al. 2003 n146
Kristo et al. 2005 n82
Ganaca et al. 1973 n50
Haye et al. 1998 n168
Merenstein et al. 2005 n135
0
10
20
30
40
50
50
40
30
20
10
Favors placebo
Favors study drug
23
Analysis of Efficacy in Placebo Controlled Trials
in Acute Bacterial Sinusitis
gemi 009
Kristo et al. 2005 n82
Norrelund et al. 1978 1978 n135
Stalman et al. 1997 n186
Garbutt et al. 2001 n161
Lindbaek et al. 1998 n70
Bucher et al. 2003 n251
Merenstein et al. 2005 n135
van Buchem et al. 1997 n206
deSutter et al. 2003 n135
Axelsson et al. 1970 n142
Varonen et al. 2003 n146
Wald et al. 1986 n93
b
a
Kaiser et al. 2001 n265a (77)b
Hansen et al. 2000 n127
Haye et al. 1998 n168
Lindbaek et al. 1996 n127
Ganaca et al. 1973 n50
0
10
20
30
40
50
50
40
30
20
10
Favors placebo
Favors study drug
24
In Vitro Resistance and Clinical Outcomes

• Given uncertainty about magnitude of treatment
  effect with any drug, correlation between in
  vitro resistance and clinical outcomes also
  unclear
• Even with susceptible organisms, lack of
  correlation between microbiological outcomes and
  clinical outcomes in patients
• Bacterial survival in the maxillary sinus
  despite a high concentration in the sinus
  illustrates that MIC values determined in the
  laboratory do not always mirror the sensitivity
  of the bacteria to antibiotics in vivo.
• Carenfeldt C et al. Scand J Infect Dis
  19757259-64.
• Patient often recover clinically despite
  persistence of organism and differences in
  potency of antimicrobials does not necessarily
  translate into differences in clinical outcomes
• Carenfeldt C et al. Acta Otolaryngol
  199010128-135.

25
Conclusions

• Need for demonstration of effect of control drug
  relative to placebo in noninferiority trial has
  been noted in regulations since 1980s
• Evaluation of previous placebo controlled trials
  in ABS does not show reliable and reproducible
  magnitude of effects of antimicrobials relative
  to placebo for studying new drugs in clinical
  trials
• Demonstration of noninferiority in acute
  bacterial sinusitis trials still leaves
  uncertainty as to whether this demonstrates
  effectiveness of drugs relative to placebo
• Demonstration of effectiveness needed to balance
  any potential harms of therapy

26
Back Up Slides
27
Substantial Evidence

• Upjohn contends that the totality of materials,
  which include these 54 articles, the material
  submitted over the years since these products
  were certified, and the clinical experience in
  totality clearly satisfy the substantial
  evidence. It says the clinical experience,
  widespread throughout the world, used by
  thousands upon thousands of doctors in 750
  million doses is a very significant factor.
• Upjohn v. Finch, 1970 Appendix A, p. 12

28
Substantial Evidence

• The Commissioner concludes that Congress itself
  has described the type of evidence that is
  suitable to support claims of effectiveness. The
  claims must be supported by adequate and well
  controlled investigations. This means that the
  experimental factors must be so controlled that
  the effectiveness of an anti-infective drug on
  the disease process in patients can be compared
  with the effect of no treatment or of a
  recognized effective treatment of patients with
  the same disease or condition.
• Upjohn v. Finch, 1970 Appendix A, p. 12

29
Substantial Evidence

• Upjohn recognizes this fact, but contends that
  several in vitro studies reported, the mouse
  study, and the uncontrolled observations reported
  in the literature, when added to the evidence
  obtained from the studies in which controls were
  attempted, raise the quality of the data to the
  level required by the law.
• Upjohn v. Finch, 1970 Appendix A, p. 12

30
Substantial Evidence

• The in vitro studies are suggestive of some
  effectivenessin laboratory experiments utilizing
  artificially cultured microorganisms as test
  systems, but because the studies are not at all
  correlated clinical trial experience they
  cannot be used as a basis for concluding the
  drugs will have the effectiveness claimed for
  them when used to treat naturally occurring
  clinical disease in man.
• Upjohn v. Finch, 1970 Appendix A, p. 12