SEQUENTIAL DESIGNS FOR SELECTING BETTERTHANSYMPTOMATIC TREATMENTS Bruce Levin Department of Biostati

1
SEQUENTIAL DESIGNS FOR SELECTING
BETTER-THAN-SYMPTOMATIC TREATMENTSBruce
LevinDepartment of BiostatisticsMailman School
of Public HealthColumbia UniversityPresented
at the2008 AAPS WorkshopApril 29,
2008Co-sponsored with FDA, MJFF, PSG
2
This talk represents joint work with my
colleaguesCheng-Shiun LeuandYing-Kuen Kenneth
Cheungof theDepartment of BiostatisticsColumbia
University
3
Overview
Part 1 A primer on sequential procedures for
selecting the best treatments. Two specific
non-adaptive examples. Part 2 An adaptive
procedure for sequential selection of the best
treatments with elimination of inferior
treatments. Part 3 An even better adaptive
procedure for selecting the best treatments, with
elimination of inferior treatments and
recruitment of superior treatments. Part 4
Selecting subsets of treatments that are better
than symptomatic.
4
Today I will focus on binary outcomes using
vector-at-a-time sampling. This is primarily
for simplicity and clarityother outcome
distributions can be usedbut the binary case
offers some interesting possibilities. For
example
5

• Occurrence of the need for rescue therapy for
  dopaminergic or non-dopaminergic symptoms within
  six months
• Sliding dichotomy allows tailoring to patients
  disease severity and prognostic factors to define
  individualized, a priori definitions of success
  (better than expected).
• Calibration to 50 success produces maximum
  (dichotomous) information per patient.
• Easier to impute for missing datause Jogas
  linearity!

6

• Can use multiple clinical outcomes or
  biomarkers to define success (e.g., at least
  three positive signs).
• Most intriguingly, in principle, can use a
  patients entire clinical history and trajectory
  of followup measures to define success to mean
  what appears to be a better-than-symptomatic
  (BTS) outcome for this patient.
• So Im going to talk about tossing a set of c
  coins and youll understand I mean c treatments.

7

• Vector-at-a-time sampling means tossing the
  entire set of coins once each, twice each, etc.
  This allows blocking for
• sites in a multi-center study
• possibly other stratification factors such
  as disease severity, time since diagnosis,
  on or off levodopa, etc.

8
PART 1TWO NON-ADAPTIVE SEQUENTIAL PROCEDURES
FOR SELECTING THE BEST OF SEVERAL TREATMENTS.
9
Given c coins with success probabilities p1, p2
,, pc .
We wish to identify the best coin, or, more
generally, to identify the best b out of c coins
(b 1,,c 1).
10
We are NOT hypothesis testing! We are discussing
early phase research and until Part 4 of this
talk, we will be quite uninterested in the null
hypothesis that p1 pc. We wont be making
any type I errors because we wont be declaring
one treatment significantly better than
another. We assume instead (for now) that we
want to select a subset of b coins whether or
not they have equal ps.
11
In the subset selection paradigm, we follow an
indifference zone approach. This means that
for values of p1, , pc sufficiently close we
are indifferentto which coins are selected.
12
However, if the odds ratio relating pb and pb1
is greater than or equal to a pre-specified
value, sayORb (pb/qb)/(pb1/qb1) ? 1,
then we will require the probability of correct
selection to be at least a pre-specified
minimum, say P...
13
...that is, we require Pcs P whenever the
success probabilities fall into the preference
zone defined by ORb ?. In this sense we can
say that we have confidence level P in making a
correct selection when the coins lie in the
preference zone. Well talk in a moment about
what can be said when the coins lie in the
indifference zone.
14

• Selection procedures are appealing for
• Final dose selection in phase IIb trials in
  preparation for phase III studies
• Optimization of combination therapies or
  intervention modalities
• Product selection for RD
• Selection of genetic markers for weeding out
  false positives

15
Lets start by considering asequential selection
procedureto identify the best coin (b 1). It
generalizes Walds sequential probability ratio
test (SPRT) for two coins.
16

• Sequential selection procedure
• Toss the coins vector-at-a-time.
• Stop the first time one coin has r more heads
  than any other coin, where r 1 is a
  pre-specified integer.
• Select the leading coin as best.

17
Theorem 1 (Levin and Robbins, 1981). For any set
C of c coins with probabilitiesp1,,pc, let
the notation be such that
p1 p2 pc ,
and let wi pi /(1 pi) be the odds on
success for coin i. Then for any i lt j,
18
Without loss of generality, we may arrange the
labels of the coins such that p1 p2 pc .

• We dont know the ordering in practice.
• The ordering is merely for notational
  convenience.
• We consider only symmetric procedures, that is,
  procedures that are invariant under permutations
  of the subscripts of the coins.

19
Theorem 2 (Levin and Robbins, 1981). Let A1
Ac be any set of constants.For the sequential
selection procedure without elimination, N, we
have
20
Theorem 2 (Levin and Robbins, 1981).
If Ai denotes a reward for choosing coin i, then
the theorem gives an explicit lower bound formula
for the expected reward.
21
For example, choosing A1 Aa 1 and Aa1
Ac 0 yields a lower bound for the
probability that the one coin selected as best is
among the best a coins
specializes to
22
In particular, choosing A1 1 andA2 Ac
0 yields an explicitlower bound formulafor the
probability of correct selection
L is a function only of r and the odds ratios
w1 /w2 ,, w1 /wc .
23
Example c4, p10.2, p2p3p40.1, P0.95. What
r shall we use? w1 1/4, w2 w3 w4 1/9. We
want L 2.25r/(2.25r 3) P 0.95, so
choose r to be the smallest integer greater than
or equal toso choose r 5. Then Pcs
0.9505 for any set of pis with odds ratio 2.25
or more. (Easy, right?)
24
Theorem 1. For any set of coins C with
probabilitiesp1,, pc, let the notation be such
that
p1 p2 pc ,
and let wi pi / (1 pi). Then for any i lt j,
IMPLIES
Theorem 2. For any constants A1 Ac , we
have
25
Theorem 2 follows directly from thisProposition
Let andbe any two sets of
positive numbers satisfying
Then for any constants A1 Ac,
26
Bonus a very simple proof of BKS! The
Bechhofer-Kiefer-Sobel (BKS) procedureSequential
Identification and Ranking Procedures
(1968) Let ?gt1 be a pre-specified constant such
that we require Pcs ? Pgt1/c for any set of
p1??pc with (p1 /q1)/(p2 /q2) ? ?.
27
Bonus a very simple proof of BKS! DefineNBKS
infn?1
infn?1 . Theorem
(BKS) The selection procedure with stopping
rule NBKS has Pcs ? P for any set of coins
with p1 ? ? pc satisfying (p1 /q1)/(p2
/q2) ? ?gt1.
28
The BKS procedure for subset selection (bgt1) is
quite cumbersome for large b and c, and not
feasible for very large b and c, requiring
enumeration of a huge number of subsets to
determine the stopping criterion. Also, it is
not clear how to modify the procedure to make it
adaptive, e.g., to eliminate inferior coins as
the evidence accumulates.
29

• The non-adaptive LRL sequential procedure for
  selecting the subset of b 1 best coins
• Toss the coins vector-at-a-time.
• Stop the first time the coin or coins with the
  bth largest tally has r more heads than the
  coin or coins with the (b1)st largest tally,
  where r 1 is a pre-specified integer.
• Select the b leading coins as best.

30
Theorem The probability of selecting the
correct subset of b best coins is bounded from
below by For design purposes, Pcs can be
made P by choosing r sufficiently large
given the least favorable configuration in the
preference zone, p1 pb gt pb1
pc where the odds ratio separating coins b and b
1 is at least ?.
31
What if the coins fall into the indifference
zone? Can anything be said of the properties of
thesubsets selected?
32
?-acceptable subset selectionFor any set of
odds, we define a neighborhood around thebth
best and (b1)st best odds as follows
The integers s and t satisfy 0 s b t c
and depend on the success odds and the design
odds ratio ?, so that ss(?,w) and tt(?,w).
33
?-acceptable subset selectionWe call a
selection of a subset of b coins ?-acceptableif
the best s coins are necessarily selected,the
worst c-t coins are necessarily not selected, and
theremaining b-s coins are selected from the
neighborhoodof coins containing the (s1)st to
the t th best coins.
34
?-acceptable subset selection
The BKS conjecture states that for any set of
odds,the BKS procedure selects a ?-acceptable
subsetwith probability at least P. While
the proof of the BKS conjecture for bgt1 is still
an open problem for the BKS procedure itself, we
have proven that the non-adaptive
Levin-Robbins-Leu procedure does satisfy the BKS
conjecture.
35
PART 2A PROCEDURE FOR SEQUENTIAL SELECTION OF
THE BEST TREATMENTS WITH ELIMINATION OF INFERIOR
TREATMENTS.
36
In a clinical trials setting, we would much
prefer to eliminate inferior treatments before
the bitter end of the trial, in order to avoid
randomizing patients to treatments with
increasingly strong evidence of inferiority.
Data monitoring committees might do this anyway,
but possibly in an intuitive and unreliable
manner.
37

• Adaptive LRL sequential selection procedurewith
  elimination of inferior coins
• Toss the coins vector-at-a-time.
• The first time one or more coins fall r
  successes behind the coin or coins with the
  bth largest tally, eliminate those coins as
  inferior.
• Continue tossing the remaining coins at
  their current tallies.
• Iterate until only b coins remain.
• Select those coins as best.

38
Leu and Levin (1999a) prove that for the
sequential elimination procedure in the case b1,
the lower bound formula for Pcs holds, and
Leu and Levin (1999b) prove that Theorem 2 is
true, for any integer r 1 and for any number
of coins c 2.
39
Example c4, p10.2, p2p3p40.1,
P0.95. r5, Pcs 0.9505 Cor. sel.
Rounds Tosses Failures Pcs
EN ET EF Without
elimination 0.972 65.6 262.5
229.7 With elimination 0.954
65.1 205.3 178.5 Fixed sample
size (n132) 0.973 132.0 528.0
462.0 Fixed sample size (n105) 0.950
105.0 420.0 367.5
40
Cor. sel. Rounds Tosses Failures
Pcs EN ET EF
. Without elimination
0.972 65.6 262.5 229.7
-------------------------------------------------
----------------------- Without elimination,
truncation at n125 0.955 62.6
250.5 219.3--------------------------------
---------------------------------------- With
elimination 0.954 65.1 205.3
178.5 ---------------------------------------
---------------------------------
41
PART 3SELECTION OF BEST TREATMENTS WITH
ELIMINATION OF INFERIOR TREATMENTSANDRECRUITME
NT OF SUPERIOR TREATMENTS.
42

• Toss the coins vector-at-a-time.
• The first time one or more coins fall r
  successes behind the coin or coins with the
  bth largest tally, eliminate those trailing
  coins as inferior.
• The first time one or more coins pull r
  successes ahead of the coin or coins with the
  (b1)st largest tally, recruit those leading
  coins as superior.
• Continue tossing the other coins at their
  current tallies.
• Iterate until b coins have been recruited
  and/or cb coins have been eliminated.
• Select the recruited coins as the b best.

43
Recruitment allows superior treatments to be
brought forward to the next stage of development
before the end of the experiment. The adaptive
LRL procedure with both elimination and
recruitment combines the best features of both
adaptations compared to either one alone for
arbitrary choices of b. Note there is no claim
that the first treatment to be recruited is best,
only that there is sufficient evidence to
consider it among the b best treatments. Similarly
, there is no claim that the first treatment to
be eliminated is the worst, only that there is
sufficient evidence to consider it among the c-b
worst treatments.
44

• Special cases
• When b 1, the adaptive LRL procedure with
  elimination and recruitment reduces to the
  elimination-only procedure.
• When b c1, the adaptive LRL procedure with
  elimination and recruitment reduces to the
  recruitment-only procedure.
• The recruitment only procedure for b 1
  reduces to the non-adaptive LRL procedure.

45

• Conjecture The probability of selecting the
  correct subset of b best coins continues to be
  bounded below by
• Proved by change of measure argument for
  the non-adaptive procedure.
• Proved for c 4 coins for adaptive LRL, any
  b.
• Extensive simulation results for c gt 4.

46

• Conjecture The probability of selecting the
  correct subset of b best coins continues to be
  bounded below by
• As before, for design purposes, the procedure
  can be designed to satisfy Pcs P by
  choosing r sufficiently large.

47

• Conjecture Lower bounds are also available for
  other selection events of interest, such as
• (s,t)-events
• ?-acceptable events
• Number of coins from among the a best, denoted
  Ya

48
Illustration b2, c4, r4, ?2, P0.8, p(0.4,
0.375, 0.25, 0.22) Nb The actual odds ratio
separating coins 2 and 3 1.8, which is inside
the indifference zone. But notice how the lower
bound formula still ensures Pcs L 0.8054 gt
0.8 P. Method Pcs EN ET
EF. Non-adaptive LRL 0.889 40.0
159.9 110.1Non-adaptive BKS 0.842 33.1
132.4 91.2Adaptive LRL (w/ ER) 0.836
37.6 114.2 78.8
49
PART 4SELECTION OF BETTER-THAN-SYMPTOMATIC
TREATMENTS.
50

• Up to now we have been unconcerned with the null
  hypothesis. It seems unavoidable that we would
  want to make declarations such as
• There exists a Better-Than-Placebo (BTP)
  b-tuple.
• There exists a Better-Than-Control (BTC)
  b-tuple.
• There exists a Better-Than-Standard (BTS)
  b-tuple.
• There exists a Better-Than-Symptomatic (BTS)
  b-tuple.

51
So suppose we have c0 control treatments and c1
active treatments, where the control treatments
can be viewed as placebos, active controls, or,
in particular, just symptomatic treatments. We
want to select a b-tuple with 1 b c1 and then
make one of two decisions
52
Abbreviations cd correct declaration BT
S Better than symptomatic (b) cs
correct selection, i.e., best (b) We shall
require that for a pre-specified maximum level of
control efficacy, say p0,
53
If then possibly Pwcd lt P or 1-?, although
not if and How do we achieve these
requirements? Data augmentation for control
outcomes. Idea If p1 pc p, augment the
control probabilities so that after
augmentation Pcd Pselect at least one
control 1?.
54
If w1 wc w before augmentation, then
suppose the success odds of the controls are
augmented to ?w. Then the type 1 error
requirement together with the lower bound formula
determines ?r
55
There are two ways to augment the
data stochastically or deterministically. Stochas
tically Let Y Bin(1,p) and, independently,
let Z Bin(1,p). The augmented outcome for a
given control subject is Y' Y Z(1Y)
Bin(1,ppq). We define p such that for p p0,
The required p is
56
There are two ways to augment the
data stochastically or deterministically. Determi
nistically The augmented outcome for a given
control subject is Y" Y p(1Y). Although Y"
is no longer a binary variable, the procedure
actually has somewhat higher probabilities of
correct declaration and somewhat greater sample
sizes.
57
In the following examples, we illustrate the null
hypothesis case, the design alternative case, and
several other scenarios in which there is no BTS
pair of active treatments. We confirm that with
stochastic augmentation, the type 1 error is
controlled under the strong null hypothesis of
equal treatments, and that Pcd P under the
design alternative. Note, though, that if
then possibly Pwcd lt P or 1-?, although
not if and
58
Illustration (Active in gold, Symptomatic in
green) b2, c4, ?0.10, p00.5 ?3,
P0.9 r4 Success probabilities Pcd EN
ET EF. (0.5, 0.5, 0.5, 0.5) 0.913
49.1 144.5 72.1(0.75, 0.75, 0.5,
0.5) 0.936 27.2 85.1 31.7 For a
classical hypothesis test of H0 equal probs at
level ?0.10 with fixed sample size of n patients
per group, with 90 power to reject H0 at (0.75,
0.75, 0.50, 0.50), would require T4n4x45180
patients withEF90 under H0 and EF 67.5
under the design H1.
59
Illustration (Active in gold, Symptomatic in
green) b2, c4, ?0.10, p00.5 ?3,
P0.9 r4 Success probabilities Pcd EN
ET EF. (0.5, 0.5, 0.5, 0.5) 0.913
49.1 144.5 72.1(0.75, 0.75, 0.5,
0.5) 0.936 27.2 85.1 31.7(0.75,
0.75, 0.5, 0.5) 0.996 24.7 78.1
29.1 (0.75, 0.5, 0.5, 0.5) 0.774 40.7
115.4 53.5(0.75, 0.67, 0.67, 0.5) 0.634
44.2 127.3 43.5 (0.75, 0.75, 0.75,
0.5) 0.741 52.3 145.0 40.5 (0.75,
0.75, 0.75, 0.75) 0.892 66.1 194.9
48.7
60
Illustration (Active in gold, Symptomatic in
green) b2, c4, ?0.10, p00.5 ?3,
P0.9 r4 Stochastic versus Deterministic Success
probabilities Pcd EN ET
EF. (0.5, 0.5, 0.5, 0.5) 0.913 49.1
144.5 72.1(0.5, 0.5, 0.5, 0.5) 0.920
63.5 185.6 92.7(0.75, 0.75, 0.5, 0.5)
0.936 27.2 85.1 31.7(0.75,
0.75, 0.5, 0.5) 0.959 30.3 96.3
36.1(0.75, 0.75, 0.75, 0.75) 0.892 66.1
194.9 48.7 (0.75, 0.75, 0.75, 0.75) 0.901
84.4 246.4 61.6
61
Illustration (Active in gold, Symptomatic in
green) b2, c4, ?0.10, p00.5 ?3,
P0.9 r4 Stochastic vs. Deterministic with
curtailment when not BTS Success
probabilities Pcd EN ET
EF. (0.5, 0.5, 0.5, 0.5) 0.913 26.6
92.7 46.3(0.5, 0.5, 0.5, 0.5) 0.920
33.2 115.7 57.8(0.75, 0.75, 0.5, 0.5)
0.936 26.1 82.8 30.9(0.75,
0.75, 0.5, 0.5) 0.959 29.2 94.0
35.2(0.75, 0.75, 0.75, 0.75) 0.892 37.4
128.5 32.1 (0.75, 0.75, 0.75, 0.75) 0.901
45.4 156.0 38.9
62
Levin, B. and Robbins, H. (1981). Selecting the
highest probability in binomial or multinomial
trials.Proc. Natl. Acad. Sci. USA, 78,
4664-4666. Levin, B. (1984). On a sequential
selection procedure of Bechhofer, Kiefer, and
Sobel.Statistics and Probability Letters, 2,
91-94. Zybert, P. and Levin, B. (1987). Selecting
the highest of three binomial probabilities.Proc.
Natl. Acad. Sci. USA, 84, 8180-8184.
63
Leu, C.-S. and Levin, B. (1999). On the
probability of correct selection in the
Levin-Robbins sequential elimination procedure.
Statistica Sinica, 9, 879-891. Leu, C.-S. and
Levin, B. (1999). Proof of a lower bound formula
for the expected reward in the Levin-Robbins
sequential elimination procedure.Sequential
Analysis, 18, 81-105.
64
Levin, B. and Leu, C.-S. (2007). A comparison of
two procedures to select the best binomial
population with sequential elimination of
inferior populations. Journal of Statistical
Planning and Inference, 137, 245-263. Leu, C.-S.
and Levin, B. (2008). A Generalization of the
Levin-Robbins Procedure for Binomial Subset
Selection and Recruitment Problems.Statistica
Sinica, 18, 203-218. Leu, C.-S. and Levin, B.
(2008). On a Conjecture of Bechhofer, Kiefer,
and Sobel for the Levin-Robbins-Leu Binomial
Subset Selection Procedures. Sequential
Analysis, 27, 106-125.
65
The End.Thank you.