Statistical considerations in small proof-of-concept trials, including crossover designs

1
Statistical considerations in small
proof-of-concept trials, including crossover
designs

• Stephen Senn

2

• People look down on marketing men
• Its not true that they are not scientists
• They work in sell biology
• I would like to take this opportunity to draw
  your attention to a book I rather like

• In fact I wrote it myself

3
Outline

• Decision analysis and proof of concept
• Value of information perspective
• Place of cross-over trials
• Carry-over
• The potential for cross-over trials in studying
  individual response

4
A Model
Probability proof of concept (POC) study
successful Probability proof of efficacy study
(POE) successful if POC successful Probability
POE study successful if POC unsuccessful
Probability POE study successful
Cost of POC including any lost sales through
extra delay Cost of POE study Expected NPV
revenue if POE initiated immediately and
successful Value of strategy of POE only Value of
strategy of POC POE
Value of POC study
5
Model Continued
6
Example
Value of two strategies plotted against ?, the
probability POE successful if POC successful
7
(No Transcript)
8
Value of Biomarker Information in Terms of
Posterior Variance

• Suppose that over all products for this
  indication the correlation of true therapeutic
  and biomarker outcomes is 0.9
• Let the prior means be zero in this class
• Let the prior variances be 1
• Let the data variance of a minimal experiment be
  also 1
• Implies prior information equivalent to one
  minimal experiment

9
Here n is the number of minimal experiments we
run Of course we expect a biomarker experiment
to be cheaper than a therapeutic one Nevertheless
note that fairly rapidly there is no interest in
getting further biomarker information
10
A Serious Warning to Statisticians
In the mathematical formulation of any problem it
is necessary to base oneself on some appropriate
idealizations and simplification. This is,
however, a disadvantage it is a distorting
factor which one should always try to keep in
check, and to approach circumspectly. It is
unfortunate that the reverse often happens. One
loses sight of the original nature of the
problem, falls in love with the idealization, and
then blames reality for not conforming to it. De
Finetti 1975 The only way that human beings
could ever have survived as a species for long as
we have is that weve developed another kind of
decision-making apparatus thats capable of
making very quick judgements based on very little
information. Malcolm Gladwell, Blink, 2005
11
My Gloomy Take on This

• We dont really understand this topic
• There may be less value in proof of concept
  studies than we propose
• Therapeutic studies may be valuable even if they
  have low power
• There is no point in undertaking POC studies
  unless you can see circumstance under which they
  would cause you to cancel projects

12
Appropriate Attitudes for Cross-over Trials

• They are not suitable for all indications and
  questions
• They are extremely valuable for some indications
  and questions
• Carry-over has to be dealt with by washout
• Dont pre-test for carry-over
• Dont rely on classical statistical approaches to
  carry-over
• Cross-over trials have great potential in
  investigating individual response

13
Carry-over
Definition Carry-over is the persistence
(whether physically or in terms of effect) of a
treatment applied in one period in a subsequent
period of treatment. If carry-over applies in a
cross-over trial we shall, at some stage, observe
the simultaneous effects of two or more
treatments on given patients. We may, however,
not be aware that this is what we are observing
and this ignorance may lead us to make errors in
interpretation.
14
The simple carry-over model.
This is a very popular model amongst applied
statisticians of a mathematical bent. The model
assumes that if a carry-over effect is present 1)
it lasts for one period exactly 2) it depends on
the engendering treatment only and not on the
perturbed treatment.
15
Three Period Bioequivalence Designs

• Three formulation designs in six sequences
  common.
• Subjects randomised in equal numbers to six
  possible sequences.
• For example, 18 subjects, three on each of the
  sequences ABC, ACB, BAC, BCA, CAB, CBA.
• A test formulation under fasting conditions,
• B test formulation under fed conditions
• C reference formulation under fed conditions.

16
Weights for the Three Period Design not
Adjusting for Carry-over
17
Properties of these weights

• Sum to 0 in any column,
• eliminates the period effect.
• Sum to 0 in any row
• eliminates patient effect
• Sum to 0 over cells labelled A
• A has no part in definition of contrast
• Sum to 1 over the cells labelled B and to -1 over
  the cells labelled C
• Estimate contrast B-C

18
Weights for the Three Period Design Adjusting
for Carry-over
B-C contrast illustration of treatment effect
and elimination of period and patient effects
19
Weights for the Three Period Design Adjusting
for Carry-over
Illustration of elimination of carry-over
effects
20
Have We Got Something for Nothing?

• Sum of squares weights of first scheme is 1/3 (or
  4/12)
• Sum of squares of weights of second scheme is
  5/12
• Given independent homoscedastic within- patient
  errors, there is thus a 25 increase in variance
• Penalty for adjusting is loss of efficiency

21
The difference between mathematical and applied
statistics is that the former is full of lemmas
whereas the latter is full of dilemmas
22
The Dangers of Pre-testing

• Situation with AB/BA design
• Two-stage procedure is very badly biased
• CARRY and PAR are highly correlated
• 1/?2 lt ? lt 1
• Three treatment design
• Same problem
• Carry-over and adjusted estimates correlated
• ? 0.45

23
The Phoenix Bioequivalence Trials

• Analysed by DAngelo, Potvin Turgeon
• 20 drug classes
• 1989-1999
• 12 or more subjects
• 96 three period designs
• 324 two period designs

D'Angelo, G.Potvin, D.Turgeon, J. J Biopharm
Stats, 11, 27-36, 2001
24
Three Treatment Designs P-Values for Carry-Over
Significant results in bold
Senn, S. J., G. D'Angelo, et al. (2004).
"Carry-over in cross-over trials in
bioequivalence theoretical concerns and
empirical evidence." Pharmaceutical Statistics
3(2) 133-142.
25
Two Treatment Designs
Significant results in bold
26
Test of Uniformity of P-Values
27
Galling as this may appear to statisticians, the
cure for carry-over is more biological and
pharmacological understanding not more statistics
28
Conclusions

• Distribution of P-values uniform
• no evidence of carry-over
• Carry-over a priori implausible
• presence testable by assay
• No point is testing for it
• leads to bias
• Or adjusting for it
• increased variance

29
Possible Strategy

• Run multi-period cross-overs
• Patient by treatment interaction becomes
  identifiable
• This provides an upper bound for gene by
  treatment interaction
• Because patients differ by more than their genes

30
Second cross-over Second cross-over Second cross-over
Responders Non-Responders Total
First cross-over Responders 24 0 24
First cross-over Non-Responders 0 8 8
First cross-over Total 24 8 32
31
Second cross-over Second cross-over Second cross-over
Responders Non-Responders Total
First cross-over Responders 18 6 24
First cross-over Non-Responders 6 2 8
First cross-over Total 24 8 32
32
Advantages and DisadvantagesPRO
CON

• Cheap
• Low tech
• Insight into sources of variation gained

• Only suitable for chronic diseases
• Demanding of patients time
• Unglamorous
• Does not produce diagnostic patents

33
An Overlooked Source of Genetic Variability

• Humans may be classified into two important
  genetic subtypes.
• One of these suffers from a massive chromosomal
  deficiency.
• This is expressed in.
• Important phenotypic differences.
• A massive disadvantage in life expectancy.
• Many treatment strategies take no account of
  this.
• The names of these subtypes are...

34
Men and Women
35
A Difficult Decision

• You have 100
• Should you spend it on beer?
• US 20 beers
• UK 15 beers
• Or on books?
• In particular 1 book
• Have I mentioned this before?