Statistical issues in the validation of surrogate endpoints

1
Statistical issues in the validation of
surrogate endpoints

• Stuart G. Baker, Sc.D.
• sb16i_at_nih.gov

2
Surrogate endpoint definition

• used to make conclusions about the effect of
  intervention on true endpoint
• obtained sooner, at less cost, or less invasively
  than the true endpoint

3
Outline

• Asking the right questions
• Hypothesis testing
• General framework for validation and application
• Graphical view
• Estimation (meta-analytic)
• General framework for validation and application
• Trial-level statistics graphical view
• Predicted effect of intervention for binary
  surrogate and true endpoints (NEW approach)
  graphical view
• Caveats

4
Asking the right questions
5
Asking the right questions

• Validation trial Both surrogate and true
  endpoints are observed
• QUESTION Are the conclusions about the effect of
  intervention on true endpoint the same when based
  on
• (i) only surrogate endpoint
• (ii) only the true endpoint ?
• Application trial Surrogate but not true
  endpoint is observed
• QUESTION What is the effect of intervention on
  true endpoint?

6
Hypothesis testing
7
General Framework for Hypothesis Testing
If Prentice Criteria hold, valid hypothesis
testing using surrogate endpoint
Surrogate endpoints True endpoints
Validation trial
Validation
Test hypothesis using surrogate endpoint
New trial
Surrogate endpoint
Application
Valid hypothesis test H0(T) no effect of
intervention on true implies H0(S)
no effect of intervention on surrogate
so that reject H0(S) implies reject
H0(T) Prentice Criteria pr (true surrogate)
not depend on group extra requirement (if
binary surrogate predicts true)
(Buyse,Molenbergs, 1998)) easy to reject hard
to show they hold
8
Understanding hypothesis testing

• Graphical illustration using
• Binary surrogate endpoint
• Binary true endpoint

9
Prentice Criterion holds
Fraction with true endpoint
Treatment A
Validation Trial
Treatment B
Fraction with surrogate endpoint
0
0
Extrapolation Prentice Criterion holds (but
how close is close enough?)
Fraction with true endpoint
B
A
Application trial
0
Fraction with a surrogate endpoint
B
A
1
0
A B for true
implies A B for surrogate
10
Prentice Criterion does not hold
Fraction with true endpoint
Treatment A
Validation Trial
Treatment B
Fraction with surrogate endpoint
0
0
Fraction with true endpoint
Hypothesis testing gives incorrect conclusion
Extrapolation same lines
A
B
Application trial
Fraction with a surrogate endpoint
0
B
A
1
0
A B for true
does not imply A B for surrogate
11
Estimation

• Meta-analytic (based on multiple previous trials)

12
General Framework for Estimation
Surrogate endpoints True endpoints
Previous trials
Predicted effect of intervention on true endpoint
model
Validation (similar confidence intervals)
Surrogate endpoint
Validation trial
Observed effect of intervention on true endpoint
True endpoint
Application trial
Predicted effect of intervention on true endpoint
Surrogate endpoint
Application
13
Meta-analytic methods of estimation

• Trial-level statistics
• Buyse et al (2000) Gail et al (2000)
• Estimated predicted effect of intervention on
  true endpoint
• proposal for binary surrogate and true endpoints
• simple computations

14
Focus

• Binary surrogate endpoint
• Binary true endpoint

15

DATA SCHEME
Application trial
Validation trial
Previous trial 1
Previous trial 2
Previous trial 3
16
Meta-analysis of trial-level statistics

• Graphical overview of approach of Buyse et al
  (2000) and Gail et al (2000)

17
Trial-level meta-analysis (simplified overview)
Regression using random effects and within trial
data
Previous study 1
Fraction with true endpoint
Previous study 2
Previous study 3
d
0
Fraction with surrogate endpoint
B
A
1
0
18
Meta-analysis of estimated predicted effects of
intervention

• A new approach for binary surrogate and true
  endpoints

19
Predicted effect of intervention on true endpoint
based on surrogates A and B in new study and data
from previous study 1
Fraction with true endpoint
d1
0
Fraction with surrogate endpoint
B
A
1
0
Note Lines for each group need not be
identicalPrentice Criterion not needed
20
Predicted effect of intervention on true endpoint
based on surrogates A and B in new study and data
from previous study 2
Fraction with true endpoint
d2
d1
0
Fraction with surrogate endpoint
B
A
1
0
Note Lines for each group need not be
identicalPrentice Criterion not needed
21
Predicted effect of intervention on true endpoint
based on surrogates A and B in new study and data
from previous study 3
Fraction with true endpoint
d2
d1
d3
0
Fraction with surrogate endpoint
B
A
1
0
d(d1w1 d2w2 d3w3)/(w1w2w3 )
22
Meta-analysis of estimated predicted treatment
effects d1, d2, d3

• d (d1 w1d2 w2 d3 w3) /(w1w2w3),
• Weights wi are based on a random-effects model
  for di (with variance s2)
• simpler than a random-effects for slopes
• wi 1 / (sampling variance of di s2 )
• Weights minimize variance of d if di are not
  correlated
• simplification since dis are correlated

23
Meta-analysis computation

• d (d1 w1d2 w2 d3 w3) /(w1w2w3), where
• difi0A pAfi1A(1-pA ) - fi0B pBfi1B(1-pB )
• Application or validation trial fraction with
  surrogate endpoint pAand pB
• Previous trials fraction with true given
  surrogate endpoint fi0A, fi1A, fi0B, fi1B
• wi1/(Vi s2 ), Vi sampling variance of (di)
• To estimate s2
• adapt method of DerSimonian and Laird for usual
  meta-analysis accounting for covariance among
  dis due to share parameters pAand pB
• To compute variance of d
• Bootstrap trials and data within trials

24
Meta-analysis simulation

• d (d1 w1d2 w2 d3 w3) /(w1w2w3), where
• difi0A pAfi1A(1-pA ) - fi0B pBfi1B(1-pB )
• Simulation
• Generate random fi0A, fi1A, fi0B, fi1B
• Generate random data for each trial
• Mean squared error
• Slightly smaller for meta-analysis of predicted
  effect of intervention than for meta-analysis of
  trial-level statistics (computed via
  method-of-moments)

25
Hypothetical data Example 1
26
Hypothetical data Example 2
27
Real Data (x 10) from multicenter trial in Gail
et al (2000) surrogate is cholesterol level,
true endpoint is artery diameter
28
Caveats
29
Caveats

• Needed even if surrogate is validated with data
  from many previous studies
• Extrapolation to a new trial
• Hypothesis testing
• Estimation using data from previous trials
• Surrogate endpoint does not predict harms that
  might arise after surrogate is observed

30
When caveats are less critical

• Preliminary drug development when the surrogate
  endpoint is used to decide on further development
  or definitive testing with a true endpoint
• Establishing dose or timing of an intervention
  previously shown effective using true endpoint at
  a different (suboptimal?) dose or timing

31
Summary
32
Types of trials

• Validation trial
• Both surrogate and true endpoint
• Do you obtain the same conclusion about effect of
  intervention on true endpoint using (i) surrogate
  endpoint and (ii) true endpoint?
• Application trial
• Only surrogate endpoint
• What is the effect of intervention on true
  endpoint?

33
Hypothesis testing

• Not validated if reject Prentices criteria
• Not clear what to conclude about surrogate if
  cannot reject Prentices criteria

34
Estimation (meta-analysis)

• Not need Prentices criteria
• Meta-analysis of trial-level statistic
• Applicable to all types of endpoints
• Meta-analysis of estimated predicted effect of
  intervention on true endpoint
• Binary surrogate and true endpoints
• Computationally simple
• Slightly smaller MSE than with meta-analysis of
  trial-level statistics (in simulation)