FDA/Industry Workshop

1
Uses and Abuses of (Adaptive) RandomizationAn
Industry Perspective

• Benjamin Lyons, Ph. D. and
• Akiko Okamoto, Sc.D.

2
Outline

• Adaptive vs. Static Randomization
• Implementation Challenges
• Errors by Investigators
• Errors in Algorithm
• Errors related to Drug Supply
• Conclusion

3
Adaptive vs. Static Randomization

• Static randomization requires that one
  randomization list is generated at the start of
  the trial.
• Adaptive (Dynamic) randomization algorithms
  (e.g., Urn model) assign treatments based on
  patient characteristics and previous treatment
  assignments.

4
Covariate Adaptive Randomization

• Treatment assignment of the (n1)st patient may
  depend upon the previous first n patients.
• Usual mechanism is a balance function that is
  minimized by assigning the new patient to a
  certain treatment.

5
Why use adaptive randomization?

• Treatment balance required within each level of
  stratification factors.
• For small trials with many stratification factors
  static-stratified randomization will not insure
  balance within each strata or overall.

6
Why avoid adaptive randomization

• May be hard to interpret using standard theory
  (see recent CPMP guidelines on adjustments for
  baseline covariates).
• Many chances to make errors.
• Implications of some errors on inference are not
  easy to understand in the context of standard
  theory.
• Some errors may put trial validity at risk.

7
Implementation Challenges

• Three types of errors
• Errors by investigators
• Errors in the algorithm
• Errors caused by a faulty drug supply method.

8
Example 1 Site Error

• Site enters the wrong strata level for a patient.
• Site assigns the wrong medication kit and perhaps
  treatment to patient.

9
Response

• Do we update the balance function by altering the
  assignment weights to reflect error?
• If corrected there are three categories of
  balance functions
• randomized before the error
• randomized after the error but before the
  correction
• randomized after the correction.
• If not corrected there are only two.

10
Analysis

• How do you report this?
• Are the pre-specified test statistics
  asymptotically valid?
• For stratification error is there a sensitivity
  analysis?
• How should you incorporate into a permutation or
  or resampling procedure?

11
Prevention

• Site training.
• Train sponsor staff on how to react to the error.
• Giving IVRS vendor staff explicit instructions on
  who decides to update the algorithm.
• Is it sound to alter the algorithm for a few
  minor errors?

12
Example 2 Algorithm Error

• Specification is correct for 11 assignment as
  indicated by simulation in SAS.
• Actual code to calculate assignment written in an
  SQL program.
• Validation of SQL program did not include any
  simulation.

13
Result

• Error in SQL program detected after 50
  enrollment. Balance is 21.
• Program fixed so that the balance at the end of
  the trial is 11.
• Probability of treatment assignment correlated
  with date of trial entry.

14
Analysis

• Is this trial randomized?
• Are the standard test statistics asymptotically
  valid.
• How should we account for the error in any
  permutation test?
• Should the trial results be reported at all?
• Could entry time be correlated with patient
  characteristics and hence outcome?

15
Prevention

• Validate the actual software that produces the
  assignment through simulation prior to roll out.
• Check balance results frequently during the
  trial.
• Vendor must have a responsible/trained
  statistician who understands the issues.

16
Example 3 Drug Supply

• Supply at sites is not adequate.
• Lack of study drug.
• Drug not re-supplied often enough.
• High enrollment in short periods.
• Uneven enrollment by site.
• In some cases all treatment arms are not
  available when a subject is randomized.

17
Response

• System provides over rides or forced
  randomizations
• the patient is assigned to available treatment
  regardless of what the algorithm says.
• Adaptive algorithm is ignored for this patient.

18
Result

• Trial should be balanced if only a few
  occurrences.
• Forced assignment included in the balance
  function.
• The algorithm has not been implemented as stated
  in the protocol and the report.
• Are subsequent randomizations that used the
  faulty balance function valid?

19
Analysis

• Are the standard test statistics asymptotically
  valid?
• How does a permutation test account for the over
  rides?
• How many forced assignments before the entire
  randomization is suspect?

20
Prevention

• Supply trials with dynamic randomization
  centrally with one kit going to each site after
  each randomization.
• OR
• Have abundant supply at all sites. OR
• Do not allow forced randomization, turn patients
  away if all arms not available.

21
Simulation

• 171 Subjects.
• Two treatment Arms A and B.
• 4 Strata Site (16) and three prognostic factors
  (2,2, and 4 levels).
• Randomization by Biased Coin.
• Entry time , stratification and response based on
  CNS trial.
• Assignment is altered in 10,000 replications.

22
Supply Algorithm

• Each site began with 4 kits 2 A and 2 B.
• Re-supplied in 1 day with two kits when one arm
  is empty.
• Patients may enter with only 1 arm available.
• If arm assigned by IVRS was missing then the
  remaining treatment was given.
• Drug supply is part of the simulation.

23
Results

• For 10000 trial simulations
• Average of 5 forced randomization per trial
• T-statistic calculated for each trial
• Distribution similar to the theoretical.
• Supply error has no effect.

24
Conclusion

• Adaptive Randomization is more difficult to
  execute then static randomization.
• There are several sources of error.
• Result of errors are poorly understood.
• Some errors may be minor errors.
• Using Adaptive randomization adds costs and risk
  to running a trial.