Lynn Torbeck

1
Preventing OOS Deficiencies

• Lynn Torbeck

2
List of Topics

• Briefly review
• Barr Case
• FDA OOS Guidance
• Able Laboratories Story
• PDA Scientific Advisory Board Committees
• Troublesome fundamentals
• Unresolved issues
• Preventing OOS deficiencies
• Final recommendations

3
Barr Case

• Audited in 1989, 1991 and 1992.
• Refused to accept a consent decree.
• FDA was forced to go to court.
• Civil action taken June 1992.
• Decision in favor of the FDA on February 4, 1993.

4
Barr and Statistical Issues

• Initial investigations
• Full investigations
• Testing
• Retesting
• Averaging
• Outliers techniques

5
(No Transcript)
6
Barr Lessons Learned

• FDA takes OOS issues very seriously.
• OOS SOPs, laboratory logs and documented
  investigations will be part of any Quality System
  review.
• Companies are still getting Form 483 observations
  for not having an adequate SOP or for not
  following the SOP.

7
Barr OOS Prevention

• Analysts, supervisors and managers should read
  and discuss the Barr case and understand the OOS
  issues in context.

8
FDA Guidance

• Investigating Out of Specification (OOS) Test
  Results for Pharmaceutical Production.
• Issued as a draft in September 1998.
• Still in draft as of today.
• FDA has sent it to the attorneys.
• Final version could be out this year.

9
Draft OOS Prevention

• All laboratory personnel, analysts, supervisors
  and managers should read, study and discuss
  in-depth, sentence by sentence if necessary, the
  draft OOS guidance.
• Then do it again when the final guidance is
  released.

10
Able Labs Cranbury, NJ

• Massive number of OOS errors
• Recall of all 46 products
• 3,184 lots recalled
• Five ANDAs withdrawn
• Hundreds of staff laid off
• Sold to Sun Pharm in December 2005
• www.ablelabs.com

11
Able Labs Lessons Learned

• It is still possible to have wide spread
  misunderstanding of the Barr case, the OOS
  guidance and OOS SOPs.
• Apparently the analysts felt they could not give
  an incorrect result.
• Management needs to instill and cultivate a GMP
  Culture in the analytical laboratory.

12
Able Labs OOS Prevention

• Review the Able Labs web site.
• Discuss the Able Labs story with laboratory
  analysts, supervisors and managers.
• Discuss what a GMP Culture means in the
  analytical laboratory and how to develop and
  reward it.

13
PDA OOS Committees

• Chemical OOS
• Lynn Torbeck, Chair
• Eight members
• Draft technical report reviewed by the FDA
• Planning a PDA/FDA conference
• Microbial Data Deviations
• Jeanne Moldenhauer, Chair
• Draft in revision

14
Troublesome Fundamentals

• Outliers
• Reportable Values
• Averaging
• Testing into compliance
• Full consideration

15
Outliers - Defined

• Extreme values vs outliers

16
Outliers Judge Wolin

• "The USP expressly allows firms to apply this
  test (outlier) to biological and antibiotic
  assays, ..., but is silent on its use with
  chemical tests.
• "In the Court's view the silence of the USP with
  respect to chemical testing and outliers is
  prohibitory."

17
Outliers - Investigation

• "In chemical procedures, where method accuracy
  variation is small, an outlier test may be
  appropriate as part of an OOS investigation,
  provided the sample and test procedure assumes
  homogeneity ... as in the composite strength
  assays. Our current thinking is that outlier
  tests are never appropriate where the purpose of
  the sample is to measure uniformity" Paul Vogel,
  September 10, 1993.

18
Outliers - Tests

• Dixon's criteria, the test in USPlt111gt, is
  general in nature and not specific to biological
  issues. It can be used anywhere the statistical
  assumptions can be met.
• In general, statisticians agree with the
  philosophy that outlier tests should be used
  infrequently and with great caution.

19
Outliers - Recommendations

• Don't use any outlier rejection test for
  rejection of chemical test results. But it can be
  used as supporting information in an OOS
  investigation to consider retesting.
• Keep all data, especially suspect data, for
  future review. Unusual data when seen in context
  and with other historical data often is not
  unusual at all, but in fact forms a known and
  well-behaved statistical distribution.

20
Reportable Values

• Reportable Values for Out of Specification Test
  Results
• Lynn Torbeck
• Pharmaceutical Technology
• Vol. 23, No. 2, February 1999
• Special Supplement

21
FDA R.V. Definition

• It should be noted that a test might consist of
  replicates to arrive at a result. For instance,
  an HPLC assay result may be determined by
  averaging the peak responses from a number of
  consecutive, replicate injections from the same
  preparation. The assay result would be calculated
  using the peak response average.

22
FDA R.V. Definition

• This determination is considered one test and
  one result.

23
Implications of FDA Definition

• A reportable value is the end result of the
  complete measurement method as documented.
• It is the value compared to the specifications.
• It is the value used for official reports.
• It is usually the value used for statistical
  analysis.

24
Figure 1
25
Figure 2
26
Figure 3
27
Interpretation

• The individual determinations do not have to meet
  the specification.
• Individual determinations are not reported out of
  the lab.
• However the variability of the determinations is
  a system suitability issue.
• Set a limit on the standard deviation or RSD.

28
R.V. OOS Prevention

• Record in writing the operational definition of
  the Reportable Value for each test method in the
  method documentation, any protocols and any
  reports.
• Add Only this reportable value can be compared
  to the specification criteria.

29
Averaging

• Specifically, the arithmetic mean the sum of all
  of the numbers divided by the count of the
  numbers. 
• More generally, it is a value that represents the
  central point of a data set. (In this sense, it
  can include the arithmetic average, the median,
  the mode, the geometric mean or the harmonic
  mean.)

30
Averaging

• "... as a general rule, firms should avoid this
  practice, because averages hide the variability
  among individual test results.
• "Averaging is particularly troubling if
  testing generates both out-of-specification and
  passing individual results which when averaged
  are within specification. 
• "Here, relying on the average figure without
  examining and explaining the individual
  out-of-specification results is highly misleading
  and unacceptable."

31
Averaging

• "Averaging the results of tests intended to
  measure the uniformity of the test article is not
  current good manufacturing practice ... because
  it may hide the variability of the sample the
  test procedure is intended to detect. For this
  reason, all individual test results must be
  reported and evaluated on an independent basis"
  Paul Vogel, September 10, 1993.

32
Averaging OOS Prevention

• Do not average out of specification reportable
  values within specification reportable values to
  get an in specification result.
• Do not average reportable values for QA to make a
  decision. QA must see all individual reportable
  values, OOS and retests.

33
Testing Into Compliance

• Torbeck, L., Preventing the Practice of Testing
  into Compliance, Pharmaceutical Technology, Oct
  2002.
• Testing into compliance is the practice of
  ignoring valid information that should be used to
  make decisions.
• Such a practice is at best not scientific and at
  worst is fraudulent, illegal, and immoral.
• Such practices if found must be stopped.

34
Testing Into Compliance

• Averaging OOS results with in specification
  results to get an in specification result.
• Physically averaging powers, granulations and
  liquids to get in specifications results.
• If not part of the validated process.
• Discarding data or not recording data until is
  known to be in specification.
• Missing samples and rejected cans.
• Overwriting HPLC chromatograms.

35
Not Testing Into Compliance

• Large initial sample sizes are acceptable if all
  data generated is reported.
• Large number of retests are acceptable if all
  data generated is reported.
• Failing system suitability is not an OOS.
• Out of limits for an in-process adjustment is not
  an OOS.

36
Compliance OOS Prevention

• Train all laboratory personnel, analysts,
  supervisors and managers to be able to identify
  specific situations of testing into compliance.
• Train to be able to defend situations that are
  not testing into compliance during an audit.

37
Full Consideration

• For inconclusive investigations . The OOS
  result should be retained in the record and given
  full consideration in the batch or lot
  disposition decision.
• This statement has caused some discussion as it
  is considered to be vague and undefined. It can,
  I think, be defined in a simple way.

38
Full Consideration

• First, all QA decisions are made with the
  Reportable Values, both OOS and retests.
• Second, QA looks at the magnitude of the retest
  values compared to the specifications.

39
Full Consideration

• If the retest values are close to the target, the
  lot can be released.
• If the retest values are close to the limit that
  the OOS exceeded, technically the lot can be
  released, but QA should consider further
  investigation to determine why the retests are
  not at target.

40
Consideration OOS Prevention

• QA should detail and document the logic and
  rational for decisions based on retesting results
  after a OOS result is found.

41
Unresolved Issues

• Specification Limits for OOS?
• What size the retest sample?
• Second analyst?
• Statistical treatment of data?

42
Specification Limits for OOS?

• Regulatory Limits
• Release accept/reject
• Action limits, Cpk1.33
• Alert, Cpk1.0
• Warning limits
• Trend
• Validation limits

43
Specification Limits
44
Specification OOS Prevention

• Define in writing the levels of specification
  criteria.
• Justify in writing which specifications are
  considered applicable to OOS and why or why not.

45
What Size the Retest Sample?

• a matter of scientific judgment,
• retesting cannot continue ad infinitum.
• Such a conclusion cannot be based on on 3 of 4
  or 5 of 6 passing results, but possibly 7 of 8.
• will vary on a case by case basis
• an inflexible retesting rule is
  inappropriate.

46
What Size the Retest Sample?

• The number of retests should be specified in
  advance
• The number of tests should not be adjusted
  on-the-fly, as results are being generated.
• a firms predetermined testing procedure
  should contain a point at which testing ends and
  the product is evaluated.

47
What Size the Retest Sample?

• This is an unresolved issue and the statisticians
  are still publishing journal articles and
  discussing it.
• Barr case n7.
• Could be too much or not enough.
• Currently n 3 to n9.
• PDA OOS committee will recommend.

48
Retest References

• Hofer, J., Considerations when determining
  routine sample size for a retest procedure,
  Pharmaceutical Technology, Nov. 2003.
• Anderson, S., An alternative to the ESD approach,
  Pharmaceutical Technology, May 2004.

49
Retest OOS Prevention

• Define in writing the sample size for retests or
  define the procedure to be used to determine the
  sample size.
• Provide scientific justification.

50
Second Analyst

• Guidance suggests a second analyst.
• Issues
• Added complication and variation
• May not have a second analyst
• May not find the root cause
• Second analyst may not be as proficient
• Recommend that the manager decide and justify
  decision in writing.

51
Statistical Treatment of Data

• Statistical treatments of data should not be
  used to invalidate a discrete chemical test
  result.
• a statistical analysis may be valuable as one
  assessment of the probability of the OOS result
• Another way to say outlier rejection.

52
Preventing OOS Deficiencies

• Setting specification criteria
• Statistical Thinking
• Sources of variation
• Common cause vs. special cause
• Variation reduction
• Training
• Education

53
Setting Specification Criteria

• Two sides to the OOS issue.
• Incorrect limits are the major source of OOS.
• Many specifications were set early in the
  development process and may not be appropriate
  for the current process.
• Many specification were set using wishful
  thinking or incorrect approach.

54
Setting Specification Criteria

• Use historical data
• Use distribution analysis
• Normal, log-normal, exponential
• Dont use X bar ? 3S
• Use Statistical Tolerance Intervals
• X bar ? K S for the alert limits
• where K is based on confidence and percent of
  future values

55
Setting Specification Criteria

• For action limits, permit the average to vary and
  widen the Tolerance Limits
• For accept/reject limits, add a further allowance
  for stability.
• Consider the clinical results as part of the
  justification for limits.

56
Statistical Thinking

• All work occurs in a system of interconnected
  processes.
• All processes have variability.
• Process understanding and variability reduction
  is the key to success.
• Variation is the enemy.

57
Sources of Variation

• Common cause variation
• People
• Materials
• Methods
• Measurement
• Machines
• Environment
• Special cause variation
• One single factor changed

58
Common vs. Special Causes

• A plot of the data with natural limits
  illustrates common cause variation.
• A value that is larger than would be expected by
  chance alone is assumed to be due to a special
  cause. Use CAPA to find it.

59
Variability Reduction

• Display boards
• Operational definitions
• Work to target, Target ( Low, High )
• Flexible consistency
• Hold constant
• Mistake proofing
• High tech equipment

60
Training

• Training is for a specific task or SOP.
• The goal is consistency.
• Freelancing causes problems.
• Little background is provided.
• An in-depth understanding is not needed to be in
  compliance if the SOP is followed.

61
Education

• Someone needs to
• Learn and understand the basic philosophy and
  principles.
• Know the background as it relates to the topic.
• Understand the material well enough to be able to
  make difficult decisions with confidence and be
  able to defend them.

62
Need for Understanding

• Why was Able Labs out of compliance?
• Defend Reportable Values.
• Defend specifications applicable to OOS
• Defend not testing into compliance.
• Defend retest sample size.
• Why variability reduction is needed.

63
Final Recommendations

• Read and understand the Barr Case.
• Read and study in-depth the OOS Guidance. Once is
  not enough.
• Audit the company SOP against the Guidance line
  by line.
• Have an active program to reduce OOS results.
• Keep management informed.

64
Thank You

• That ends my presentation.
• We are now ready for questions and answers.

65
References

1. USA vs. Barr Laboratories, Inc. Civil Action No.
   92-1744, US District Court for the district of
   New Jersey, February 4, 1993.
2. FDA, CDER, Guidance for Industry, Investigating
   Out of Specification (OOS) Test Results for
   Pharmaceutical Production, September 1998.
3. WWW.AbleLabs.com
4. Torbeck, L., Reportable Values for
   Out-of-Specification Test Results,
   Pharmaceutical Technology, February 1999.
5. Torbeck, L., Preventing the Practice of Testing
   into compliance, Pharmaceutical Technology,
   October 2002.
6. Hahn, G and Meeker, W., Statistical Intervals,
   John Wiley Sons, 1991.
7. Torbeck, L., Statistical Thinking,
   Pharmaceutical Technology, July 2001.