Validation, Verification and Quality Control

1
Validation, Verificationand Quality Control

• Thomas Novicki Ph.D. D(ABMM)
• Marshfield Clinic
• Marshfield WI

2
Todays Topics

• Validation and Verification of ID and AST systems
• Quality Control (QC) of Identification (ID) and
  Antimicrobial susceptibility test (AST) systems
• (We will not be discussing proficiency testing
  due to time constraints)

3
Disclosures

• I have nothing relevant to this presentation to
  disclose

4
Validation and VErification
5
Validation and Verification

• Validation Documenting in vitro diagnostic
  device performance in your lab
• Verification An ongoing demonstration of the
  vendors performance claims using a variety of
  tools

6
Validation and Verification(or is it
Verification and Validation?)

• Note that some entities (e.g. CLSI, CLIA) reverse
  the definitions
• Initially Verify
• Then periodically Validate
• The definitions used dont matter, but their
  consistent use in your lab is important.
• What does your accrediting body use?

7
Who Guides Us?

• Vendors of commercial systems
• The Clinical and Laboratory Standards Institute
  (CLSI)
• The Feds
• CLIA Clinical labs
• FDA Instruments
• Your accrediting body (COLA, CAP, JC)
• (Did I miss anything?)

8
Validation and Verification

• The Centers for Medicare and Medicaid Services
  (CMS), (a part of the US Department of Health and
  Human Services) regulates clinical laboratory
  testing through the Clinical Laboratory
  Improvements Amendments (CLIA) that became
  effective in 2003.
• (Recall that it all began in 1988)

9
Validation

• For FDA cleared and approved tests, the lab
  demonstrates performance claims by the device
  manufacturer, typically found in the product
  label (package insert)
• For lab-developed tests (LDTs), a more rigorous
  set of studies demonstrating performance is
  necessary

10
Validation of LDTs

• Will not discuss further except to say that any
  modification of an FDA IVD makes it an LDT
• Be wary of modifying the more critical
  FDA-approved test (e.g. TB, HIV IVD devices )
• Qualitative tests easier to modify than
  quantitative
• Simple cleared tests (e.g. MRSA chromo-agars for
  additional anatomical sites) are reasonably
  modified

11
iClick? 1

• CLIA requires a lab to validate which of the
  following for every new diagnostic test
• Accuracy
• Precision
• Reportable and Reference ranges
• All of the above
• 1 and 2

12
iClick? 1 - Discussion

• CLIA specifies that the user of an FDA IVD device
  of Moderate/High complexity will
• Validate accuracy,
• Precision,
• Reportable range, and
• Reference range
• You must satisfactorily complete validation of
  the device before beginning its use.
• You must document all validation study data and
  maintain it for at least 2 years after test
  retirement.

13
Validation AST

• Two methods for an unmodified FDA-cleared
  instrument
• Compare to a reference method
• Compare to an existing system
• Method B probably the best fit for most labs with
  an existing system For method A, see Cumitech 31A
• (Cumitech 31A, Verification and validation of
  procedures in the clinical microbiology
  laboratory, ASM Press 2009)

14
Disclaimer What Im about to say on validation
is more what youd call guidelines than actual
rules
15
Validation AST

• Source of samples/isolates to test
• gt 30 well-characterized clinical isolates per
  panel that reflects the mix common to your
  population and also some with unusual AST
  patterns
• Can include proficiency or commercial validation
  panels, but be wary
• Reflective of your population?
• Incorrect sample matrix?
• If from the device manufacturer, a
  self-fulfilling prophecy?
• Note that with care a single panel can be used
  for both AST and ID validations

16
Validation AST

• How to deal with discrepancies when two
  non-reference systems are compared?
• ? No reference method, so grade results, then
  apply predetermined limits on the number and
  types of discrepancies

17
Validation AST

• For each bug/drug result, calculate s for
• Categorical (S/I/R) Errors
• Minor Error I vs S or I vs R
• Major Error S vs R or R vs S
• No Very Major Error category True result is
  not known
• Essential Errors (gt 1 MIC dil. difference)
• Can only score when both methods generate
  discrete numbers (i.e. do not score lt or gt values)

18
Validation AST

• Using the total number of evaluable bug/drug data
  points, an acceptable validation has
• lt 5 Major Errors AND
• lt 10 Major Minor Errors AND
• lt 10 Categorical Errors AND
• lt 10 Essential Errors
• Also look for trends in a particular drug or bug
  which may point towards a problem area
• Consult your vendor rep with any validation
  problems

19
Validation AST

• Precision (reproducibility)
• Test five isolates X3 for three to five days
• Calculate intra-and inter-run categorical and
  essential agreements
• Isolates should have known susceptibility
  profiles, such as ATCC strains
• Use both Gram positive and Gram negative strains
  if validating GP and GN panels
• Acceptable precision gt 95agreement

20
Validation ID

• Test a range of Gram positive and Gram negative
  organisms IDd by reference or current method(s)
• Minimum of 20 patient isolates larger labs test
  more. Additionally, test QC strains
• lt 10 discrepancies between reference and new IDs

21
Validation ID

• Level of agreement may need to be adjusted e.g.
• Old ID of Coag-negative Staphylococcus sp
• New instrument IDs to species level
• Precision
• Test 2-4 isolates X3 for three to five days
• Calculate intra- and inter-run agreements
• Accept gt 95 agreements

22
Validation

• What to do if validation fails? Do not use the
  new test and either
• Withdraw test from further consideration OR
• Develop and implement a corrective plan with the
  manufacturer. Then, re-validate

23
Verification Component of the QA Process

• An ongoing, multi-faceted process that assures
  the validated test continues to perform at the
  expected level

Personnel competency Proficiency challenges
QC organisms or analytes Comparison with other labs
Instrument PM/calibration Trend analysis
Discrepant resolution Pre- and post-analytic controls

Written procedures documenting the process Written procedures documenting the process
24
Quality Control
I think, therefore I control quality
25
iClick? 2

• You have an instrument that uses micro-broth
  dilution MIC plates in your lab. This system
  incubates and reads plates automatically but also
  has a manual plate reader. Your tech rep tells
  you to read a plate manually if QC fails on the
  instrument. If QC then passes, your QC is
  acceptable for the entire system and no follow-up
  action is needed.
• True
• False

26
iClick? 2 discussion

• QC is performed to assure the accuracy of a test
  result by assessing all analytical components
• Reagents
• Instrument (where applicable)
• Test precision (i.e. repeatability)
• Operator technique
• Data interpretation

27
CAP on AST QC

• For AST of either disk or dilution type, control
  organisms are tested with each new lot or batch
  of antimicrobials or media and each day the test
  is performed thereafter.
• (Note that gt3 failures per month per drug/bug
  combo is unacceptable)

28
CAP on AST QC

• However, the frequency of test monitoring may be
  reduced to weekly if the laboratory can document
  satisfactory performance with daily control tests
  as suggested by CLSI guidelines.
• The CAP checklist then goes on to briefly
  describe how to do so

29
AST QC Resources

• COLA, JC have similar verbiage, but is the
  checklist guidance enough?
• IMHO, No!
• You need CLSI AST documents in your lab.
• (In my humble opinion.)

30
CLSI AST Documents

• M2 (disk diffusion) and M7 (MIC), 3 year cycle
• Test performance
• ? QC (including daily-to-weekly QC) ?
• M100, annual cycle
• Recommended drugs to test and report,
  interpretive breakpoints
• ? QC strains, QC troubleshooting, when to
  re-validate QC ?
• Others documents for other classes of
  microorganisms
• (Explore these and more at http//www.clsi.org/
  choose Microbiology from the Shop drop down menu)

31
iClick? 3

• How many of you have these CLSI documents in your
  lab
• 2012 M2 (disk diffusion), M7 (MIC) and M100
  (Breakpoints, QC)
• 2012 M7 and M100
• 2012 M2 and M100
• 2012 M2 and M7
• Any older versions of these documents
• None of the above

32
Conversion to Weekly AST QC

• When day-of-use QC is being successfully
  performed at least once a week, a lab may
  consider weekly AST QC by a validation process
• Test QC strains 20 or 30 consecutive days
• If 0-1 failures per drug-bug combo in 20 days OR
  2-3 failures in 30 days
• THEN Weekly QC may be implemented

33
Weekly AST QC

• Perform QC testing once per week and with any new
  reagent
• If weekly AST QC fails
• If readily identifiable error (e.g. wrong QC
  strain see M2 or M7 for more examples) correct
  error and retest once. If QC passes, document
  findings and continue weekly QC
• If no error is found

34
Weekly AST QC Re-validation

• Perform 5 consecutive days of QC on the failed
  drug-bug combo
• (We automatically take out a fresh QC strain from
  the freezer at this point)
• If ALL 5 days pass, document and resume weekly
  testing
• If a failure occurs again, STOP TESTING and
  thoroughly evaluate the process for error

New QC strain New lot of reagent
Correct process errors Consult the manufacturer
35
Weekly AST QC Re-validation

• Once a likely error has found, re-validate
• Do not include data from the prior 5-day sequence
• Must again pass these criteria
• 0-1 failures per drug-bug combo/20 days OR
• 2-3 failures per drug-bug combo/30 days
• Remember that, during re-validation the lab is on
  a daily QC schedule-a failure means that that
  drugs results are not released

36
AST QC References

• Disk diffusion
• M02-A11 Section 15
• M100-S22 Tables 3C, 3D
• MIC
• M07-A9 Section 16
• M100-S22 Tables 4F, 4G

37
AST QC Special Considerations

• M100-S22 Tables 3C 4F, Reference Guides to QC
  Frequency
• When and how much re-validation to perform when a
  system change occurs (e.g. AST instrument repair,
  Abx formula change Etc)
• Release of patient results when QC fails?
• CLSI says OK with many qualifiers, but our lab
  generally just says No

38
iClick? 4

• You are performing a weekly AST QC re-validation
  for drug X on your favorite automated ID/AST
  instrument when you incur a failure on the same
  AST panel for drug Y. What do you do?
• Ignore the failure of drug Y and press on with
  the 5-day testing of drug X
• Now treat the failure of drug Y as a second
  failure and start a 5 day test of drug Y
• Stop testing, evaluate the whole process, then
  perform a new 20/30 day validation on the whole
  AST panel

39
iClick? 4 discussion

• The correct answer isnt clear, but Marshfield
  Labs treats the failure of the second drug as a
  failure and starts a 5 day test on that drug also
• Why? ? Subsequent failures may reflect a larger
  problem

40
ID QC

• CLIA requires a lab to show positive and negative
  reactions for each new lot and/or shipment of
  biochemical.
• ? This includes kit and automated instrument ID
  methods!

41
(No Transcript)
42
ID QC

• There is a QC solution for automated ID
  instrument users
• Streamlined QC
• Stay Tuned for the Next Talk!

43
The End

• Thank you
• Questions?