Test Method Validation

1
Test Method Validation

• John Pfister, MS, RM(AAM)
• Microbiologist/Epidemiologist
• Wisconsin State Laboratory of Hygiene
• 608-262-7404, jp_at_slh.wisc.edu
• WSLH Audio Conference Program
• December 17, 2003

2
Presentation Objectives

• explain concepts of method validation
• review terminology
• identify regulatory requirements
• outline experimental procedures
• introduce data analysis tools
• discuss decision-making
• focus on qualitative, not quantitative,
  laboratory tests

3
Introduction

• A laboratory test should consistently produce
  accurate, relevant and timely results at an
  acceptable cost
• A laboratory must use comprehensive, systematic
  procedures for
• evaluation of new methods (Verification)
• ongoing review of established methods
  (Validation)

4
CLIA Regulation

• 493.1253(b)(1) Verification of performance
  specifications.
• Each laboratory that introduces an unmodified,
  FDA-cleared or approved test system must do the
  following before reporting patient test results
• (i) Demonstrate that it can obtain performance
  specifications comparable to those established by
  the manufacturer for the following performance
  characteristics (A) accuracy, (B) precision, (C)
  reportable range of test results for the test
  system.
• (ii) Verify that the manufacturers reference
  intervals (normal values) are appropriate for the
  laboratorys patient population.

5
Method validation is all about error assessment

• Assess how much error and what type of error
  might be present in a test result produced by a
  method in your laboratory
• Evaluate the ability to detect or minimize these
  errors
• Ensure that these errors wont affect the
  interpretation of the test result and thereby
  compromise patient care

6
Possible Sources of Error

• Different lots of calibrators and reagents
• Changes in instrumentation
• Effects of shipment and storage
• Local climate control conditions
• Quality of water and other reagents
• Contamination/cross-contamination
• Timing of procedure steps
• Labeling, transcribing
• Skill of the analyst

7
Evaluation of a Laboratory Method

• Define purpose of test
• Decide on analyte
• Decide on reference method (gold standard)
• Determine the clinical (or public health)
  usefulness and test performance requirements
• Survey the literature
• Evaluate test characteristics
• Perform test verification

8
1.) Define Purpose of Test

• Screening test Performed in the absence of
  signs/symptoms in order to identify illness at an
  early stage or to identify predictors of future
  illness.
• Diagnostic test Performed in response to a
  sign/symptom, or condition known to be present in
  a patient
• Confirmatory test Performed after obtaining a
  positive screening or diagnostic test to ensure
  the accuracy of that initial result

9
2.) Decide on Analyte

• Organism
• Antigen
• Nucleic acid
• Antibody

10
3.) Decide on Reference Method or Gold Standard

• Often elusive not always available
• Often imperfect new test may be better than
  the gold standard
• Often inadequate may need to use multiple tests
  (expanded gold standard)
• Often elaborate or expensive cell culture,
  molecular assays, biopsy,surgical exploration

11
4.) Determine the clinical (or public health)
usefulness and test performance requirements

• Consider clinical and public health objectives
• Establish preliminary requirements for test
  sensitivity, specificity, predictive value, etc.
• Consult with end-users of the data clinicians,
  epidemiologists, etc.

12
5.) Survey the Literature

• Published and unpublished performance claims
• Critical review

13
6.) Determine Test Characteristics and
Requirements

• Costs
• Practicality
• Logistics
• Specimen Requirements
• Quality Assurance
• Safety

14
7.) Perform Test Verification

• Compare with selected reference method
• Test an appropriate number of specimens
• Resolve discordant results, if possible
• Conduct Error Assessment!
• Analyze the experimental data

15
Quantitative Tests

• Used to measure the concentration of an analyte
  usually expressed as a continuous range of values
  
• Method verification must consider
• Accuracy
• Precision
• Linearity
• Reportable range

16
Qualitative Tests

• Used to determine the presence or absence of an
  analyte dichotomous results (positive/negative,
  normal/abnormal)
• Method verification must consider
• Sensitivity
• Specificity
• Predictive Value

17
How Many Specimens?

• For commercially-available (FDA-cleared) test
  kits, evaluate
• At least 20 positives and 50 negatives (ASM)
• At least 50 positives and 50 negatives (NCCLS)
• For home-brew or modified tests, evaluate a
  minimum of 50 positives and 100 negatives

18
Test Verification

• A test is considered verified if
• it meets the users pre-determined requirements
• A thorough error assessment reveals no
  substantial risk of undetectable errors
• the calculated sensitivity and specificity are no
  lower than 5 below those of the reference
  method, those appearing in peer-reviewed
  journals, or those claimed by the manufacturer in
  the product insert

19
Test Verification Decisions

• Further verification studies are required
• The test is unacceptable for routine use until
  corrective action is taken by the manufacturer,
  the user, or both.
• The test is acceptable for routine use

20
Ongoing Test Validation

• Quality Control
• Proficiency Testing
• Instrument Calibration
• Historical Data and Documentation
• Competency of Testing Personnel

21
Epictetus, ca. A.D. 55-135 Discourses, Chapter
xxvii.

• Appearances to the mind are of four kinds
• Things either are what they appear to be
• or they neither are, nor appear to be
• or they are, and do not appear to be
• or they are not, and yet appear to be.
• Rightly to aim in all these cases is the wise
  mans task.

22
Relationships Between Disease Status and Test
Result
DISEASE
TEST
23
Sensitivity

• The proportion of people with the disease who
  have a positive test.
• How good is the test at detecting infection in
  those who have the disease?
• A sensitive test will rarely miss people who have
  the disease (few false negatives).

24
Specificity

• The proportion of people without the disease who
  have a negative test.
• How good is the test at calling uninfected people
  negative?
• A specific test will rarely misclassify people
  without the disease as infected (few false
  positives).

25
Sensitivity and Specificity
DISEASE
TEST
Sensitivity TP/TPFN Specificity TN/TNFP
26
Sensitivity vs Specificity

• Desirable to have both high, but usually not
  possible
• Trade-offs
• Establishment of cut-off point
• Optimal performance often achieved through
  combinations of tests (algorithms)

27
Hypothetical Test Results
28
Predictive Value

• The probability of the presence or absence of
  disease, given the results of a test
• PVP is the probability of disease in a patient
  with a positive test result.
• PVN is the probability of not having disease when
  the test result is negative.

29
Predictive Value

• How predictive is this test result for this
  particular patient?
• Determined by the sensitivity and specificity of
  the test, and by the pre-test likelihood that the
  disease is present in the patient being tested
  (or the prevalence of disease in the population
  being tested).

30
Predictive Value
DISEASE
TEST
Predictive Value Positive (PVP)
TP/TPFP Predictive Value Negative (PVN) TN/TNFN
31
Hypothetical HIV Antibody Test Verification
Study, n70
Gold Standard
Test

Sensitivity 19/20 95.0 Specificity 48/50
96.0 Predictive Value Positive (PVP) 19/21
90.5 Predictive Value Negative (PVN) 48/49
98.0
32
Hypothetical HIV Test PerformancePrevalence
20.0 n1,000
Gold Standard
Test
Sensitivity 190/200 95.0 Specificity
768/800 96.0 Predictive Value Positive (PVP)
190/222 85.6 Predictive Value Negative (PVN)
768/778 98.7
33
Hypothetical HIV Test PerformancePrevalence
2.0 n1,000
Gold Standard
Test
Sensitivity 19/20 95.0 Specificity 941/980
96.0 Predictive Value Positive (PVP) 19/58
32.8 Predictive Value Negative (PVN) 941/942
99.9
34
Predictive Value Positive
35
Improving Predictive Value

• It is possible to increase the pre-test
  likelihood (and thereby improve the predictive
  value) by targeting testing to those at highest
  risk, or by selectively screening specific
  populations.

36
Summary

• Test verification and validation are integral
  parts of a laboratorys on-going quality
  assurance program
• Assays must be evaluated in the context of how
  and where the test is intended to be used
• Test verification and validation must involve a
  comprehensive error assessment

37
Epictetus, ca. A.D. 55-135
What concerns me is not the way things are, but
rather the way people think things are.
38
References

• U.S. Department of Health and Human Services.
  Medicare, Medicaid and CLIA Programs Laboratory
  Requirements Relating to Quality Systems and
  Certain Personnel Qualifications. Final rule. Fed
  Regist 2003 683640-3714.
• Elder BL, Hansen SA, Kellogg JA, Marsik FJ,
  Zabransky RJ. Verification and Validation of
  Procedures in the Clinical Microbiology
  Laboratory. Coordinating ed., McCurdy BW.
  American Society for Microbiolgy (ASM), 1997.
  (Cumitech 31)
• National Committee for Clinical Laboratory
  Standards (NCCLS). User Protocol for Evaluation
  of Qualitative Test Performance Proposed
  Guideline. Document EP12-P NCCLS, 1999.
• National Committee for Clinical Laboratory
  Standards (NCCLS).Assessment of the Clinical
  Accuracy of Laboratory Tests Using Receiver
  Operating Characteristic (ROC) Plots Approved
  Guideline. Document GP10-A NCCLS, 1995.
• Westgard JO. Basic Method Validation. Westgard
  QC, Inc 2nd ed. 2003.
• Fletcher RH, Fletcher SW, Wagner EH. Clinical
  Epidemiology The Essentials. William Wilkins
  3rd ed, 1996.