Validation of screening methods (2002/657/EC)

1
Validation of screening methods (2002/657/EC)

• N. Van Wouwe

AFSCA-FAVV
2
Definition (2002/657/EC)

• Screening method
• used to detect the presence of a substance or
  class of substances at the level of interest.
• have the capability for a high sample throughput
  
• gt are used to sift large numbers of samples for
  potential non-compliant results.

Exemple ELISA, plate test, biosensor, receptor
test,
3
Definition (2002/657/EC)

• Minimum criteria to use an analytical method as
  screening method
• must be validated (traceability)
• must have a false compliant rate of lt5 (ß-error)
  at the level of interest

4
Performance characteristics for method validation
(screening)
Qualitative method identifies a substance on
basis of its chemical, biological or physical
propriety (binary response /-,
absence/presence) Quantitative method determines
the amount or mass fraction of a substance
(response numerical value of appropriate unit)
determination is mandatory
5
Validation of screening test

• Definition of the scope of the method
• Analyte of group of analytes
• Range of concentration
• List of matrices
• Initial validation with the most often used
  matrice in national monitoring program
• Detection capacity (CCß)
• Selectivity/Specificity
• Applicability/ Ruggedness/Stability
• Precision (only for semi-quantitative method)

If possible different sources of blank material,
different technicians, different days on the same
spiked sample
6
Validation of screening test

• Targeted test for 1 compound
• validation for this compound
• Targeted test for a family of compounds
• validation for 1 representative molecule of the
  family (antibody)
• Wide range test for more than 50 different
  molecules
• Validation for at least a list of representative
  compounds
• Common pattern of activity on a specific
  bacteria?
• Common way of action (acting target)?
• Published reference data on validation available?
  

7
Proposition of the CRL for antimicrobials (in
milk)
Representative Compound Antimicrobial class
Cefalonium/Cephapirin/Cefquinome CEPHALOSPORINS
Penicillin G/Cloxacillin PENICILLINS
Tetracycline/Doxcycline TETRACYCLINS
Gentamicin/Streptomycin/Spectinomycin AMINOGLYCOSIDES
Enrofloxacin/Flumequine QUINOLONES
Sulfathiazole/sulfaguanidine/Sulfamerazine SULFONAMIDES
Erythromycin/Tylosin MACROLIDES
Lincomycin LINCOSAMIDES
Thiamphenicol PHENICOLATED
Trimethoprim/colistine MISCELLANEOUS
8
Performance characteristics

• Detection capacity
• Selectivity/Specificity
• Applicability/ Ruggedness/Stability
• Precision (only for semi-quantitative method)

9
Detection capability (CCß)

• The smallest content of the substance that may be
  detected, identified and/or quantified in a
  sample with an error probability of ß
• In case of MRPL, CCß lowest concentration at
  which the method is able to detect truly
  contaminated sample with a statistical certainty
  of 1-ß
• In case of MRL, CCß concentration at which the
  method is able to detect the MRL concentrations
  with a statistical certainty of 1-ß

10
Detection capability (CCß)

• No permitted limit
• Analyse 20 blank materials gt CCa 3x
  signal/noise
• Analyse 20 blank materials fortified at CCa
• gt CCß CCa 1.64 x SDRW
• Calibration curve procedure (ISO 11843)
• Analyse of blank material fortified at 0 MRLP,
  0.5 MRLP, 1 MRLP, 1.5 MRLP and 2 MRLP
• Plot analytical results (y-axis) vs
  concentration(x-axis)
• CCa y-intercept (blank) 2.33 x SDRW
• CCß CCa 1.64 x SDRW

11
Detection capability (CCß)

• No permitted limit
• If no quantitative results
• Analyse fortified blank samples at and above CCa
• (n 20 / concentration level)
• CCß concentration level where only 5 false
  compliant results remain

12
Detection capability (CCß)
CCa
CCb
Blank
1.64xSDRW
2.33xSDblank
a1
ß5
Signal orConcentration
13
Detection capability (CCß)

• Permitted limit (MRL)
• Analyse 20 blank materials fortified at MRL
• gt CCa MRL 1.64 x SDRW
• Analyse 20 blank materials fortified at CCa
• gt CCß CCa 1.64 x SDRW
• Calibration curve procedure (ISO 11843)
• Analyse of blank materials fortified at 0.5 MRL,
  1 MRL, 1.5 MRL and 2 MRL
• Plot analytical results (y-axis) vs
  concentration(x-axis)
• CCa MRL 1.64 x SDRW
• CCß CCa 1.64 x SDRW

14
Detection capability (CCß)
CCa
CCb
MRL
1.64xSDMRL
1.64xSDRW
ß5
a5
Signal orConcentration
15
Performance characteristics

• Detection capacity
• Selectivity/Specificity
• Applicability/ Ruggedness/Stability
• Precision (only for semi-quantitative method)

16
Selectivity/specificity

• Specificity ability of a method to distinguish
  between analyte being measured and other
  substances
• problem of interference?
• F(measuring technique, class of compounds,
  matrices,)

17
Selectivity/specificity

• How to test specificity for qualitative screening
  method?
• Analyse 20 different blank samples and 20
  positive samples (blind study, same or different
  days/technicians)

Specificity 100 NA/N- Other parameters Accurac
y 100 (PANA)/(N- N) Sensitivity 100
PA/N False positive 100 FP/(N- N) False
negative 100 FN/(N- N)
True positive (N) True negative (N-)
test result positive Positive agreement (PA) False positive (FP)
test result negative False negative (FN) Negative agreement (NA)
18
Selectivity/specificity

• How to test specificity for semi-quantitative
  screening methods?
• Select potentially interfering substances
  (metabolites, derivatives,)
• Analyse relevant blank samples (n 20)
• Analyse fortified blank samples with interfering
  substances at a relevant concentration
• Estimate the effect of the interferences
• False identification?
• Influence in quantification?
• Identification of the target analyte is hindered?
  

19
Performance characteristics

• Detection capacity
• Selectivity/Specificity
• Applicability/ Ruggedness/Stability
• Precision (only for semi-quantitative method)

20
Applicability

• Scope of the method must be define in term of
• Matrix (solid/liquid matrix, type of tissue)
• Animal species
• To introduce a new matrix
• Analyse at least 10 different blank material
  fortified at level of interest for the new matrix
  (CCß) test of interferences
• If 10 positive results gt method applicable for
  the new matrix
• If 1 negative result gt 10 additional analyses
• If 1 negative resultgt CCß must be recalculated
  for the new matrix

21
Ruggedness

• Ruggedness the susceptibility of an analytical
  method to changes in experimental conditions
• sample material
• analytes
• storage condition
• environmental condition
• sample preparation condition

22
Ruggedness

• How to test ruggedness? (during development)
• Identify possible factor that could influence the
  results (the analyst, solvents, pH, T, rate of
  heating,)
• Vary each factor slightly
• If one factor is found to influence results of
  the representative molecule, conduct further
  experiments
• gt acceptability limits for this factor
• (in the method protocol)

Recommendation of CRL analyses of 10 blank and
10 spiked samples at the same concentration and
with minor change of factor to detect influence
on results
23
Stability

• Test are not necessary if stability data already
  exist (from other lab or from publication)
• To include in the validation report
• Stability test
• the analyte in solution
• the analyte in matrix
• Aliquots of a fresh solution or sample stored
  under different conditions (T and/or storing
  time)

24
Performance characteristics

• Detection capacity
• Selectivity/Specificity
• Applicability/ Ruggedness/Stability
• Precision (only for semi-quantitative method)

25
Precision (for quantitative screening)

• Precision the closseness of agreement between
  independent test results obtained under
  predetermined conditions
• Expressed in terms of imprecision / standard
  deviation of test results

26
Precision (for quantitative screening)

• How to test precision?
• Repeatability test
• within-laboratory reproducibility test(or
  intermediate precision)
• Reproducibility test (between laboratories
  interlaboratory studies)
• determination of RSD () lt Precision criteria

27
Precision (for quantitative screening)

• Repeatability
• 3 concentrations
• 1x 1,5x 2x MRPL
• 0,5 1x 1,5x MRL
• 6 replicates/level
• 3 times
• same conditions

• Within-laboratory Reproducibility
• 3 concentrations
• 1x 1,5x 2x MRPL
• 0,5 1x 1,5x MRL
• 6 replicates/level
• 3 times
• different conditions (analyst, env. condition,)

28
Precision (for quantitative screening)

• ANOVA treatment of data gt RSDr RSDRW
• Comparison with precision criteria
• Horwitz equation RSDR() 2(1-0.5logC)
• Criteria for repeatability RSDr 1/2 to 2/3
  RSDR
• Criteria for within-lab reproducibility RSDRW
  2/3 to 1 RSDR

!
For concentration lt 100 µg/kg, RSDR becomes too
high!
29
Other recommendations

• False negative rate lt5 Analyses of 20 negative
  and 20 positive samples in order to test the
  screening method (see selectivity).
• One QC sample must be added in routine and
  results must be added to the validation file
• Method transfer/Commercial test
• Bibliographical survey to compil the evaluation
  of performance of the test
• Collection of data from supplier on validation
  study
• Experimental plan to test skillness of technician
  to perform the test
• Use of QC sample
• Participation to proficiency test

30
Exemple analyse of PCDD/F by CALUX bioassay

• PCDD/F 17 toxic congeners to analyse in various
  matrices (TCDDmost toxic dioxin)
• Results expressed in TEQ (Sum (CCixTEFi)i1-17)
• MRL for each matrix (milk, meat, egg, fish oil,)
  
• MRL expressed in pg TEQ/g fat or ng TEQ/ kg
• Reference method GC-HRMS
• Screening method immunoassay, bioassay,

31
Exemple analyse of PCDD/F by CALUX bioassay
CALUX bioassay genetically modified cell-based
bioassay (luciferase) Amount of light produced is
proportional to the toxicity (TEQ) of extracts
All substances fixing the Ah receptor
32
Analyse of PCDD/F by CALUX bioassay

• Advantage
• Rapid
• Cheaper than GC-HRMS
• Time for analyses
• Disadvantage
• Various compounds can fix the Ah receptor (PAH,
  PCB, PHDD/F,)
• specificity!!!!

33
Analyse of PCDD/F by CALUX bioassay protocol
Extraction of fat
Clean-up on silica acid carbon columns
Fraction with PCBs
Fraction with PCDD/F
Fraction with interfering compounds
Evaporation
Reading plate
Dosing plate
34
Analyse of PCDD/F by CALUX bioassay validation

• Selectivity/specificity
• Ruggedness/Stability
• Precision
• Detection capability

35
Analyse of PCDD/F by CALUX bioassay selectivity

• Possible interfering compounds?
• PAH mostly in environmental sample
• PCB fractionation during clean-up
• Other compounds? (PHDD/F) dependant of the
  matrix? (matrix effect?)
• Results of the selectivity test
• No interferences for feedstuff, milk, egg, fat
• Interferences for fish oil
• CALUX results 2 x GC-HRMS results

36
Analyse of PCDD/F by CALUX bioassay selectivity

• Matrix effect for fish oil

37
Analyse of PCDD/F by CALUX bioassay ruggedness

• What are the critical point in the protocol?
• Carbon column (interferences)
• Solvent (interferences)
• Curve (results)
• Evaporation time (recovery)
• Age of CALUX cell line (RSD)

38
Analyse of PCDD/F by CALUX bioassay ruggedness

• Carbon column amount of carbon used

(Rdt PCDD/F 60)
(Rdt PCDD/F 80)
DX fraction
PCB fraction
Not collected fraction
(Rdt PCDD/F 80)
39
Analyse of PCDD/F by CALUX bioassay ruggedness

• Evaporation time

40
Analyse of PCDD/F by CALUX bioassay ruggedness

• Solvent tested before use on a TCDD solution
  (antagonist/agonist effect)
• Curve tested with an independant TCDD solution
• Age of CALUX cells new cell every 2 months

41
Analyse of PCDD/F by CALUX bioassay precision

• Validation protocol

Day 1 Day 2 Day 3 Day 4 Day 5 Day 6
Blank solvent 1 1 1 1 1 1
Blank sample 1 1 1 1
Sample at MRL/2 3 3 2
Sample at MRL 6 2 3 3
Sample at 2MRL 3 2 3
Quality sample 1 1 1 1 10 10
42
Analyse of PCDD/F by CALUX bioassay precision

• ANOVA results for the TEQ determination of PCDD/F
  in feedstuff by CALUX bioassay
• At MRL (0.75ng TEQ/kg) XMRL 0.751 ng TEQ/kg
• Sr 0.063 gt RSDr 8.4
• SRW0.073 gtRSDRW 9.7
• At MRL/2 (0.376ng TEQ/kg) XMRL/2 0.464 ng
  TEQ/kg
• Sr 0.051 gt RSDr 11
• SRW0.051 gtRSDRW 11
• At 2MRL (1.5ng TEQ/kg) X2MRL 1.571 ng TEQ/kg
• Sr 0.107 gt RSDr 6.8
• SRW0.115 gtRSDRW 7.3

RSD lt 30 (2002/70/EC)
43
Analyse of PCDD/F by CALUX bioassay detection
capacity

• CCß for the TEQ determination of PCDD/F in
  feedstuff by CALUX bioassay
• CCa MRL 1.64 x SRW
• CCa 0.75 1.64 x 0.073 0.87 ng TEQ/kg
• CCß CCa 2.33 x SRW
• CCß 0.87 2.33 x 0.073 1.04 ng TEQ/kg

2002/70/EC false negative rate lt 1 !
gt At a concentration of 1.04ng TEQ/kg, we are
sure that the sample is a positive sample with
99 certainty
44
Analyse of PCDD/F by CALUX bioassay confirmatory
range
?
NON COMPLIANT
COMPLIANT
SUSPICIOUS
MRL
CCa
CCb
CC
-2.33sMRL
1.64sMRL
2.33ssample
Signal orConcentration
1
a5
ß5
45
Analyse of PCDD/F by CALUX bioassay confirmatory
range

• Lower limit of the confirmatory range for the TEQ
  determination of PCDD/F in feedstuff by CALUX
  bioassay
• CC MRL-2.33 x SDRW
• CC 0.75- 2.33 x 0.073 0.58 ng TEQ/kg
• Conclusion
• Sample lower than 0.58 ng TEQ/kg are negative
  with 99 certainty (false negative rate lt 1)
• Sample above 0.58 ng TEQ/kg must be confirmed by
  GC-HRMS