Reliability of Screening Tests

1
Reliability of Screening Tests
RELIABILITY The extent to which the screening
test will produce the same or very similar
results each time it is administered. --- A test
must be reliable before it can be valid. ---
However, an invalid test can demonstrate
high reliability.
2
Reliability of Screening Tests

• Sources of variability that can affect the
  reproducibility of results of a screening test
• 1. Biological variation (e.g. blood pressure)
• 2. Reliability of the instrument itself
• 3. Intra-observer variability (differences in
  repeated measurement by the same screener)
• 4. Inter-observer variability (inconsistency in
  the way different screeners apply or
  interpret test results)

3
Use of Multiple Screening Tests
Sequential (two-stage) testing A less
expensive, less invasive, or less uncomfortable
test is performed first those who screen
positive are referred for further testing using a
test which may have greater sensitivity and
specificity reduces false positives, hence an
increase in net specificity.
4
Use of Multiple Screening Tests
Simultaneous testing Multiple tests are used
simultaneously Person tests positive if there
is a positive result on any of the tests
employed reduces false negatives, hence an
increase in sensitivity but at the expense of
decreased specificity.
5
Measuring the performance (yield) of a screening
test
6
Performance Yield

• People with positive screening test results
  will also test positive on the diagnostic test
• Predictive Value Positive (PV)
• People with negative screening test results
  are actually free of disease
• Predictive Value Negative (PV-)

7
Performance Yield
True Disease Status
-

Results of Screening Test
a
b

-
c
d
Predictive value positive (PV) The probability
that a person actually has the disease given that
he or she tests positive. PV a / (a
b)
8
Performance Yield
True Disease Status
-

Results of Screening Test
a
b

-
c
d
Predictive value negative (PV-) The probability
that a person is truly disease free given that he
or she tests negative. PV- d / (c d)
9
Performance Yield
True Disease Status
-

Results of Screening Test
400
995

-
98905
100
Sensitivity a / (a c) 400 / (400 100)
80 Specificity d / (b d) 98905 / (995
98905) 99 PV a / (a b) 400 / (400
995) 29 PV- d / (c d) 98905 / (100
98905) 99
10
Performance Yield
True Disease Status
-

Results of Screening Test
400
995

-
98905
100
PV a / (a b) 400 / (400 995)
29 Among persons who screen positive, 29 are
found to have the disease.
11
Performance Yield
True Disease Status
-

Results of Screening Test
400
995

-
98905
100
PV- d / (c d) 98905 / (100 98905)
99.9 Among persons who screen negative, 99.9
are found to be disease free.
12
Performance Yield
Factors that influence PV and PV- 1. The more
specific the test, the higher the PV 2. The
higher the prevalence of preclinical disease in
the screened population, the higher the
PV 3. The more sensitive the test, the
higher the PV-
13
Performance Yield
Prevalence () Sensitivity Specificity
PV 0.1 90 95
1.8 1.0 90 95 15.4
5.0 90 95 48.6 50.0 90
95 94.7
14
Performance Yield
Thus, the PV is maximized when used in high
risk populations since the prevalence of
pre-clinical disease is higher than in the
general population. screening a total
population for a relatively infrequent disease
can be very wasteful of resources and may yield
few previously undetected cases.
15
Effectiveness of Screening
Evaluating if the screening program
reduced morbidity and mortality from the
disease 1. Overall shift in severity of disease
at the time of diagnosis. 2. Compare
cause-specific mortality from among those whose
disease was picked up by screening versus those
with a diagnosis related to symptoms.
16
Effectiveness of Screening
Evaluating if the screening program
reduced morbidity and mortality from the
disease 3. Reduction in disease-related
complications. 4. Improvement of quality of
life in screened individuals.
17
Effectiveness of Screening
In reality, establishing the sensitivity and
specificity of screening tests may be
difficult often times, data are only available
on persons who screen positive and are referred
for further testing.
a b
c d
Data are available for cells a and b
only. Permits calculation of PV only
18
Effectiveness of Screening
Sources of bias in evaluating screening
programs 1. Self-selection bias (volunteer
bias) 2. Lead time bias 3. Length bias 4.
Over-diagnosis bias
19
Effectiveness of Screening
1. Self-selection bias (volunteer
bias) --- Volunteers for screening programs may
be healthier, on average, than persons who do
not participate in screening programs. On the
other hand. --- The worried well may be more
likely to participate and may be at overall
higher risk due to family history or lifestyle
characteristics.
20
Effectiveness of Screening
2. Lead time bias Lead time The interval
between diagnosis of disease at screening and
when it would have been detected from clinical
symptoms. --- Survival may appear to be
increased among screen-detected cases simply
because diagnosis was made earlier in the
course of the disease.
21
Effectiveness of Screening
3. Length bias (prognostic selection) ---- The
overrepresentation among screen- detected cases
of those with a long pre- clinical phase, and
thus a more favorable prognosis. --- Those with
a long pre-clinical phase are more readily
detectable by screening than more rapidly
progressing cases with a short pre-clinical
phase.
22
Effectiveness of Screening
4. Over-diagnosis bias ---- Persons who screen
positive and are truly disease free (false
positives), yet are erroneously
diagnosed as having the disease. --- Since
these persons are truly disease free, we expect
a more favorable long- term outcome giving the
appearance of an effective screening program.
23
Effectiveness of Screening
Possible interpretations for null results in a
screening program evaluation include ---- The
disease has an extremely short
detectable pre-clinical phase. --- Current
therapeutic intervention is no more effective
when provided earlier than at usual time of
diagnosis. --- Inadequacies of care provided to
those who screen positive.
24
Effectiveness of Screening
Study Designs Used to Evaluate Screening
Programs 1. Ecological Studies 2. Observational
Analytic Studies 3. Randomized Trials
(infrequent and difficult to carry out)