Assessing Applicability of Medical Test Studies in Systematic Reviews

1
Assessing Applicability of Medical Test Studies
in Systematic Reviews

• Prepared for
• The Agency for Healthcare Research and Quality
  (AHRQ)
• Training Modules for Medical Test Reviews Methods
  Guide
• www.ahrq.gov

2
Overview of a Medical Test Review
Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
3
Learning Objectives

• Explain how applicability extends to both the
  individual study and bodies of evidence
• Describe three main challenges related to
  assessing applicability and how they are often
  related to context
• Understand the four principles for assessing and
  reporting applicability
• Apply the PICOTS (population, intervention,
  comparator, outcomes, timing, and setting)
  typology to assess applicability

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
4
Background

• Decisionmakers need to understand whether a
  medical test is meaningful in a specific context
  in order to make an informed decision. For
  example
• Whether the test has been studied in the
  patient/disease/care settings of interest
• Whether the test has been part of a management
  strategy of interest
• Evidence-based Practice Centers use four
  principles to assess and report applicability
• Determine the most important factors that affect
  applicability
• Systematically abstract key characteristics that
  may affect applicability
• Make judgments about major limitations to the
  applicability of individual studies
• Consider the applicability of the body of evidence

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
5
Challenges Specific to Assessing Applicability(1
of 4)

1. Key questions lack clarity.
2. Studies are not specific to the key questions.
3. Tests are rapidly evolving.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
6
Challenges Specific to Assessing Applicability(2
of 4)

• Key questions lack clarity
• How the test is to be used should be stated
  clearly in the key questions to avoid confusion
  among decisionmakers about the management
  strategy being evaluated.
• Example Studies comparing the accuracy of one
  test versus another (i.e., as a replacement), but
  the new test is actually meant to be a
  triage/add-on test
• Hinders the review process (slows
  inclusion/exclusion decisions)
• Leads to inappropriate lumping and splitting
  of groups
• Lumping subgroups that are fundamentally
  different can cause a false finding of a
  difference (type 1 error).
• Splitting subjects into subgroups without
  evidence of different effects may reduce power to
  detect a difference (type II error).

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
7
Challenges Specific to Assessing Applicability(3
of 4)

• Studies are not specific to the key questions
• Studies may not reflect the subgroups or
  comparisons identified in the key questions.
• Reviewers need to decide whether the deviation is
  minor or crucial.
• Crucial deviations are those deemed likely to
  meaningfully affect
• Test performance
• Clinical decisionmaking
• Health outcomes

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
8
Challenges Specific to Assessing Applicability(4
of 4)

• Tests are rapidly evolving and change often.
• The literature often references tests that are
  not yet widely available or no longer common.
• Secular trends in use patterns and market forces
  may shape applicability in unanticipated ways.
• Example An old test that gives dichotomous,
  qualitative results (present vs. absent) is
  replaced by a test that reports a continuous,
  quantitative measure.
• Review the old test as a point of reference for
  determining the merit of a new test?
• Only use the more limited, newer data from the
  new test?

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
9
Principles of Assessing Applicability

1. Identify important contextual factors
2. Be prepared to deal with additional factors
   affecting applicability
3. Justify decisions to restrict the scope of a
   review
4. Maintain a transparent process

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
10
Principle 1 Identify Important Contextual
Factors (1 of 10)

• It is not practical to consider all possible
  factors related to the impact of test on health
  outcomes
• Consider factors that affect steps in a causal
  chain of direct relevance to the key question.
• Example Slice thickness is a relevant factor for
  assessing the applicability/accuracy of cardiac
  magnetic resonance imaging for detecting
  atherosclerosis.
• Also consider factors that affect later links in
  the causal chain.
• Example What factors impact effectiveness of
  treatment for lesions identified by cardiac MRI
  versus angiography?
• Consider contextual issues relevant to medical
  tests, for example, patient populations,
  management strategies, time effects, and secular
  trends in disease prevalence and severity.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
11
Principle 1 Identify Important Contextual
Factors (2 of 10)

• Spectrum effect
• The severity or type of disease may affect test
  accuracy.
• Example Cardiac magnetic resonance imaging is
  generally accurate in identifying cardiac
  anatomy/functionality, but arrhythmias, lesion
  location, and obesity can affect performance of
  the test.
• Reviewers must identify these disease-related
  factors ahead of time and justify when to split
  key questions or to conduct subgroup analyses.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
12
Principle 1 Identify Important Contextual
Factors (3 of 10)

• The medical test as part of a management strategy
• It is important to clarify conditions in which a
  test is studied and used.
• Example The test performance of cardiac magnetic
  resonance imaging (MRI) differs when used as an
  add-on test after initial screening versus when
  used for triage or for patients without symptoms.
• When cardiac MRI is used as add-on, the
  comparator for diagnostic accuracy is x-ray
  coronary angiography (the gold standard).
• When cardiac MRI is used for triage, the
  comparator is stress electrocardiography or other
  initial screening test.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
13
Principle 1 Identify Important Contextual
Factors (4 of 10)

• Methods of the medical test over time
• Diagnostics evolve rapidly.
• Awareness of these changes should guide selection
  of the date range and eligible test types for
  included literature,
• Example 1 The slice thickness of magnetic
  resonance imaging scans has decreased steadily
  over time, allowing resolution of smaller
  lesions.
• Excluding studies with older technologies and
  presenting results of included studies by slice
  thickness may be appropriate.
• Example 2 Antenatal tests are applied earlier
  and earlier in gestation.
• Studies of test performance need to be examined
  by varied cutoffs for stage of gestation.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
14
Principle 1 Identify Important Contextual
Factors (5 of 10)

• Secular trends in population risk and disease
  prevalence
• Direct/indirect changes in secular trends (or
  cultural differences) can influence test
  performance and applicability.
• Example Obesity and the average age of pregnancy
  have risen dramatically in the past 30 years,
  both of which are linked to type 2 diabetes.
• The underlying prevalence of undiagnosed type 2
  diabetes in pregnancy is increased.
• As a result, the predictive values, the
  cost/benefit ratios, and the sensitivity and
  specificity of screening tests for gestational
  diabetes change over time.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
15
Principle 1 Identify Important Contextual
Factors (6 of 10)

• Secular trends in population risk and disease
  prevalence
• Secular trends in population characteristics can
  influence the ability of clinicians to conduct a
  medical test.
• Example Obesity diminishes image quality in
  tests, such as ultrasound for gallbladder disease
  or magnetic resonance imaging for spinal
  conditions.
• Studies of these tests often restrict enrollment
  to people with normal body weight.
• These tests may not perform the same in practice
  because of the rising trend of obesity in the
  actual patient population.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
16
Principle 1 Identify Important Contextual
Factors (7 of 10)

• Secular trends in population risk and disease
  prevalence
• Prevalence is tied to disease definitions that
  can change. Some examples are
• Criteria to diagnose AIDS
• Cystometrically defined detrusor
  instability/overactivity versus the symptom
  complex overactive bladder
• Continuous refinement of classifications of
  mental health conditions in the Diagnostic and
  Statistical Manual of Mental Disorders
• The literature may not capture changes in
  diagnostic criteria.
• Having expert knowledge with a historical vantage
  point is helpful.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
17
Principle 1 Identify Important Contextual
Factors (8 of 10)

• Routine preventive care over time
• Routine use of a medical test as a screening test
  may indirectly alter population prevalence.
• Preventive lipid testing in individuals with
  cardiovascular disease has changed the proportion
  able to be diagnosed and have the course of their
  disease altered.
• The vaccine for the human papilloma virus (HPV)
  is postulated to change the distribution of viral
  subtypes in the population.
• Preventive practices change the utility of
  medical tests.
• Example The utility of the test to detect HPV
  infection is decreasing as increasing numbers of
  people are being vaccinated for the virus.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
18
Principle 1 Identify Important Contextual
Factors (9 of 10)

• Treatment trends
• Literature about the impact of diagnostic tools
  on outcomes needs additional interpretation in
  light of new therapeutics.
• Example The implications of testing for carotid
  arterial stenosis are changing as hypertension
  treatment and the use of lipid-lowering drugs
  have improved.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
19
Principle 1 Identify Important Contextual
Factors (10 of 10)

• Two steps to ensure that population/subgroup data
  are uniformly collected and useful
• Use the PICOTS (population, intervention,
  comparator, outcomes, timing, and setting)
  typology to identify the range of possible
  factors that might affect applicability and
  consider the six contextual factors.
• Review the list of factors affecting
  applicability with stakeholders to
• Ensure common vantage points
• Identify hidden factors that may influence
  applicability
• Capture crucial applicability features in the
  evidence review

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
20
Using the PICOTS Framework To Assess and Describe
Applicability of Medical Tests (1 of 7)
PICOTS Element Potential Core Characteristics To Describe and Assess Challenges When Assessing Studies Example Potential Systematic Approaches for Decisions About Including Studies
Population Justification for lumping or splitting key questions Method of identification/ selection Inclusion and exclusion criteria of review Demographic characteristics of those included in the review Prevalence of condition in practice and in studies Spectrum of disease in practice and in studies Source of population not described Study population poorly specified Key characteristics not reported Unclear whether test performance varies by population Education/ literacy level not reported in study of pencil-and-paper functional status assessment Exclude a priori if key element crucial to assessing intended use case is missing Or include, but Flag missing elements in tables/text Organize data within key questions by presence/absence of key elements Include presence/ absence as parameter in meta-regression or sensitivity analyses Note need for challenge to be addressed in future research
Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
21
Using the PICOTS Framework To Assess and Describe
Applicability of Medical Tests (2 of 7)
PICOTS Element Potential Core Characteristics To Describe and Assess Challenges When Assessing Studies Example Potential Systematic Approaches for Decisions About Including Studies
Intervention Version of test used in practice and in studies How and by whom tests are conducted in practice and in studies Cutoff/diagnostic thresholds applied in practice and in studies Skill of assessors when interpretation of test required in studies Version/ instrumentation not specified Training/quality control not described Screening and diagnostic uses mixed Ultrasound machines and training of sonographers not described in a study of fetal nuchal translucency assessment to detect aneuploidy Exclude a priori if version critical and not assessed Or include, but Contact authors for clarification Flag version of test or deficits in reporting in tables/text Discuss implications Model cutoffs and conduct sensitivity analyses
Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
22
Using the PICOTS Framework To Assess and Describe
Applicability of Medical Tests (3 of 7)
PICOTS Element Potential Core Characteristics To Describe and Assess Challenges When Assessing Studies Example Potential Systematic Approaches for Decisions About Including Studies
Comparator Use of gold standard versus alloy standard in studies How the test is used as part of a management strategy (e.g., triage, replacement, or add-on) in practice and in studies Gold standard not applied Correlational data only Cardiac computerized tomography compared with a treadmill stress test without use of angiography as a gold standard Exclude a priori if there is no gold standard Or include, but Restrict to specified comparators Group by comparator in tables/text
Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
23
Using the PICOTS Framework To Assess and Describe
Applicability of Medical Tests (4 of 7)
PICOTS Element Potential Core Characteristics To Describe and Assess Challenges When Assessing Studies Example Potential Systematic Approaches for Decisions About Including Studies
Outcome (of using the test) How accuracy outcomes selected for review relate to use in practice Accuracy of disease status classification Sensitivity/ specificity Predictive values Likelihood ratios Diagnostic odds ratio Area under curve Discriminant capacity Failure to test normal patients, or a subset thereof, with the gold standard Precision of estimates not provided Tests may be used as part of management strategy in which exact diagnosis is less important than ruling out a disease P-value provided for mean of continuous test results by disease status, but confidence bounds not provided for performance characteristics Exclude a priori if test results cannot be mapped to disease status (i.e., 22 or other test performance data cannot be extracted) Exclude if a subset of normal patients not tested Or include, but Flag deficits in tables/text Discuss implications Assess heterogeneity in meta-analysis and comment of sources of heterogeneity in estimates
Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
24
Using the PICOTS Framework To Assess and Describe
Applicability of Medical Tests (5 of 7)
PICOTS Element Potential Core Characteristics To Describe and Assess Challenges When Assessing Studies Example Potential Systematic Approaches for Decisions About Including Studies
Clinical Outcomes (from test results) How studies addressed clinical outcomes selected for the review Earlier diagnosis Earlier intervention Change in treatment given Change in sequence of other testing Change in sequence/intensity of care Improved outcomes, quality of life, costs, etc. Populations and study designs of included studies heterogeneous with varied findings Data not stratified or adjusted for key predictors Bone density testing reported in relation to fracture risk reduction without consideration of prior fracture or adjustment for age Exclude if no disease outcomes and outcomes key to understanding intended use case Or include, and Document details of deficits in tables/text Discuss implications Note need for challenge to be addressed in future research
Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
25
Using the PICOTS Framework To Assess and Describe
Applicability of Medical Tests (6 of 7)
PICOTS Element Potential Core Characteristics To Describe and Assess Challenges When Assessing Studies Example Potential Systematic Approaches for Decisions About Including Studies
Timing Timing of availability of results to care team in studies and how this might relate to practice Placement of test in the sequence of care (e.g., relationship of test to treatment or follow-on management strategies) of studies and how this might relate to practice Timing of assessment of disease status and outcomes in studies Sequence of use of other diagnostics unclear Time from results to treatment not reported Order of testing varies across subjects and was not randomly assigned D-dimer studies in which it is unclear when results were available relative to imaging studies of deep vein thrombosis Exclude if timing/sequence is key to understanding intended use case Or include, and Contact authors for information Flag deficits in tables/text Discuss implications Note need for challenge to be addressed in future research
Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
26
Using the PICOTS Framework To Assess and Describe
Applicability of Medical Tests (7 of 7)
PICOTS Element Potential Core Characteristics To Describe and Assess Challenges When Assessing Studies Example Potential Systematic Approaches for Decisions About Including Studies
Setting How setting of test in studies relates to key questions and current practice Primary care versus specialty care Hospital-based Routine processing versus specialized lab or facility Specialized personnel Screening versus diagnostic use Resources available to providers for diagnosis and treatment of condition vary widely Provider type/ specialty vary across settings Comparability of care in international settings unclear Diagnostic evaluation provided by geriatricians in some studies and unspecified primary care providers in others Exclude if care setting known to influence test/outcomes or if setting is key to understanding intended use case Or include, but Document details of setting Discuss implications
Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
27
Principle 2 Be Prepared To Deal With Additional
Factors Affecting Applicability

• Some contextual factors are only found after a
  large volume of literature has been reviewed.
• Example A meta-analysis may find that a test is
  inaccurate in older patients, but age was not
  considered in the key questions.
• May be a false association
• May be an oversight
• When in doubt between a false association or
  oversight, it is appropriate to comment on an
  apparent association and clearly state that it is
  only a hypothesis.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
28
Principle 3 Justify Decisions To Restrict the
Scope of a Review

• May be necessary to restrict the scope of the
  review to
• Specific versions of the test
• Selected study methods or types
• Specific populations (e.g., women, patients who
  are obese)
• Specific settings (e.g., primary care practice)
• Restriction is appropriate when
• All agree that a top priority is applicability to
  a particular target group or setting
• Test performance is different in a specific
  subgroup, setting, or from another version of the
  test
• Restriction is not appropriate when a fully
  comprehensive review is desired

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
29
Principle 4 Maintain a Transparent Process

• Address applicability when defining review
  methods and documenting decisions in a protocol.
• Example Factors that change with time should
  prompt adding timeframes to inclusion criteria or
  description/ analysis of possible impacts on
  applicability.
• Transparency is essential when a review decision
  may be controversial.
• Example Exclusion criteria leave a reviewer
  empty handed the reviewer may decide some
  excluded evidence is relevant by extension or
  analogy.
• In this event, clearly state how evidence may not
  be directly applicable to key questions but
  represents the limited state of the science.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
30
Illustrative Example Cervical Cancer Screening
(1 of 3)

• 2002 Cervical Cancer Screening Summary of the
  Evidence for the U.S. Preventive Services Task
  Force
• Challenges
• Changes in secular trends and availability/format
  of tests
• Liquid-based cervical cytology (new technique)
  rapidly being used in practice
• Resource shortages for reviewing conventional Pap
  smears
• New computer systems to screen cytology slides to
  enhance accuracy
• Literature starting to include adjunct usage of
  human papilloma virus vaccine
• No randomized controlled trials available

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm. Hartmann KE, Hall
SA, Nanda K, et al. Systematic Evidence Review
No. 25. Available at www.ncbi.nlm.nih.gov/books/NB
K42831.
31
Illustrative Example Cervical Cancer Screening
(2 of 3)

• Since no data compared new screening
  tools/strategies to outcomes, the 2002 Cervical
  Cancer Screening Summary of the Evidence for the
  U.S. Preventive Services Task Force report
  focused on medical test characteristics
  (sensitivity, specificity, predictive value, and
  likelihood ratios).
• Three computer technologies
• Two liquid cytology approaches
• All methods of human papilloma virus testing
• Included all technologies
• Restricting technologies only to those available
  in the United States (most applicable setting)
  would have reduced the scope substantially.
• The purpose was to determine if comparable or
  superior methods were being overlooked or no
  longer offered.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm. Hartmann KE, Hall
SA, Nanda K, et al. Systematic Evidence Review
No. 25. Available at www.ncbi.nlm.nih.gov/books/NB
K42831.
32
Illustrative Example Cervical Cancer Screening
(3 of 3)

• There are two specific areas of applicability
  interest age and hysterectomy status.
• The underlying incidence and prevalence data
  included
• Provided context
• Informed modeling of testing impact
• Reviewers could not estimate the usefulness of
  human papilloma virus testing among older women
  from trends in younger women.
• Extrapolating results from one context to another
  rarely enhances applicability.
• The review found that older women are
  disproportionately more likely to have
  false-positive findings under current screening
  recommendations and supported the discontinuation
  of Pap testing after hysterectomy.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm. Hartmann KE, Hall
SA, Nanda K, et al. Systematic Evidence Review
No. 25. Available at www.ncbi.nlm.nih.gov/books/NB
K42831.
33
Summary (1 of 3)

• To adequately assess applicability of a medical
  test, reviewers need to familiarize themselves
  with
• Availability of the test
• Technology of the test
• Contemporary clinical use of the test
• Current treatment modalities for the related
  disease condition
• Potential interplay of disease severity and the
  performance characteristics of the test
• Implications of particular study designs and
  sampling strategies for bias in the findings
  about applicability

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
34
Summary (2 of 3)

• Restricting inclusion of only marginally related
  or outdated studies can aid applicability.
• Extrapolating results from one context to another
  does not help.
• Guidance from clinical, medical testing, and
  statistical experts is helpful when considering
  restriction of scope or exclusion of publications
  with specific types of flaws.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
35
Summary (3 of 3)

• The review team must group findings in a
  meaningful way, whether the review is exhaustive
  or narrow.
• For medical tests, this means gathering and
  synthesizing data in ways that help to readily
  understand applicability.
• Grouping summaries of findings using familiar
  structures like the PICOTS (population,
  intervention, comparator, outcomes, timing, and
  setting) enhances clarity in framing
  applicability.
• Studies may be presented more than once within a
  review, in different applicability axes.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
36
Key Messages

• Identify contextual factors that may affect test
  performance early in the review process.
• Carefully consider and document justification for
  addressing these factors by restricting key
  questions or creating a broader review.
• Protocols should document which factors will be
  excluded and how subgroups will be assessed.
• Reviewers should document how they plan to handle
  studies that only partly fit the key questions or
  inclusion/exclusion criteria.
• Final reviews should include a description of
  test use in usual practice and how the included
  studies fit with this use.

Hartman KE, Matchar DB, Chang SM. Assessing
applicability of medical test studies in
systematic reviews. In Methods guide for medical
test reviews. Available at www.effectivehealthcare
.ahrq.gov/medtestsguide.cfm.
37
Practice Question 1 (1 of 2)

• When conducting most systematic reviews, it is
  important to limit the contextual factors
  considered in applicability assessment rather
  than considering all possible factors.
• True
• False

38
Practice Question 1 (2 of 2)

• Explanation for Question 1
• The statement is true. It is not practical to
  consider all possible factors that may impact
  applicability in all cases. This approach would
  be overwhelming and, in some cases, leave the
  reviewer with no applicable studies. A reasonable
  list of factors should be considered and agreed
  upon by the stakeholders in the systematic review
  before initiating the literature review.

39
Practice Question 2 (1 of 2)

• The routine use of the human papilloma virus
  vaccine to prevent cervical cancer may change the
  utility of the test to detect the virus.
• True
• False

40
Practice Question 2 (2 of 2)

• Explanation for Question 2
• The statement is true. This is an example of how
  the changes in routine preventive care over time
  can affect the prevalence of disease. Knowledge
  of these trends is important when looking at the
  applicability of a body of literature.

41
Practice Question 3 (1 of 2)

• One of the general principles in assessing the
  applicability is to keep the process as
  confidential as possible without involving the
  stakeholders and future decisionmakers.
• True
• False

42
Practice Question 3 (2 of 2)

• Explanation for Question 3
• The statement is false. Transparency is essential
  while addressing applicability. Review methods
  should be documented clearly in a protocol.

43
Practice Question 4 (1 of 2)

• The PICOTS (population, intervention, comparator,
  outcomes, timing, and setting) framework is a
  tool that can help to systematically abstract and
  report key characteristics that may affect
  applicability.
• True
• False

44
Practice Question 4 (2 of 2)

• Explanation for Question 4
• The statement is true. The PICOTS (population,
  intervention, comparator, outcomes, timing, and
  setting) framework is a tool that can be used in
  a systematic review to meticulously think through
  all of the factors that may affect applicability.

45
Authors

• This presentation was prepared by Brooke
  Heidenfelder, Rachael Posey, Lorraine Sease, Remy
  Coeytaux, Gillian Sanders, and Alex Vaz, members
  of the Duke University Evidence-based Practice
  Center.
• The module is based on chapter 6, Assessing
  applicability of medical test studies in
  systematic reviews. In Chang SM and Matchar DB,
  eds. Methods guide for medical test reviews.
  Rockville, MD Agency for Healthcare Research and
  Quality June 2012. p. 6.1-13. AHRQ Publication
  No. 12-EHC017. www.effectivehealthcare.ahrq.gov/me
  dtestsguide.cfm

46
References (1 of 2)

• Agency for Healthcare Research and Quality.
  Methods Guide for Effectiveness and Comparative
  Effectiveness Review. Rockville, MD Agency for
  Healthcare Research and Quality April 2012. AHRQ
  Publication No. 10(12)-EHC063-EF. Chapters
  available at www.effectivehealthcare.ahrq.gov/meth
  odsguide.cfm.
• Hartmann KE, Hall SA, Nanda K, et al. Screening
  for Cervical Cancer. Systematic Evidence Review
  No. 25 (Prepared by the Research Triangle
  Institute/University of North Carolina under
  Contract No. 290-97-0011, Task Order No. 3).
  Rockville, MD Agency for Health Care Research
  and Quality January 2002. www.ncbi.nlm.nih.gov/bo
  oks/NBK42831.
• Hartman KE, Matchar DB, Chang SM. Assessing
  applicability of medical test studies in
  systematic reviews. In Chang SM and Matchar DB,
  eds. Methods guide for medical test reviews.
  Rockville, MD Agency for Healthcare Research and
  Quality June 2012. p. 6.1-13. AHRQ Publication
  No. 12-EHC017. www.effectivehealthcare.ahrq.gov/me
  dtestsguide.cfm.

47
References (2 of 2)

• Hartmann KE, Nanda K, Hall S, et al. Technologic
  advances for evaluation of cervical cytology is
  newer better? Obstet Gynecol Surv. 2001
  Dec56(12)765-74. PMID 11753179.
• Hutchinson ML, Zahniser DJ, Sherman ME, et al.
  Utility of liquid-based cytology for cervical
  carcinoma screening results of a
  population-based study conducted in a region of
  Costa Rica with a high incidence of cervical
  carcinoma. Cancer. 1999 Apr87(2)48-55. PMID
  10227593.
• McCrory DC, Matchar DB, Bastian L, et al.
  Evaluation of Cervical Cytology. Evidence
  Report/Technology Assessment No. 5 (Prepared by
  the Duke University Evidence-based Practice
  Center under Contract No. 290-07-0014. Rockville,
  MD Agency for Health Care Policy and Research
  January 1999. AHCPR Publication No. 99-E010.
  www.ncbi.nlm.nih.govbooks/NBK11840.