Methodological Issues in Systematic Reviews: Formulating Questions

1
Methodological Issues in Systematic Reviews
Formulating Questions

• Thomas A Trikalinos, MD, PhD
• Co-Director, Tufts Evidence-based Practice Center

Author has no actual or potential conflicts of
interest in relation to this activity
2
Asking the right question
3
We will cover

• PICO(TS) approach to generate answerable research
  questions
• Analytic framework to identify potential
  questions
• Other considerations in formulating questions

4
Systematic Review/Meta-Analysis
5
Systematic review protocol

• Well focused study question PICO(TS)
• Identification of studies (studies design,
  source, search strategy)
• Eligibility criteria (study, patient, and disease
  characteristics, treatments, outcomes)
• Data extraction (definition of outcomes, quality
  assessment)
• Data summary and analysis (outcomes used,
  intention to treat)

6
PICO(TS) approach to formulating answerable
research questions

• Population (participants)
• Intervention
• Comparator
• Outcome
• Time (duration)
• Study design (experimental / observational, N)

7
Poorly formulated questionWhat drugs should be
used to treat patients with hypertension?
8
Example The Well Formulated Question
9
Diagnostic test research question- test
performance -
10
Diagnostic test research question- clinical
impact -
11
We will cover

• PICO(TS) approach to generate answerable research
  questions
• Analytic framework to identify potential
  questions
• Other considerations in formulating questions

12
Approach to (de)constructing research question

• Question of interest is often over-arching and
  not directly addressable
• Ex. How to manage hypercholesterolemia?
• Need to break it down into answerable subunits
• Multiple questions frequently needed
• Final answers (recommendations) typically involve
  additional information, balancing the benefits
  and harms, feasibility, costs, etc

13
Simplified Evidence Model Linking Biological
Effects (intermediate) and Health Outcomes
14
USPSTF generic analytic framework for screening
topics
15
Analytic framework for omega-3 FA intake and CVD
16
We will cover

• PICO(TS) approach to generate answerable research
  questions
• Analytic framework to identify potential
  questions
• Other considerations in formulating questions

17
Problems defining interventions and comparators
Pulmonary rehabilitation vs conventional care
AHRQ Evidence Report/Technology
Assessment Pulmonary rehabilitation for COPD,
2007 http//www.ahrq.gov
18
OK, but there are things that are self-evident,
no?
19
Would You Say You Had Sex If ?
JAMA 1999281275-7
20
Percentage Indicating Had Sex
JAMA 1999281275-7
21
Problems measuring genetic variables (TA)n in ER
alpha
Ann Intern Med 2006145255-64
22
Standardizing across labs
Ann Intern Med 2006145255-64
23
Problems in coming up with questions

• Sometimes you dont know what should be the
  question because there is not yet agreement among
  investigators about the meaning
• Pre-emptive analgesia different investigators
  have different ideas about what it is
• Solution Rather than pick an arbitrary
  definition, try to group what has been studied
  into several similar categories and address them,
  this will provide evidence of the spectrum of
  mechanisms
• Question What is the effect of timing of
  analgesics on post-op requirements of pain
  medication and patient outcomes?

24
Problems with diagnosis

• varying definitions (lack of an adequate
  reference standard)
• different levels of rigor
• tradeoffs between rigor and real world practice
  (sinus puncture to diagnose sinusitis vs. actual
  clinical practice efficacy vs. effectiveness)

25
Sinus Trouble
26
Possible definitions of acute bacterial sinusitis
and their effect
27
Many diseases have diverse manifestations
28
Some Components of the Clinical and Paraclinical
Spectrum of Poliomyelitis
Jenicek M. Epidemiology The logic of modern
medicine. Epimed, Montreal, 1995.
29
Hypothetical Infectious Disease Example
Jenicek M. Epidemiology The logic of modern
medicine. Epimed, Montreal, 1995.
30
Management of Clinically Inapparent Adrenal
Mass(Adrenal Incidentaloma)The Evidence Report

• NIH State-of-the-Science ConferenceFebruary 4,
  2002

31
Incidentaloma is NOT a disease

• The prevalence of this phenomenon and the
  pathologies and their likelihood will depend on
• Patients with known cancers or not
• Presenting signs and symptoms
• Patients age and sex
• Biochemical activity findings
• Initial diagnostic test used (CT, US, MRI)
• Size of the mass (lower limit)

32
What are the pathologies reported among adrenal
incidentaloma?

• Adenoma
• Adrenocortical carcinoma
• Angiomyelolipoma
• Cyst
• Epithelial carcinoma
• Hyperplasia
• Hematoma
• Lymphoma

• Metastases
• Myelolipoma
• Neurinoma
• Pheochromocytoma
• Regenerative hepatic nodule
• Renal angiomyolipoma
• Retroperitoneal fibrosis
• . . . more

33
Some possible definitions of incidentaloma
34
Issues in formulating a question

• Narrow versus broad (for individuals/ subgroups
  or entire population)
• Clinically meaningful and useful (based on sound
  biological and epidemiological principles)
• Very broadly defined questions may be criticized
  for mixing apples and oranges
• Very narrowly focused questions have limited
  generalizability and sometimes may lead to biased
  interpretations / conclusions

35
Benefits of getting the questions right at the
start

• Improves clarity of the problem
• Facilitates subsequent steps of the systematic
  review
• Helps select right kind of studies for review
• Anticipates issues, resource needs
• Improves efficiency, reduces needs to revisit
  questions / re-review papers, minimizes wasted
  efforts

36
Summary

• PICO(TS) method is a very useful technique to
  formulate answerable research questions
• Analytic framework can be useful to visualize and
  think about related issues
• Defining criteria
• Biologically meaningful
• Question and criteria guide the entire systematic
  review process