Title: Metaanalysis and systematic reviewsin qualitative research: Mission impossible
1Meta-analysis and systematic reviews in
qualitative research Mission impossible?
Ellen MacEachen (Institute for Work Health,
University of Toronto) Scott Reeves (Li Ka Shing
Knowledge Institute, University of Toronto)
- Qualitative Inquiry Group Seminar
- University of Toronto, March 27, 2009
2Overview 4 parts
- 1. Where did systematic reviews come from why
do we do them? - 2. Brief overview of the systematic review
process - 3. Cases that draw out systematic review issues
- Case I (Scott)
- Case II (Ellen)
- 4. Reflections on the conduct and usefulness of
systematic reviews of qualitative studies
31. What are systematic reviews?
- Positivistic roots / standardization
- Synthesis of primary studies
- Multiple check points / dual reviewers /
transparency - Assess quality of evidence (only include
rigorous studies) - Inference through statistical analysis
- Newer Qual types (meta-ethnography )
4Why do SRs?
- Useful
- Knowledge translation (policymakers/
practitioners) - Initial entry into field (academics)
- Help define field (academics)
- Outline areas of future research (acad/policy)
- Grant applications (academics)
52. Brief overview of the systematic review
process
- A walk through the recipe followed for
systematic reviews
6(No Transcript)
7What databases will be searched?
What keywords are used in the search?
What is the scope of the search? Languages? Dates?
What stakeholders are involved and at what stages?
Medline Total 1620
EMBASE Total 1683
CINAHL Total 443
PsycINFO Total 1174
Sociological Abstracts Total 279
ASSIA Total 74
ABI Inform Total 381
EconLit Total 58
Business Source Premier Total 585
Content Experts Refs Total 840
How will we find papers not listed on databases?
Other Total 240
Quality control process for what is being
included/excluded
Step 1 Library Search
What articles are included/excluded?
Step 2 Study Relevance
Excluded at this level n 4256
Studies considered for QA (qualitative,
quantitative and mixed methods (n 609 )
Supplemental articles from reference list n 63
Two reviewers per paper, consensus approach
Step 3 Division of QNT/QL Studies
Excluded at this level n 20
Excluded at this level n 13
How will we assess quality?
Two reviewers per paper at this stage, consensus
approach
Step 4 Quality Appraisal
What data will we extract?
Studies passing quality appraisal n 5
Studies passing quality appraisal n 14
Step 5 Data Extraction
Step 6 Finding Synthesis
n 19
83. SR cases
- Case I Making decisions about how to
construct systematic reviews the early days - Interprofessional education (Scott)
- Case II Struggles with the paradigm of
systematic reviews later days - Work health in small businesses (Ellen)
9Interprofessional education (IPE)
- Case I Making
- decisions about how
- to construct
- systematic reviews
- the early days
10Mixed method studies Quality assessment, data
extraction, synthesis
Qualitative studies Quality assessment, data
extraction, synthesis
11- Context information
- Policymaker demands for IPE evidence (early
1990s) - SR political response
- In dark about SR process new activity
- Side line work (enthusiasm)
12- SR processes
- Inclusive approach to review team 9 members
- Conceptually inclusive (implicit/explicit IPE)
- Methodologically inclusive (qual/quan)
- Open stance seeking guidelines/standards
- Pragmatic trial error
- Abstraction pre-determined categories
13- Thinking about Quality
- quality of study (appropriate design,
sampling, recruitment, validity, reliability) - quality of information (good contextual info,
explicit rationale, clear research questions,
clear results) - (e.g. CASP, EPPI Centre, Popay et al 1998)
14- Large and small SRs
- Big was good 107 studies
- Practitioners/policymakers - larger numbers
(width) - Small was better - 21 studies
- Researchers small numbers (higher quality)
15- Some reflections
- Lots of discussion/debate create parameters
first IPE review - Proceed with caution
- One step forward, two back
- Gradual movement from inclusion to exclusion
16- Some reflections
- Different types of qualitative research difficult
to synthesize - Qual / Quan / mixed methods more problems
- Quality assessment best effort (pragmatics)
- SR team dynamics
17Case II Systematic review on work and health in
small businesses (Ellen)
- Struggles with the paradigm of systematic reviews
- 2 parts
- qualitative review
- mixed method review
18(No Transcript)
19Struggles in a Qualitative review
- How to observe the broad focus that can be taken
with qualitative research while also subjecting
it to the recipe of a SR process - The SR process is systematic, not exploratory.
- Pre-defined parameters
20Struggles in a Qualitative review
- How to synthesize the findings insights of
Qualitative studies while also preserving their
context and theory - The systematic review extracts findings to
answer a specific question. - What insights do we end up with? What do we
gain/lose?
21Struggles in a Qualitative review
- What is being counted/considered in a qualitative
systematic review? - Varying approaches to data extraction.
- pre-set-findings categories?
- grounded theory process?
22Struggles in a Qualitative review
- What counts as a qualitative study?
- Studies can self-identify as qualitative but
not use recognizable qualitative methods. - E.g. Participatory studies
- Our criteria Some qualitative data some
qualitative analysis of it
23Struggles in a Qualitative review
- How to handle interesting data from studies that
dont meet quality criteria? - Nuggets Pawson (2006).
- Descriptions of process that are a nugget of
gold for the question being asked.
24Struggles in a Mixed Method review
- How to achieve a steady process for both
Qualitative Quantitative sides of the review - Quantitative team-- focused on definitions,
outcomes - Qualitative team--no fixed categories for
outcomes. - Qual quan work happens at different
times--affects possibilities for concurrent
synergy.
25Struggles in a Mixed Method review
- Theoretical differences about conflict of
interest during the quality assessment process - Qualitative team--sensitive to social/power
relations - heated issue, agreed to disagree.
26Struggles in a Mixed Method review
- Differences about levels of evidence
- QN additive approach
- E.g. "best evidence synthesis guidelines.
- How much evidence is there that this has an
effect? - QL studiesnot about effectiveness
- Some additive approach e.g. concepts in 3
studies theme - No overall conclusions about strength of evidence
27IV. Reflections on the conduct and
usefulness of systematic reviews
- SR process developed from Quantitative paradigm
we are trying to adapt Qualitative literature to
it - Can this be meaningfully done?
28Issues
- Reviews are not entirely systematic
- Messiness
- Paired reviewers anti-bias ideal
- Politics of agreement/consensus
- BUT
- Power relations among reviewers
- Assessing papers own merits or against overall
standard? - QA approaches change over course of the review
29Issues
- The issue of the podium
- Arrogance decisions on inclusion of peer
reviewed papers - A side effect of systematic reviews
- Claims of ownership of a field from SR work
30Issues
- Debates
- Can the synthesis integrate research using
different methods/theories - Although there are multiple descriptions/
explanations of data, these all ultimately relate
to some underlying reality/truth - (Bondas Hall 2007, Mays et al 2005)
31Some gains
- Think carefully about papers
- Learn a lot about a field
- Read a lot of poor papers (learn good, bad ugly)
- Hone critical appraisal skills (teaching, journal
reviewing/editing) - Opportunities to engage with people (outside
academia)
32Thank you