Metaanalysis and systematic reviewsin qualitative research: Mission impossible - PowerPoint PPT Presentation

1 / 32
About This Presentation
Title:

Metaanalysis and systematic reviewsin qualitative research: Mission impossible

Description:

... of data, these all ultimately relate to some underlying reality/truth ... Read a lot of poor papers (learn: good, bad ugly) ... – PowerPoint PPT presentation

Number of Views:43
Avg rating:3.0/5.0
Slides: 33
Provided by: Sco789
Category:

less

Transcript and Presenter's Notes

Title: Metaanalysis and systematic reviewsin qualitative research: Mission impossible


1
Meta-analysis and systematic reviews in
qualitative research Mission impossible?
Ellen MacEachen (Institute for Work Health,
University of Toronto) Scott Reeves (Li Ka Shing
Knowledge Institute, University of Toronto)
  • Qualitative Inquiry Group Seminar
  • University of Toronto, March 27, 2009

2
Overview 4 parts
  • 1. Where did systematic reviews come from why
    do we do them?
  • 2. Brief overview of the systematic review
    process
  • 3. Cases that draw out systematic review issues
  • Case I (Scott)
  • Case II (Ellen)
  • 4. Reflections on the conduct and usefulness of
    systematic reviews of qualitative studies

3
1. What are systematic reviews?
  • Positivistic roots / standardization
  • Synthesis of primary studies
  • Multiple check points / dual reviewers /
    transparency
  • Assess quality of evidence (only include
    rigorous studies)
  • Inference through statistical analysis
  • Newer Qual types (meta-ethnography )

4
Why do SRs?
  • Useful
  • Knowledge translation (policymakers/
    practitioners)
  • Initial entry into field (academics)
  • Help define field (academics)
  • Outline areas of future research (acad/policy)
  • Grant applications (academics)

5
2. Brief overview of the systematic review
process
  • A walk through the recipe followed for
    systematic reviews

6
(No Transcript)
7
What databases will be searched?
What keywords are used in the search?
What is the scope of the search? Languages? Dates?
What stakeholders are involved and at what stages?
Medline Total 1620
EMBASE Total 1683
CINAHL Total 443
PsycINFO Total 1174
Sociological Abstracts Total 279
ASSIA Total 74
ABI Inform Total 381
EconLit Total 58
Business Source Premier Total 585
Content Experts Refs Total 840
How will we find papers not listed on databases?
Other Total 240
Quality control process for what is being
included/excluded
Step 1 Library Search
What articles are included/excluded?
Step 2 Study Relevance
Excluded at this level n 4256
Studies considered for QA (qualitative,
quantitative and mixed methods (n 609 )
Supplemental articles from reference list n 63
Two reviewers per paper, consensus approach
Step 3 Division of QNT/QL Studies
Excluded at this level n 20
Excluded at this level n 13
How will we assess quality?
Two reviewers per paper at this stage, consensus
approach
Step 4 Quality Appraisal
What data will we extract?
Studies passing quality appraisal n 5
Studies passing quality appraisal n 14
Step 5 Data Extraction
Step 6 Finding Synthesis
n 19
8
3. SR cases
  • Case I Making decisions about how to
    construct systematic reviews the early days
  • Interprofessional education (Scott)
  • Case II Struggles with the paradigm of
    systematic reviews later days
  • Work health in small businesses (Ellen)

9
Interprofessional education (IPE)
  • Case I Making
  • decisions about how
  • to construct
  • systematic reviews
  • the early days

10
Mixed method studies Quality assessment, data
extraction, synthesis
Qualitative studies Quality assessment, data
extraction, synthesis
11
  • Context information
  • Policymaker demands for IPE evidence (early
    1990s)
  • SR political response
  • In dark about SR process new activity
  • Side line work (enthusiasm)

12
  • SR processes
  • Inclusive approach to review team 9 members
  • Conceptually inclusive (implicit/explicit IPE)
  • Methodologically inclusive (qual/quan)
  • Open stance seeking guidelines/standards
  • Pragmatic trial error
  • Abstraction pre-determined categories

13
  • Thinking about Quality
  • quality of study (appropriate design,
    sampling, recruitment, validity, reliability)
  • quality of information (good contextual info,
    explicit rationale, clear research questions,
    clear results)
  • (e.g. CASP, EPPI Centre, Popay et al 1998)

14
  • Large and small SRs
  • Big was good 107 studies
  • Practitioners/policymakers - larger numbers
    (width)
  • Small was better - 21 studies
  • Researchers small numbers (higher quality)

15
  • Some reflections
  • Lots of discussion/debate create parameters
    first IPE review
  • Proceed with caution
  • One step forward, two back
  • Gradual movement from inclusion to exclusion

16
  • Some reflections
  • Different types of qualitative research difficult
    to synthesize
  • Qual / Quan / mixed methods more problems
  • Quality assessment best effort (pragmatics)
  • SR team dynamics

17
Case II Systematic review on work and health in
small businesses (Ellen)
  • Struggles with the paradigm of systematic reviews
  • 2 parts
  • qualitative review
  • mixed method review

18
(No Transcript)
19
Struggles in a Qualitative review
  • How to observe the broad focus that can be taken
    with qualitative research while also subjecting
    it to the recipe of a SR process
  • The SR process is systematic, not exploratory.
  • Pre-defined parameters

20
Struggles in a Qualitative review
  • How to synthesize the findings insights of
    Qualitative studies while also preserving their
    context and theory
  • The systematic review extracts findings to
    answer a specific question.
  • What insights do we end up with? What do we
    gain/lose?

21
Struggles in a Qualitative review
  • What is being counted/considered in a qualitative
    systematic review?
  • Varying approaches to data extraction.
  • pre-set-findings categories?
  • grounded theory process?

22
Struggles in a Qualitative review
  • What counts as a qualitative study?
  • Studies can self-identify as qualitative but
    not use recognizable qualitative methods.
  • E.g. Participatory studies
  • Our criteria Some qualitative data some
    qualitative analysis of it

23
Struggles in a Qualitative review
  • How to handle interesting data from studies that
    dont meet quality criteria?
  • Nuggets Pawson (2006).
  • Descriptions of process that are a nugget of
    gold for the question being asked.

24
Struggles in a Mixed Method review
  • How to achieve a steady process for both
    Qualitative Quantitative sides of the review
  • Quantitative team-- focused on definitions,
    outcomes
  • Qualitative team--no fixed categories for
    outcomes.
  • Qual quan work happens at different
    times--affects possibilities for concurrent
    synergy.

25
Struggles in a Mixed Method review
  • Theoretical differences about conflict of
    interest during the quality assessment process
  • Qualitative team--sensitive to social/power
    relations
  • heated issue, agreed to disagree.

26
Struggles in a Mixed Method review
  • Differences about levels of evidence
  • QN additive approach
  • E.g. "best evidence synthesis guidelines.
  • How much evidence is there that this has an
    effect?
  • QL studiesnot about effectiveness
  • Some additive approach e.g. concepts in 3
    studies theme
  • No overall conclusions about strength of evidence

27
IV. Reflections on the conduct and
usefulness of systematic reviews
  • SR process developed from Quantitative paradigm
    we are trying to adapt Qualitative literature to
    it
  • Can this be meaningfully done?

28
Issues
  • Reviews are not entirely systematic
  • Messiness
  • Paired reviewers anti-bias ideal
  • Politics of agreement/consensus
  • BUT
  • Power relations among reviewers
  • Assessing papers own merits or against overall
    standard?
  • QA approaches change over course of the review

29
Issues
  • The issue of the podium
  • Arrogance decisions on inclusion of peer
    reviewed papers
  • A side effect of systematic reviews
  • Claims of ownership of a field from SR work

30
Issues
  • Debates
  • Can the synthesis integrate research using
    different methods/theories
  • Although there are multiple descriptions/
    explanations of data, these all ultimately relate
    to some underlying reality/truth
  • (Bondas Hall 2007, Mays et al 2005)

31
Some gains
  • Think carefully about papers
  • Learn a lot about a field
  • Read a lot of poor papers (learn good, bad ugly)
  • Hone critical appraisal skills (teaching, journal
    reviewing/editing)
  • Opportunities to engage with people (outside
    academia)

32
Thank you
  • Questions? Comments?
Write a Comment
User Comments (0)
About PowerShow.com