Workshop:

1
Workshop Thematic Synthesis and Framework
Synthesis

• Parts 1-4 Data Extraction, Quality Assessment,
  Synthesising Across Studies, Completing the
  Analysis

2
Shared Topic Adherence to Antiretroviral therapy
(ART) for HIV in Zambia

• BACKGROUND Antiretroviral therapy (ART) has
  significantly improved morbidity and mortality
  of individuals infected with HIV. However,
  lack of adherence to highly active antiretroviral
  therapy (HAART) remains a key challenge to
  successful management of patients with
  HIV/AIDS. Adherence rates lower than 95 are
  associated with development of viral resistance
  to antiretroviral medications.
• Efforts to sustain adherence in Africa and
  elsewhere remain important goals to optimize
  outcomes for individuals and global HIV
  treatment. (Mills, Nachega, Buchan, Orbinski,
  Attaran, Singh et al., 2006).

3
Shared Topic Adherence to Antiretroviral therapy
(ART) for HIV in Zambia

• Different Emphases
• Barriers and Facilitators to ART (Framework
  Synthesis)
• Theory Explaining Adherence to ART (Thematic
  Synthesis)

4
Reading and Data Extraction
5
Data extraction

• What is it?
• The process by which researchers obtain the
  necessary information about study characteristics
  and findings from the included studies (CRD
  Report 4)
• An attempt to reduce a complex, messy,
  context-laden and quantification resistant
  reality to a matrix of categories and numbers
  (Orwin, 1994)

6
Data extraction form

• Location
• Setting
• Sample (n)
• Age
• Gender
• Ethnicity
• Socio-economic status
• Intervention (if any)
• Quality assessment criteria
• Results?
• Further citations
• See Handouts 1-3
• Format?????

7
What results do you extract?

• What is your question?
• Keep the question in mind as you read
• Are the data relevant to this question?
• Is the question answered by the data?

8
Data Extraction
Framework Synthesis Thematic Synthesis
Extracts data against framework. Coding framework with definitions provided to increase consistency. Data not explained by framework is parked for subsequent inductive stage. Distinction typically made between original data extracts and authors analysis. Key themes and concepts extracted and reviewed for inclusiveness. Distinction preserved between original (participant) extracts and (authors analysis) ?ndings. Findings coded in duplicate. Discrepancy between codes resolved by third person.
9
Some Further Reading

• Munn, Z., Tufanaru, C., Aromataris, E. (2014).
  JBI's systematic reviews Data extraction and
  synthesis. AJN The American Journal of
  Nursing, 114(7), 49-54.
• Orwin RG. Evaluating coding decisions. In Cooper
  H, Hedges LV (editors). The Handbook of Research
  Synthesis. New York (NY) Russell Sage
  Foundation, 1994.

10
Quality Assessment

• Andrew Booth, Reader in Evidence Based
  Information Practice, Co-Convenor Cochrane
  Collaboration Qualitative Methods Group

11
Before You Begin

• Consider how you will use judgements of quality
  (cp. 50 of published Cochrane Quantitative
  Reviews performed quality appraisal but did not
  make it clear how judgements were used!)
• To exclude or to moderate?
• Will chosen instrument militate against certain
  types of research?
• Quality of reporting or quality of study?

12
Variability in Practice - 1

• 21 papers did not describe appraisal of candidate
  studies
• 6 explicitly mentioned not conducting formal
  appraisal of studies
• 5 papers did a critical appraisal, but did not
  use a formal checklist
• 7 described modifying existing instruments
• 1 used an existing instrument without
  modification
• Dixon-Woods, Booth Sutton (2007)

13
Variability in Current Practice - 1

• 23 papers did not describe critical appraisal
• 5 papers explicitly pleaded against quality
  assessment of papers or provided valid reason for
  not conducting quality appraisal.
• Criteria used varied between detailed
  descriptions of relevant items in existing or
  modified checklists to a set of broad criteria
  evaluating, for example, rich description of
  data, credibility or relevance of the original
  study.
• Hannes and Macaitis (2012)

14
Variability in Current Practice - 2

• One team used overall judgement (Smith et al.,
  2005).
• Five opted for self-developed assessment
  instrument
• Three used previously developed checklists to
  create own.
• Two mentioned critical appraisal, but did not
  specify tool.
• Most used existing instruments/frameworks. 24
  different assessment tools identified
• Critical Appraisal Skills Programme (CASP) (n
  18)
• Mays and Pope criteria (n 6)
• Popay criteria (n 6)
• Joanna Briggs Institute (n 4).
• Hannes and Macaitis (2012)

15
Appraising research quality

• 1. Epistemological criteria Judgement of
  trustworthiness requires criteria tailored to
  different research paradigms.
• 2. Theoretical Criteria Explicit theoretical
  framework shaping the design of the study and
  informing claims for generalisability
• 3. Prima facie Technical criteria Used to
  assess quality common to all research
  traditions e.g.
• Sufficient explanation of background
• Method appropriate to question
• Succinct statement of objectives/research
  questions
• Full description of methods include approach to
  analysis
• Clear presentation of findings including
  justification for interpretation of data etc.

Noyes J (2005)
16
Two dimensional approach to appraising
qualitative research
Technical markers CASP Epistemological and theoretical markers Popay et al
Technical Quality High Description thicker Privileges Subjective experience and meanings Use of theory to build explanations
Technical Quality Low Description - thinner Imposed pre-determined framework on respondents narratives. Limited/no/inappropriate use of theory, little explanatory insight
(Noyes, 2005)
17
Available Tools - 1

• CASP 10 questions to help you make sense of
  qualitative research http//www.casp-uk.net/wp-con
  tent/uploads/2011/11/CASP_Qualitative_Appraisal_Ch
  ecklist_14oct10.pdf
• Joanna Briggs Institute - Critical Appraisal
  Checklist for Interpretive Critical Research
  http//www.jbiconnect.org/agedcare/downloads/QARI_
  crit_apprais.pdf
• National Centre for Social Research. Quality in
  Qualitative Evaluation A Framework for Assessing
  Research Evidence. London National Centre for
  Social Research/UK Cabinet Office, 2003
  http//www.civilservice.gov.uk/wp-content/uploads/
  2011/09/a_quality_framework_tcm6-38740.pdf

18
Available Tools - 2

• Dixon-Woods M, Shaw RL, Agarwal S Smith JA
  (2004) The problem of appraising qualitative
  research. Quality Safety in Health Care, 13,
  223-5.
• Hannes K, Lockwood C, Pearson A (2010). A
  comparative analysis of three online appraisal
  instruments' ability to assess validity in
  qualitative research. Qualitative Health
  Research. 20(12)1736-43.
• Popay J, Rogers A Williams G (1998) Rationale
  standards for the systematic review of
  qualitative literature in health services
  research. Qualitative Health Research, 8, 341-51.
  
• Seale C Silverman D (1997) Ensuring rigour in
  qualitative research. European Journal of Public
  Health, 7, 379-84.

19
CASP Technical/Procedural Tool
20
JBI Theoretical Tool
21
Key Issue

• How are you going to use the quality assessment?
• From quantitative assessment we know authors
  frequently say they do it but they dont
  incorporate it into results
• Is it technical proceduralism gone mad?
• Or can we use the assessments to improve our
  synthesis and subsequent interpretation?

22
Quality Assessment
Framework Synthesis Thematic Synthesis (e.g. as first stage of Meta-Ethnography)
Pragmatic so tends to include all studies . Focuses explicitly on quality of reporting. Qualitative sensitivity analysis used to test robustness of synthesis. Quality Assessment as Hurdle (often used when plenty of studies to draw upon) Studies using qualitative design and analysis method included. Studies assessed for relevance first to continue to full-text review. Studies passing quality appraisal (are retained.
23
Some Further Reading

• Carroll C, Booth A. Quality assessment of
  qualitative evidence for systematic review and
  synthesis Is it meaningful, and if so, how
  should it be performed? Research Synthesis
  Methods 2015 6(2) 149-54.
• Carroll C, Booth A, Lloyd-Jones M. Should We
  Exclude Inadequately Reported Studies From
  Qualitative Systematic Reviews? An Evaluation of
  Sensitivity Analyses in Two Case Study Reviews,
  Qualitative Health Research, 2012 22 1425-1434
• Garside, R. (2014). Should we appraise the
  quality of qualitative research reports for
  systematic reviews, and if so, how?. Innovation
  The European Journal of Social Science
  Research, 27(1), 67-79.

24
Data Synthesis
25
What is Data Synthesis?

• Process of moving from focus on single studies
  (cp. Data Extraction and Quality Assessment) to
  focus on cross-study analysis
• Requires identification of patterns across data,
  including contradictory findings and data that
  does not fit
• Iterative and requires ongoing refinement
• Acts as prelude to Analysis which seeks to
  explain patterns, contradictions and differences

26
Methods of qualitative evidence synthesis

• Thematic synthesis Critical Interpretive
  Synthesis Meta-ethnography
• Only include good qualitative studies (?)
• Constant comparison iterative interpretations
  generated from the data by reviewers
• Create a theory
• Inductive (theory-generating)
• Examples
• Thomas J, Harden A. Methods for the thematic
  synthesis of qualitative research in systematic
  reviews. BMC Medical Research Methodology 2008
  8.
• Campbell R et al. Evaluating meta-ethnography a
  synthesis of qualitative research on lay
  experiences of diabetes and diabetes care. Social
  Science Medicine 2003 65671-684.

27
Methods of qualitative evidence synthesis

• Framework synthesis
• Only include good qualitative studies (?)
• Map data from included studies onto an existing
  framework to test the framework/theory (a role
  for theory)
• Build a conceptual model or framework
• Deductive (theory-testing)
• Examples
• Oliver S et al A multidimensional conceptual
  framework for analysing public involvement in
  health services research. Health Expectations
  2008, 1172-84.
• Brunton G, Oliver S, Oliver K, Lorenc T. A
  Synthesis of Research Addressing Childrens,
  Young Peoples and Parents Views of Walking and
  Cycling for Transport London. London,
  EPPI-Centre, Social Science Research Unit,
  Institute of Education, University of London
  2006.

28
Methods of qualitative evidence synthesis

• Best-fit framework synthesis
• Identify relevant pre-existing conceptual models
  or frameworks
• Identify and extract all relevant qualitative
  studies satisfying reviews inclusion criteria
• Code data from included studies against framework
• Use secondary thematic analysis/synthesis to
  generate completely new themes to supplement the
  frameworks themes
• Create new framework and conceptual model or
  theory
• Deductive and Inductive
• Framework and Thematic synthesis
• Carroll C, Booth A, Cooper K. A worked example of
  best-fit framework synthesis A systematic
  review of views concerning the taking of
  potential chemopreventive agents, BMC Medical
  Research Methodology 2011 11 29

29
Data Synthesis
Framework Synthesis Thematic synthesis (may be taken forward as Meta-Ethnography)
Original best fit framework is expanded to include new themes. Relationship between themes is examined and the data is used to reconstitute a new model. Particular attention is directed at discrepant cases. Second-order constructs pertinent to adherence identi?ed and cross-compared and presented in results section. Key themes consolidated into line of argument (third-order analysis), presented in the synthesis / discussion section.
30
Data Analysis

• To move from a list of themes/framework to a
  conceptual model/theory
• Examine relationships
• Synergies, Antagonisms, Pre-requisites, Causal
  Paths, Disconfirming Cases etcetera
• Testing the Robustness of the synthesis

31
A word about constructs

• Conventionally we distinguish (using the
  terminology of the most common method of
  qualitative synthesis, meta-ethnography)
• 1st Order Constructs First hand accounts of the
  people being studied
• 2nd Order Constructs Interpretations by the
  Authors of the studies
• 3rd Order Constructs Interpretations of the
  interpretations by the Systematic Review Team

32
Example
1st Order
3rd Order
33
The Contribution of Meta-ethnography

• Using the meta-ethnographic approach, we were
  able to produce a model of adherence to TB
  treatment by re-interpreting meaning across many
  individual qualitative studies. We also derived
  plausible hypotheses that can be used by policy
  makers and programme managers to re-organise
  treatment and care systems to improve adherence.
  Adapting the method for use in synthesising
  qualitative health research raises a number of
  methodological challenges that require further
  exploration. (atkins et al, 2008)

34
From Kates research
35
From Constructs to Arguments
36
Booth et al, 2011
37
Some Practicalities

• Tabulation of data looking for and explaining
  differences (e.g. majority, split, exception)
• Post-Its arranging according to patterns or
  clusters
• Mapping e.g. Mind Map, Process Maps (e.g.
  Pathways of Care), Logic Models
• Integration (with quantitative) congruence,
  contradictions, gaps with explanation

38
Further Reading

• Antin, T. M., Constantine, N. A., Hunt, G.
  (2014). Conflicting Discourses in Qualitative
  Research The Search for Divergent Data within
  Cases. Field Methods, 1525822X14549926.
• Atkins S, Lewin S, Smith H, Engel M, Fretheim A,
  Volmink J. Conducting a meta-ethnography of
  qualitative literature lessons learnt. BMC Med
  Res Methodol. 2008 Apr 16821. doi
  10.1186/1471-2288-8-21.
• Booth, A., Carroll, C., Ilott, I., Low, L. L.,
  Cooper, K. (2013). Desperately Seeking Dissonance
  Identifying the Disconfirming Case in Qualitative
  Evidence Synthesis. Qualitative health
  research, 23(1), 126-141.
• Carroll, C., Booth, A., Lloyd-Jones, M. (2012).
  Should we exclude inadequately reported studies
  from qualitative systematic reviews? An
  evaluation of sensitivity analyses in two case
  study reviews. Qualitative Health
  Research, 22(10), 1425-1434.
• Finfgeld-Connett, D. (2014). Metasynthesis
  Findings Potential Versus Reality.Qualitative
  health research, 1049732314548878.