Title: University of Oxford Continuing Education Qualitative Research and Systematic Reviews Workshop: ORGA
1University of Oxford Continuing
EducationQualitative Research and Systematic
Reviews Workshop ORGANISING AND MANAGING
QUALITATIVE DATA IN QUALITATIVE SYNTHESIS
- Study Selection Critical Appraisal and Inclusion
and Exclusion Criteria
2This presentation
- Inclusion and Exclusion Criteria
- Critical appraisal and
- The Debates Surrounding Critical Appraisal and
the Exclusion/Inclusion of Studies.
3Evidence for health policy and practice
4The Evidence Interests of Clinicians
5- Clinicians need information up to 60 times per
week (twice per every three patients), and it
could affect eight decisions per day - D. G. Covell, G. C. Uman and P. R. Manning.
Information Needs in OfficePractice. Are they
being met? Annals of Internal Medicine 1985,103,
pp. 596-9.
6- Self-reports from Oxford in 1997 showed that
while medical students might spend 60 mins per
week reading professional journals, house
officers spent none, and even consultants spent
as little as 30 - D. L. Sackett. Surveys of self-reported reading
times of consultants in Oxford, Birmingham,
Milton Keynes, Bristol, Leicester, and Glasgow.
In W. M. C. Rosenberg et al, Evidence-based
Medicine. Churchill Livingstone, 1997.
7Clinicians and policy makers express a need for
- Concise summaries of the best available appraised
evidence that addresses their evidence interests - To inform (not direct) services/practices
8The SR Process
- Quantitative Reviews
- Question
- Inclusion Criteria
- Search Strategy
- Critical Appraisal
- Extraction
- Synthesis
- Qualitative Reviews?
- ?
- ???
- ?
- ???
- ?
- ?
9Inclusion Criteria
- Naturally flow from well constructed review
questions
10- Effects
- Population
- Intervention
- Comparison
- Outcome
- Feasibility, Appropriateness, Meaningfulness
- Setting
- Perspective
- Intervention/Phenomena of Interest
- (Comparison)
- Evaluation
- Adapated from Booth A. Using research in
practice Australian supermodel? A practical
example of evidence-based library and information
practice. Health Information and Libraries
Journal 2006 23 69-72
11Critical Appraisal
12- Critically appraising studies and excluding those
judged to be of poor quality is a well accepted
step in the systematic review process. - Critical appraisal aims at ensuring that
systematic reviews draw their conclusion from
good-quality research evidence and is one of the
characteristics of systematic reviews that
differentiates them from literature reviews - There is, however, disagreement amongst those who
conduct systematic reviews of effects on the
degree to which critical appraisal improves the
rigour of reviews. Edwards et al (1998) for
example, suggest that reviews may underestimate
the evidence if they only include those of a
certain methodological quality - Edwards, A.G., Russell, I.T. and Stott, N.C.
(1998) Signal versus noise in the evidence
basefor medicine an alternative to hierarchies
of evidence?, Family Practice. 15 4, 319-22
13Applying critical appraisal criteria to
qualitative studies is a contentious practice!
- Dixon-Woods et al (2004) argue that Some means
of appraising qualitative research is needed if
it is to contribute appropriately to systematic
reviews - Attree and Milton 2006 argue that Critical
appraisal must be central to research syntheses,
thus enabling reviewers to make only good-quality
qualitative evidence accessible for policy makers
and practitioners - Dixon-Woods, M., Shaw, R. L., Agarwal, S. and
Smith, J.A. (2004a) The problem of appraising
qualitative research, Quality and Safety in
Health Care.133, 223-5 - Attree, P. and Milton, B. (2006) Critically
appraising qualitative research for systematic
reviews defusing the methodological cluster
bombs Evidence Policy. 21, 109-26
14Applying critical appraisal criteria to
qualitative studies is a contentious practice!
- Garratt and Hodkinson (1998) argue, however, that
it is illogical to attempt to predetermine a
fixed set of criteria to appraise qualitative
research, as the meaning of research only emerges
in the interaction between the findings and the
critical reader. - Others argue that important understandings
emerging from qualitative studies judged to be of
poor quality may contribute to theorising and
that, therefore, quality should not be used as a
criterion to exclude studies in qualitative
reviews - Garratt, D. and Hodkinson, P. (1998) Can there
be criteria for selecting research criteria? A
hermeneutical analysis of an inescapable
dilemma, Qualitative Inquiry. 4 4, 515-39
15Popay, Rogers and Williams (1998) suggest that
the development of standards for assessing
evidence from qualitative research is both
possible and desirable.Popay, J., Rogers, A.
and Williams, G. (1998) Rationale and standards
for the systematicreview of qualitative
literature in health services research,
Qualitative Health Research. 8 3, 341-51
16Popay, Rogers and Williams (1998) provide the
following as a guide to common standards
- evidence of responsiveness to social context and
flexibility of design - evidence of theoretical or purposeful sampling
- evidence of adequate description
- evidence of data quality
- evidence of theoretical and conceptual adequacy
and - potential for assessing typicality.
17Dixon-Woods et al (2004)suggest that
- Some means of appraising qualitative research is
needed if it is to contribute appropriately to
systematic reviews. - Proposals for criteria that might define high
quality qualitative research have proliferated,
but sometimes do not overlap or are difficult to
operationalise. - A minimal set of prompts is proposed to help cue
attention to the range of dimensions of
qualitative research that require appraisal. - There is a need for additional criteria that
recognise the diversity of study designs and
theoretical perspectives in qualitative research,
and to distinguish between minor errors and fatal
flaws. - The measurement of all aspects of quality of
qualitative research will remain difficult.
18Dixon-Woods et al (2004)suggest the following
criteria
- Are the research questions clear?
- Are the research questions suited to qualitative
inquiry? - Are the following clearly described?
- sampling
- data collection
- analysis
- Are the following appropriate to the research
question? - sampling
- data collection
- analysis
- Are the claims made supported by sufficient
evidence? - Are the data, interpretations, and conclusions
clearly integrated? - Does the paper make a useful contribution?
19Pearson (2004) takes the view that a transparent
approach to appraising qualitative research -
sensitive to the nature of the qualitative
research and its basis in subjectivity - is
central to its ongoing credibility,
transferability and theoretical potential.
Pearson A. (2004) Balancing the Evidence
Incorporating the Synthesis of Qualitative Data
into Systematic Reviews. JBI Reports245-64
20Pearson (2004) suggests the following framework
for appraising qualitative research
- Congruity between the philosophical/theoretical
position adopted in the study and the study
methodology the study methods the
representation of the data and the
interpretation of the results - The degree to which the biases of the researcher
are made explicit and - The relationship between what the participants
are reported to have said/done and the
conclusions drawn in analysis.
21(No Transcript)
22The debates
- The ongoing, and frequently fierce, contemporary
debates surrounding the place critically
appraising or assessing the quality of -
qualitative studies can only be understood by
situating the various viewpoints within the
traditions they are grounded in and locating the
points of difference between these positions
within the broader evidence based health care
discourses. - Qualitative researchers who support the notion
that qualitative research findings represent
evidence and the development of approaches to
systematically review qualitative findings that
address specific phenomena of interest or
questions have yet to reach consensus on a common
process and there is continuing, vigorous debate
surrounding these issues. - As Dixon-Woods et al (2004) states qualitative
research is not a unified field thus,
qualitative researchers are grounded in diverse
research traditions and the field is
characterised by ongoing debate.
23fit for purpose
- Much of the work to-date on qualitative research
review is generated by scholars from the social
sciences whose interests lie more in
understanding and theorising the social world and
individual experience. Given that evidence based
practice is commonly associated with decision
making by health and social care practitioners,
Newman et al (2006) question the fit of
qualitative research with the evidence-based
practice model. They suggest that
Conceptualising qualitative clinical questions
and the corresponding answers or products in this
way raises the question of whether qualitative
research should fit within the evidence-based
practice decision making framework at all. - Newman, M., Thompson, C., and Roberts, A.P.
(2006) Helping practitioners understand the
contribution of qualitative research to
evidence-based practice. Evidence-Based Nursing
94-7
24- Social scientists interests in the systematic
review of qualitative research findings focus on
those suggested by Booth (2001) - 1 For the generation of models and theories
- 2 For the subsequent validation and gauging of
the empirical support for theories - In methodological reviews as a means of
determining the existence of certain bodies or
"schools" of thought. - To chart the development of concepts
longitudinally in the literature over a period of
time - To complement the findings and interpretation of
quantitative systematic reviews, for example in
looking at issues around patient acceptance or
compliance. - As a means of identifying significant domains or
attributes to enable the development of prototype
instruments or scales.
25- Health and social care practitioners, policy
makers and managers look to systematic reviews of
qualitative evidence to provide summarised
information on which to base decision making.
Their interests focus on the more pragmatic
matters suggested by Lavis et al (2005) - Rigorous reviews that are potentially
reproducible, though generally researchers are
assumed to know their business - Trustworthy, transparent methods
- Relevant, up-to-date answers to their questions
in their context/population - Accessible presentation of findings with clear
messages - Timeliness
- Information about risks (harms) as well as costs
benefits, preferably by population sub-groups - Some indication of uncertainty associated with
estimates - Lavis J, et al. Towards systematic reviews that
inform health are management and policy making.
JHSRP 200510(suppl 1) 35-48
26The purpose of the process
- Clarifying the purposes of qualitative
meta-synthesis may well facilitate consensus on
appropriate processes. Qualitative reviewers with
a clinical or health background and who see the
purpose of qualitative reviews as informing
clinical decision making do not generally oppose
adapting the current Cochrane Collaboration
approach in ways that sensitive to the
qualitative approach to inquiry. There is a
growing number of published systematic reviews
that follow all of the steps of the Cochrane
Collaboration approach modified to accommodate
agreed criteria for inclusion criteria, critical
appraisal, data extraction and data synthesis - Such an approach is not supported by reviewers
whose interests lie more in the development of
theory or in illuminating a phenomena (albeit as
a way of assisting practitioners, policy makers
and planners to gain insights to enhance decision
making).
27the debate continues!!