Issues and Challenges around Searching the Literature

1
Issues and Challenges around Searching the
Literature
2
Overview

• What is Already Known on Searching the
  Qualitative Research Literature
• Overview of Methodological Issues/Challenges
• Recent Developments (with focus on literature of
  last two years)
• Outstanding Issues/Challenges

3
What is already known
4
What is Already Known? 1 (Gallacher et al, 2013)

• Finding relevant qualitative studies arduous due
  to inadequate indexing (Dixon Woods et al, 2007
  Ring et al  2011)
• Papers often lack abstracts or include
  non-informative titles, making it difficult to
  establish relevance  (Dixon Woods et al, 2007).
• Several papers outline strategies (filters) for
  searching for qualitative studies (Walters et
  al, 2006 Wilczynski et al, 2007 Wong et al,
  2004 McKibbon et al, 2006)
• Helpful techniques involve electronic or hand
  searching (Greenhalgh et al 2005 Bates, 1989)
• Reference or footnote tracking (looking backwards
  at references in articles found).
• Citation tracking (tracking forward subsequently
  citing articles).
• Personal knowledge and personal contacts.
• Contacting the authors of known papers or experts
  in the field.
• Hand searching relevant journals.
• Internet browsing such as berry picking (where
  one search leads to another and clusters of
  papers are often found together).
• Greenhalgh et al (2005) found only 30 of primary
  sources from predefined search strategy. 51
  found by other predefined methods (i.e.
  reference, footnote and citation tracking).
• However this may be topic specific and may
  reflect a priori decisions to concentrate on
  non-electronic sources

5
What is Already Known? - 2

• Literature searchesopen-ended iterative
  processes where the topic or research question of
  interest is honed over time as the nature of the
  evidence becomes more apparent (Finfgeld-Connett
  Johnson 2013)
• Number of articles not only critical factor
  reports may lack enough thick description to
  fully develop concepts and the interrelationships
  among them (Finfgeld-Connett Johnson 2013)
• Unpublished studies may contain rich, thick
  description
• Goal may not be aggregative theoretical
  saturation may play a part selection of sample
  is crucial

6
References for What is Known Already - 1

• Bates MJ The design of browsing and berrypicking
  techniques for on-line search interface. Online
  Rev 1989, 13407-424.
• Dixon-Woods M, Bonas S, Booth A et al. How can
  systematic reviews incorporate qualitative
  research? A critical perspective. Qual
  Res 2006, 627-44.
• Finfgeld-Connett, D., Johnson, E. D. (2013).
  Literature search strategies for conducting
  knowledge-building and theory-generating
  qualitative systematic reviews. Journal of
  Advanced Nursing, 69(1), 194-204.
• Greenhalgh T, Peacock R Effectiveness and
  efficiency of search methods in systematic
  reviews of complex evidence audit of primary
  sources. BMJ 2005, 3311064-1065.
• Ring N, Jepson R, Ritchie K Methods of
  synthesising qualitative research for health
  technology assessment. Int J Technol Assess
  Health Care 2011, 27384-390

7
References for What is Known Already - 2

• Walters LA, Wilczynski NL, Haynes RB Developing
  optimal search strategies for retrieving
  clinically relevant qualitative studies in
  EMBASE. Qual Health Res 2006, 16162-168. PubMed A
  bstract  Publisher Full Text 
• Wilczynski NL, Marks S, Haynes RB Search
  strategies for identifying qualitative studies in
  CINAHL. Qual Health Res 2007, 17705-710. PubMed A
  bstract  Publisher Full Text 
• Wong SL, Wilczynski NL, Haynes RB Developing
  optimal search strategies for detecting
  clinically relevant qualitative studies in
  MEDLINE. In Medinfo 2004 Proceedings of the 11th
  World Congress on Medical Informatics San
  Francisco. Edited by Fieschi M, Coiera E, Jack Li
  YC. Amsterdam IOS Press 2004311-314. 
• McKibbon KA, Wilczynski NL, Haynes RB Developing
  optimal search strategies for retrieving
  qualitative studies in PsycINFO. Eval Health
  Prof 2006, 29440-454. PubMed Abstract  Publisher
   Full Text 

8
Overview of Methodological Issues/Challenges

• Bias towards/Predominance of quantitative
  research and publication of resultant reports
• Non-optimal indexing of qualitative studies
  (CINAHL more evolved than MEDLINE)
• Qualitative research represents various research
  methodologies, including ethnography,
  phenomenology, grounded theory and narrative
  analysis, which may hinder retrieval
• Lack of informative manuscript titles and
  abstracts

9
Overcoming barriers

• Searches should be as transparent as possible
  without jeopardizing the creativity and
  complexity of the process (Finfgeld-Connett
  Johnson 2013)
• Key test for knowledge-building/theory
  generating review would small amounts of
  con?icting information substantially change the
  ?ndings? Theoretical Saturation/ Qualitative
  Sensitivity Analysis
• Weaknesses in indexing mean that sensitivity of
  searches may need to be reduced to allow time for
  other search strategies (Pearson et al, 2011)

10
Simple search strategies vs complex ones

• Three broad-based terms (i.e. qualitative,
  ?ndings and interviews) as effective as more
  complex search strategies in identifying relevant
  qualitative research reports (Flemming Briggs
  2007).
• Search strategies with broad search terms
  (qualitative research or qualitative studies
  or interviews) combined with CAM terms had
  highest recall and precision (Franzel, 2013).
• Within time-limited context, protocol-driven,
  targeted, and reference-checking search
  strategies most effective (Pearson et al, 2011)

11
Intervention Searching vs Condition Searching
(Lorenc et al, 2012)

• Tying search terms of SR of qualitative evidence
  too closely to interventions may compromise
  consistency of the review.
• Dilemma Performing condition-wide searches (with
  no other change to strategies) would become
  highly over-inclusive and volumes of records
  impracticably large. (Suggests need for
  alternative sampling stategies)

12
Is More Necessarily Better?

• For knowledge-building and theory-generating
  systematic reviews, more is better only when it
  helps to fully explicate a concept, substantiate
  an interconnection between or among concepts, or
  build a line of argument. Simply, more of the
  same does not necessarily help to achieve these
  objectives. In fact, collecting more of the same
  may merely escalate the cost of a study, clutter
  the database and obfuscate important inferences.
  Concepts and interrelationships among them can
  only be more fully explicated based on data that
  adds depth, breadth, meaning and understanding to
  a phenomenon (Finfgeld-Connett Johnson 2013).

13
Searching for religion and mental health studies
required health, social science, and grey
literature databases (Wright et al, 2014)

• PsycINFO best performing database
• ArabPsyNet, CINAHL, Dissertations and Theses,
  EMBASE, Global Health, Health Management
  Information Consortium, MEDLINE, PsycINFO, and
  Sociological Abstracts essential to retrieve
  included references.
• Citation tracking and personal library of one of
  the research teams made significant contributions
  of unique, relevant references.
• Religion studies databases (Am Theo Lib Assoc,
  FRANCIS) did not provide unique, relevant
  references.
• Literature searches for reviews/evidence
  syntheses of religion and health studies should
  include social science, grey literature,
  non-Western databases, personal libraries, and
  citation tracking activities.

14
Exhaustive versus Expansive
That is the Question!
15
Exhaustive versus Expansive (Finfgeld-Connett
Johnson 2013)

• Exhaustive searches when conducting summative and
  aggregative systematic reviews
• Expansive searches when conducting
  knowledge-building and theory-generating
  systematic reviews

16
Exhaustive versus Expansive

• Multiple databases
• Comprehensive list of terms
• Assumption of homogeneity
• Other approaches are supplementary

• Databases to reflect contributing disciplines
• Terminology may characterise narrative of each
  discipline
• Likelihood of divergent cases
• Sibling studies (shared context)
• Cluster searching
• Citation searching
• Supplementary techniques may be more important

Combing the Area
Following Up Leads
17
Outstanding challenges (Gallacher et al, 2013)

• Creating an appropriately sensitive and specific
  search strategy was a significant challenge.
• Adding qualitative methods made search strategy
  considerably more specific while retaining
  sensitivity, as demonstrated by return of all key
  papers identified in scoping search.
• Our final results showed that 94 of papers were
  identified by our predefined database search.

18
Recent Developments

• Importance of Context/Theory
• Database Coverage
• Importance of Supplementary Search Methods
• Appropriate Selection of Sampling Methods
• Reporting Standards (e.g. ENTREQ)

19
Searching for Contextual Richness

• Requires identification of related (sibling)
  reports i.e. cluster searching (cluster becomes
  unit of analysis, not study) (Booth et al, 2013)

20
Cluster searching for Siblings (Booth et al,
2013)
Element Procedural Steps
Citations Backwards reference chaining
Lead Authors Author searching Backwards reference chaining
Unpublished materials Web searches repositories
Scholar searches Citations
Theories Backwards reference chaining
Early Examples Cited works (Forward reference chaining)
Related Projects Co-citations
21
Searching for Theories

• Theory not typically reported in Abstracts
• Theoretical base differs by discipline cp. HSR vs
  Public Health vs Nursing vs Psychology vs
  Sociology
• Reporting of Theory differs by discipline
• Level of Theory may vary e.g. Individual versus
  Society (Psychology vs Sociology)
• When is a theory a Theory? labelling (model,
  framework, concepts ) and naming (Health Belief
  Model)

22
Database Coverage

• CAM search - PubMed yielded 87 of relevant
  included qualitative studies (Franzel et al,
  2013).
• Five different QES PubMed coverage values 35/44
  (79.5) 9/10 (90) 10/11 (91) 9/9 (100)
  7/28 (25 - Grey literature) (Booth 2012
  Unpublished)
• But 5/28 of studies located only through
  supplementary searches of three sources
  (Stansfield et al, 2012). 21 search sources
  required to locate all studies. Explanation -
  Role of Grey Literature and Geographical focus
  UK only

23
Supplementary Strategies

• More diffuse topic, move beyond electronic
  searching

24
Supplementary Strategies - 1

• Multiple search strategy more likely to identify
  relevant QR than sole reliance on electronic
  searching.
• Purpose of synthesis determines appropriate
  sampling/search strategy. E.g. mapping out key
  conceptual developments if aim not aggregative,
  omission of papers unlikely to have dramatic
  effect on results.
• Suggests max. circa 40 papers - difficult to
  maintain sufficient familiarity with gt 40 papers
  (Campbell et al, 2011)

25
Supplementary Strategies - 2

• Need belt and braces (hand-searching
  consultation with experts)
• Searching for books/theses particularly
  challenging (not indexed in same way as journal
  papers) (Campbell et al, 2011)
• Snowballing and consultation with experts for a
  realist review (Pawson et al, 2004)
• Obtaining authors' suggestions -
  resource-intensive process with negligible
  results (Pearson et al, 2011)
• Additional search techniques essential to locate
  further high quality references (Papaioannou et
  al, 2010)

26
Sampling Appropriate ? Comprehensive (Suri, 2011)

• 16 strategies for sampling in QES
• E.g. Snowball sampling - seeking information from
  key informants about other information-rich
  cases.
• The chain of recommended informants would
  typically diverge initially as many possible
  sources are recommended, then converge as a few
  key names get mentioned over and over (Patton,
  2002, p. 237).
• Identify most cited primary research reports by
  footnote chasing (searching citation indices,
  browsing through bibliographies, previous
  research syntheses, primary research reports,
  policy documents, papers written by practitioners
  and papers written for practitioners).

27
Combination or Mixed Purposeful Sampling (Suri,
2011)

• Employ two or more sampling strategies to select
  evidence to adequately address purpose.
• Mixed purposeful sampling can facilitate
  triangulation and flexibility in meeting the
  needs of multiple stakeholders (e.g. extensive
  sampling for generalisations at higher level of
  abstraction. Typical case sampling to provide
  readers with immediacy of typical studies that
  contributed towards informing more abstract
  generalisations).
• When selecting combination of sampling
  strategies, synthesists must reflect on how
  strategies complement each other.

28
Footnote chasing (Suri, 2011)

• cp. footnote chasing for exhaustive sampling,
  footnote chasing for snowball sampling involves
  locating most cited papers.
• However, may reinforce confirmatory bias (i.e.
  studies agreeing with prevalent wisdom more
  likely to be published and cited, studies that
  contest conventional wisdom less likely to be
  published or cited)

29
ENTREQ Searching (Tong et al, 2012)
3 Approach to searching Indicate whether search was pre-planned (comprehensive search strategies to seek all available studies) or iterative (to seek all available concepts until they theoretical saturation is achieved).
4 Inclusion criteria Specify inclusion/exclusion criteria (e.g. in terms of population, language, year limits, type of publication, study type).
5 Data sources Describe information sources used (e.g.electronic databases (MEDLINE, EMBASE, CINAHL, psycINFO, Econlit), grey literature databases (digital thesis, policy reports), relevant organisational websites, experts, information specialists, generic web searches (Google Scholar) hand searching, reference lists) and when searches conducted provide rationale for using data sources.
6 Electronic Search strategy Describe literature search (e.g. provide electronic search strategies with population terms, clinical or health topic terms, experiential or social phenomena related terms, filters for qualitative research, and search limits).
30
Improvement in Search Reporting
1988-2004 2005-2008
(Hannes Macaitis, 2012)
31
Outstanding Issues/Challenges

• How to understand publication bias in qualitative
  research (not around positive/negative findings)?
• How to systematise (and document) more intuitive
  search approaches e.g. cluster searching and
  searching for theory?
• How does sampling strategy translate into search
  strategy?
• How to construct sampling frames for studies?
• How to sample for diversity?
• How many sources are enough?
• How to retrieve rich data? How to retrieve data
  on theory and context?

32
Publication Bias in Qualitative Research? Part
One

• This does not mean that publication biases do
  not exist in.qualitative research.a bias of
  potentially greater proportions may threaten
  searches for qualitative research reports..in
  some circles, qualitative research is
  perceivedof lesser quality and value than
  quantitative researchqualitative studies may be
  less frequently conducted, submitted for
  publication and/or published in high quality and
  easily accessible journals.raw data (i.e.
  research ?ndings).needed to conduct a
  qualitative systematic review may not be readily
  available (Finfgeld-Connett Johnson 2013)

33
Publication Bias in Qualitative Research? Part
Two

• ignoring grey literature, such as
  dissertations/theses, government reports,
  monographs and books, on the basis that it may be
  of lesser quality (as with quantitative research)
  is empirically and logically invalid..such
  documents may be particularly rich sources of
  qualitative data as page limits are not generally
  imposed. Also, although lengthy report formats
  are relatively uncommon in the health sciences,
  they tend to be the norm in disciplines, such as
  anthropology.., where context-rich data are
  likely to be found. (Finfgeld-Connett Johnson
  2013)
• Qualitative researchers often choose to publish
  in book form (Truncation bias)

34
Conclusions

• Increasing Importance of Explanatory Sources
  (e.g. Context and Theory)
• Need for Ongoing Investigation of Database
  Coverage and Supplementary Search Techniques
• Requires Exploration/Selection of Appropriate
  Sampling Methods
• Bottomline Value versus Effort Trade-Off

35
References - 1

• Booth A, Harris J, Croot E, Springett J, Campbell
  F, Wilkins E. Towards a methodology for cluster
  searching to provide conceptual and contextual
  "richness" for systematic reviews of complex
  interventions case study (CLUSTER). BMC Med Res
  Methodol. 2013 Sep 2813118.
• Campbell R, Pound P, Morgan M, Daker-White G,
  Britten N, Pill R, et al. Evaluating meta-ethnogra
  phy systematic analysis and synthesis of
  qualitative research. Health Technol
  Assess 201115(43). 
• Flemming, K., Briggs, M. (2007). Electronic
  searching to locate qualitative research
  evaluation of three strategies. Journal of
  Advanced Nursing, 57(1), 95-100.
• Gallacher K, Jani B, Morrison D, Macdonald S,
  Blane D, Erwin P, May CR, Montori VM, Eton DT,
  Smith F, Batty GD, Mair FS International
  Minimally Disruptive Medicine Workgroup.
  Qualitative systematic reviews of treatment
  burden in stroke, heart failure and diabetes -
  methodological challenges and solutions. BMC Med
  Res Methodol. 2013 Jan 281310.

36
References - 2

• Franzel, B., Schwiegershausen, M., Heusser, P.,
  Berger, B. (2013). How to locate and appraise
  qualitative research in complementary and
  alternative medicine. BMC complementary and
  alternative medicine, 13(1), 125.
• Guise, J. M., Chang, C., Viswanathan, M., Glick,
  S., Treadwell, J., Umscheid, C. A., ...
  Trikalinos, T. (2014). Systematic Reviews of
  Complex Multicomponent Health Care Interventions
  Internet. http//www.ncbi.nlm.nih.gov/books/NBK1
  94851/
• Hannes, K., Macaitis, K. (2012). A move to more
  systematic and transparent approaches in
  qualitative evidence synthesis update on a
  review of published papers. Qualitative
  Research, 12(4), 402-442.
• Papaioannou, D., Sutton, A., Carroll, C., Booth,
  A., Wong, R. (2010). Literature searching for
  social science systematic reviews consideration
  of a range of search techniques. Health
  Information Libraries Journal, 27(2), 114-122. 

37
References - 3

• Pearson, M., Moxham, T., Ashton, K. (2011).
  Effectiveness of search strategies for
  qualitative research about barriers and
  facilitators of program delivery. Evaluation
  the Health Professions, 34(3), 297-308.
• Stansfield, C., Kavanagh, J., Rees, R.,
  Gomersall, A., Thomas, J. (2012). The selection
  of search sources influences the findings of a
  systematic review of peoples views a case study
  in public health. BMC Medical Research
  Methodology, 12(1), 55.
• Suri, H. (2011). Purposeful sampling in
  qualitative research synthesis. Qualitative
  Research Journal, 11(2), 63-75.
• Tong, A., Flemming, K., McInnes, E., Oliver, S.,
  Craig, J. (2012). Enhancing transparency in
  reporting the synthesis of qualitative research
  ENTREQ. BMC medical research methodology, 12(1),
  181.
• Suri, H. (2013). Towards Methodologically
  Inclusive Research Syntheses Expanding
  Possibilities. Routledge.
• Wright JM, Cottrell DJ, Mir G. Searching for
  religion and mental health studies required
  health, social science, and grey literature
  databases. J Clin Epidemiol. 2014
  Jul67(7)800-10.