Searching beyond the RCT - looking for sibling studies on qualitative, economic and process research

1
Searching beyond the RCT - looking for sibling
studies on qualitative, economic and process
research
Faten Hamad and Christine Urquhart
2
Introduction Systematic reviews producing the
evidence We have the evidence on what works (or
doesnt) what happens next? Finding
qualitative, economic and process research
evidence preliminary findings of research to
identify sibling studies associated with
particular randomised controlled trials.
3
Aims and objectives of presentation are
to Discuss searching strategies to find
evidence beyond the randomised controlled trial
Present preliminary findings Assess your
reactions to the findings!
4
Background Sibling studies is the name which has
been chosen to indicate the relationship that
groups a set of related studies (randomised
control trial, qualitative, process and economic
evaluations). An intervention may be viewed as
a complex system where intervention itself is
a fragile creature that is delivered in a social
system of interacting elements, such as an
individuals capacity, interpersonal
relationships, institutional setting and
infrastructure.
5
Search strategies and search filters Existing
work by Cochrane Information Retrieval Methods
Group and others (e.g. Hedges team) Aim to
maximise recall (sensitivity), keeping precision
reasonable, and ensuring specificity Search
strategies vary with the database depends on
index terms, which terms have to be added as free
text terms Relevance judgements
6

• Seed studies selection
• We chose a range of topic areas, and dates for
  the large RCTs and chose five seed studies with
  different characteristics. We will discuss the
  result of the following two of seed studies
• Telemedicine and diabetes (a known RCT, with many
  known direct siblings, that could be used to
  validate and checking the search strategies for
  their sensitivity).
• Tamoxifen for breast cancer prevention (known
  qualitative sibling, two RCTs involved).

7

• Search strategies and databases under
  investigation
• Different search strategies and different
  databases were explored for the reason of
  exploring the differences in search performance
  in regard to different study areas.
• - Author-Subject search (using a very simple
  subject term combination) with each of the author
  names in the seed article in turn, in MEDLINE on
  PubMed.
• Related article search in MEDLINE on PubMed (for
  the seed article).
• - E-library search (with a combination of ISI
  (WoS), OCLC WorldCat, OCLC Articles First, EBSCO
  Business Complete, and EBSCO International
  Bibliography) - simple subject term combination
  only, and limited to the first 300 documents
  retrieved.

8

Search strategies and databases under
investigation-Cont. - SCOPUS search
(author-subject search as in MEDLINE on
PubMed). - CINAHL (author-subject search as in
MEDLINE on PubMed). - Cited reference in ISI, Web
of Science (with the seed article as the
reference). - Cited reference search in SCOPUS. -
Cited reference search in CINAHL.
9
Results
Table 1 IDEATeL study search strategy and
retrieval performance

search strategy Relevant/R Total retrieved Unique Relevant/R Odds Ratio
Related Search(PubMed) 25 186 5 1.01219
Author Subject(PubMed) 23 157 4 1.09189
Citation(Web of Science) 14 57 1 1.86855
Subject search(e-library) 39 296 33 1.19589
SCOPUS Author subject 39 52 5 23.64179
SCOPUS citation 32 64 12 7.13514
CINAHL Author subject 17 18 0 100.85393
CINAHL citation 4 7 0 6.90196
Total relevant retrieved without duplicates 106
Total retrieved without duplicate 634
Total non-relevant retrieved without duplicates 528
10
Table 2 IDEATeL study search strategies
retrieval per study type and odds ratio
calculations.

search strategy RCTs RCTs RCTs Qualitative Qualitative Qualitative Economical evaluation Economical evaluation Economical evaluation Process evaluation Process evaluation Process evaluation
search strategy R1 R/N2 OR3 R1 R/N2 OR3 R1 R/N2 OR3 R1 R/N2 OR3
Related Search(PubMed) 8 39 0.1032 10 21 0.2396 3 8 0.1887 4 13 0.1548
Author-Subject(PubMed) 8 39 0.1271 9 22 0.2534 2 9 0.1377 4 13 0.1906
Citation(Web of Science) 4 43 0.1990 8 23 0.7442 2 9 0.4755 0 17 0
Subject search(e-library) 25 22 0.2034 20 11 0.3254 9 2 0.8055 6 11 0.0976
SCOPUS Author - subject 17 30 2.9205 12 19 3.2551 2 9 1.1453 8 9 4.5812
SCOPUS citation 19 28 1.5692 8 23 0.8044 2 9 0.5139 3 14 0.4955
CINAHL Author subject 10 37 24.054 4 27 13.185 1 10 8.9 2 15 11.867
CINAHL citation 1 46 0.7391 3 28 3.6429 0 11 0 0 17 0
1 Relevant retrieved. 2 Relevant not
retrieved. 3 Odds Ratios.
11
Table 3 Tamoxifen study search strategy and
retrieval performance

search strategy Relevant/R Total Unique Relevant/R Odds Ratio
Related Search(PubMed) 17 200 13 0.58277
AuthorSubject(PubMed) 17 451 7 0.24573
Citation(Web of Science) 5 53 2 0.60155
Subject search (e-library) 59 288 54 2.316121
SCOPUS Author subject 72 229 47 4.76069
SCOPUS citation 4 59 2 0.41722
CINAHL Author subject 19 41 7 5.49701
CINAHL citation 1 4 0 1.87527
Total relevant R without duplicates 156
Total retrieved without duplicate 1028
Total non-relevant R without duplicates 872
12
Table 4 Tamoxifen study search strategies
retrieval per study type and odds ratio
calculations.
search strategy RCTs RCTs RCTs Qualitative Qualitative Qualitative Economic Economic Economic Process evaluation Process evaluation Process evaluation
search strategy R1 R/N2 OR3 R1 R/N2 OR3 R1 R/N2 OR3 R1 R/N2 OR3
Related Search(PubMed) 12 83 0.1098 3 31 0.0735 1 0 DIV/0 1 25 0.0304
Author-Subject(PubMed) 12 83 0.0463 1 33 0.0097 0 1 0 4 22 0.0582
Citation(Web of Science) 2 93 0.0677 2 32 0.1966 0 1 0 1 25 0.1258
Subject search(e-library) 26 69 0.1629 18 16 0.4864 0 1 0 13 13 0.4323
SCOPUS Author-subject 52 43 0.6470 10 24 0.2229 0 1 0 10 16 0.3344
SCOPUS citation 0 95 0 4 30 0.3685 0 1 0 0 26 0
CINAHL Author subject 15 80 1.1676 1 33 0.1887 0 1 0 3 23 0.8123
CINAHL citation 1 94 0.0750 0 34 0 0 1 0 0 26 0
1 Relevant retrieved. 2 Relevant not
retrieved. 3 Odds Ratios.
13
Discussion

• CINAHL author-subject search was the most
  effective search that can precisely retrieve
  direct and indirect siblings of a certain seed
  study in general with higher score in the case of
  the IDEATeL study in specific.
• - IDEATeL, the number of retrieved records was 41
  (with 19 relevant)
• Tamoxifen, the number retrieved was 18 (with 17
  relevant)
• CINAHL author- subject odds ratios were quite
  different, but this does not affect the fact this
  search strategy was the best search strategy, in
  terms of the chances of finding relevant material
  from a search, with proportionally fewer
  irrelevant items retrieved.

14
The SCOPUS authorsubject strategy was the second
best search performance. For both the seed
studies, the e-library subject search retrieved
most of the unique studies. Simple subject
terms and / or the combination of simple subject
terms and author names for each seed study,
appeared to be the most effective method of
retrieving most of the siblings, outperforming
citation searching (apart from SCOPUS citation
with the IDEATeL seed study).
15
Conclusions and future works The analysis
indicates that there is neither a winner in the
search strategies nor for the databases. The
CINAHL author-subject performed well (in terms of
precision). The SCOPUS author-subject search
performed the next best, (higher recall for the
relevant studies for both seed studies). The
e-library author-subject search produced a good
number of relevant studies (and a high proportion
of unique items as well).
16
Conclusions and future works Further research
will examine how expanding the number of
databases and changing the selection will affect
the relative performance of the e-library
Meta-lib search. Further work is needed to
identify how grey literature, conference
proceedings and thesis and dissertation material
can be obtained efficiently, as Web of Knowledge
found some of the direct sibling publications for
the IDEATel studies that could not be obtained on
any of the other databases used.
17
Thank you for Listening