Does consumer refereeing improve the quality of systematic reviews of healthcare interventions? Perspectives of editors and authors

1
Does consumer refereeing improve the quality of
systematic reviews of healthcare interventions?
Perspective of editors and authors
Gyte Ga, Grant-Pearce Cb, Henderson Sa, Horey Da,
Oliver Sc and Sakala Ca a Cochrane Pregnancy
and Childbirth Group, University of Liverpool
(UK) b PREST, University of Manchester c SSRU,
Institute of Education, University of London.
Abstract
Objective To determine how editors and review
authors view consumer refereeing within the
editorial process for preparing systematic
reviews of effects of healthcare interventions
in particular, their assessment of the impact of
consumer involvement on the quality of Cochrane
reviews, and lessons for consumer involvement in
healthcare research more generally. This
information was sought to help plan a more
extensive evaluation.   Design An independent
researcher undertook semi-structured telephone
interviews with editors, review authors,
consumers, consumer co-ordinators and the
co-ordinator of a Cochrane review group. The
researcher examined routine editorial
documentation and undertook mapping interviews to
understand aims of involving consumers in
research and the Cochrane Collaborations
rationale for involving consumers as referees. A
short questionnaire, asking for overall views of
consumer input into the editorial process,
identified review authors and consumers for
telephone interview. This presentation reports
results from interviews with five review authors
selected to give diverse views, along with four
editors and the Groups co-ordinator. Consumer
views are reported elsewhere. Interviews were
transcribed, and the main issues, impressions and
themes from each were summarised, with resulting
data explored to identify themes.   Results
Key points identified were ? quality of
consumer input was perceived to be positive ?
those with an overview of the review process
considered that consumer input improved the final
review ? earlier consumer input may be
beneficial   Conclusions This evaluation has
identified key issues surrounding consumer
refereeing of systematic reviews undertaken
within the Cochrane Collaboration. Consumers
were considered to provide important
contributions, and suggestions for improvements
in process were made. Further research is
planned to assess more specifically what
additional contribution consumers make, and
whether objectively consumers improve the quality
of Cochrane systematic reviews of healthcare
interventions.
Acknowledgements This research was
supported by a Discretionary Funding grant from
the Cochrane Collaboration Steering Group
2
Aims To determine how editors and review
authors view consumer refereeing within the
editorial process for preparing systematic
reviews of effects of healthcare interventions
in particular, their assessment of the impact of
consumer involvement on the quality of Cochrane
reviews.
This
information was sought to help plan a more
extensive evaluation.
Background Cochrane Pregnancy and Childbirth
Group involves consumers as referees in the
production of systematic reviews of healthcare
interventions to ? ensure the reviews are
targeted at problems important to people ?
take into account outcomes important to people
? ensure accessibility to people making
decisions ? adequately reflect variability in
values, conditions healthcare in different
countries The PCG Consumer Panel (PCGCP)
refereeing process ? Panel includes part-time
paid coordinator, 2 volunteer regional
coordinators, 72 volunteer consumers ?
Consumers choose of the 60 protocols/reviews a
year on which they wish to comment ? Consumer
comments are collated and summarised by the
Consumer Panel Coordinator

• Design
• ? 2 mapping interviews were undertaken to
  understand aims of consumer involvement
• A preliminary questionnaire was sent to all
  review authors and consumers with the group
  (around 300), to select for interview. 70
  replied
• ? 5 authors and 5 consumers chosen to give
  a range of views. Consumer views are reported
• elsewhere. 4 editors with the longest
  experience with the group were chosen
• ? Interview questions were developed by the
  independent researcher based on an understanding
  of
• the review group processes, the aims
  of involving consumers as referees and the aims
  of the project
• ? Interviews were transcribed, and the main
  issues, impressions and themes from each were
• summarised, with resulting data
  explored to identify overall themes
• 18 semi-structured one-hour telephone interviews
  were undertaken with 4 editors, 5 review authors,
  3 consumer coordinators,
• 5 consumers and the review group coordinator

it works well because weve got a a dedicated
person, that can co-ordinate and work with
consumers, who understands what they are saying,
at the same time understands what the review
needs.
I think I can speak for all the editors on
this because it comes up at every single one of
our Editorial meetings, that the input in general
is fantastic. Its the highest quality we get.
Its certainly the most detailed and thoughtful

• Consumer input was viewed positively,
  especially
  by editors
• Consumers brought added value in the form of a
  different perspective
• Consumers provided suggestions for
• ? clarity of language
• ? more meaningful outcomes
• ? more comprehensive enquiry

Results
editor
So its to do with clarity of language, but its
also to do with the centrality of, the importance
of trying to put their views right at the centre,
and their perspective.
editor
it was very long when it came. I think it was
about 18 pagesit took me a while to get over the
heart-sink. I mean it was enormously valuable
a very important role for the consumers in
recommending what issues need to be addressed in
future trials
I think the summary comments from our consumer
coordinator, they are by far the biggest
influence
author
editor
editor
author
they thought particular outcomes were
important, and we hadnt actually considered
those as important outcomes and in fact when we
thought about it, they were..
its important that the Consumer Panel
continues to teach us about the importance of the
persons, of the patients view, of adverse
effects, and of an accessible language
author
author

• Consumers, in coming from a different
  perspective, add an important and valued
  contribution
• The next phase of the evaluation should proceed,
  seeking more objective measures of quality

Conclusion
Contact Gill Gyte at ggyte_at_cochrane.co.uk
Acknowledgements This research
was supported by a Discretionary Funding grant
from the Cochrane Collaboration Steering Group