Can Consumers Improve the Quality of Cochrane Reviews?

1
Can Consumers Improve the Quality of Cochrane
Reviews?

• Carol Grant-Pierce, Carol Sakala, Dell Horey
• Gill Gyte, Sonja Henderson and Sandy Oliver
• 12th Cochrane Colloquium
• Ottawa, Canada
• October 2-6, 2004

2
Cochrane Reviewers Handbook10.2 Consumer
Involvement

• Essential, to help ensure that reviews
• are targeted at problems important to people
• take into account outcomes important to people
• are accessible to people making decisions
• adequately reflect variability in values,
  conditions healthcare in different countries

Cochrane Pregnancy and Childbirth Group
3
Consumer Participation in Research

• Growing involvement of consumers in Cochrane and
  other research
• Growing literature on consumers and research
• Cochrane Review Editorial peer-review for
  improving the quality of biomedical studies

Cochrane Pregnancy and Childbirth Group
4
Pregnancy Childbirth Group Consumer Panel
Background

• Established 1998
• Since January 1999, comments on all protocols
  reviews going through editorial process over 300
  documents altogether
• 1st 2 years described in Birth 200128(2)133-7
• For current editorial process, see The Editorial
  Team. Pregnancy and Childbirth Group. About the
  Cochrane Collaboration (Collaborative Review
  Groups (CRGs)). In The Cochrane Library 2004,
  Issue 3.

Cochrane Pregnancy and Childbirth Group
5
Cochrane Pregnancy Childbirth Group Editorial
Process
Contact Editor
Authors
Start of Process Draft Protocol/Review

• Referees
• Internal peer
• External peer
• Statistician

Office
Office

• Consumer Panel
• Consumer Panel Coordinator
• Regional Consumer Coordinators
• Europe Australasia North America
  Low-, mid-income nations
• 45 12 10
  4
• Active Consumers

6
PCG Consumer PanelWhy Evaluate?

• Assess impact resource allocation
• Improve administrative process
• Provide feedback to PCG participants
• Obtain needed resources in future
• Improve quality of Cochrane external research

Cochrane Pregnancy and Childbirth Group
7
Consumer Panel EvaluationPhase I

• Interviews with 4 Editors, Review Group
  Coordinator, 5 authors, 5 consumers, all 3
  Consumer Coordinators
• What is their assessment of
• whether the quality of reviews is improved?
• how the process works?
• With support from Cochrane Collaboration Steering
  Group Discretionary Fund

Cochrane Pregnancy and Childbirth Group
8
Editor Author ResultsFavorable Assessment of
Impact

• Strong overall assessment
• Adds value to professional refereeing
• Identifies overlooked outcomes
• Improves accessibility
• Helps represent range of perspectives

Cochrane Pregnancy and Childbirth Group
9
Editor Author ResultsFavorable Assessment of
Impact

• Editor The input in general is fantastic. Its
  the highest quality we get.
• Editor As an editor, well theres no doubt some
  of the best, most constructive comments about the
  review come from the Consumer Panel.
• Author I think we get excellent constructive
  feedback on the reviews.

Cochrane Pregnancy and Childbirth Group
10
Editor Author ResultsFavorable Assessment of
Impact

• Author Certainly in terms of outcome measures
  that were felt important to consumers, we
  hadnt actually considered those as important
  outcomes and in fact, when we thought about it,
  they were.
• Author The language, will be accessible to
  someone who is not doing medicine all the day.
• Editor Some of the approaches the consumers
  guide us into thinking through are really
  probably new from a health profession point of
  view.

Cochrane Pregnancy and Childbirth Group
11
Editor Author ResultsConcerns About Impact

• Input with agenda or emotionality perceived as
  less generally relevant
• Outcomes may not be perceived as valid if not
  already in literature
• Unwieldy to make all technical information
  comprehensible to general audience

Cochrane Pregnancy and Childbirth Group
12
Editor Author ResultsConcerns About Impact

• Author Well the only negative comment is that
  I dont think you can explain absolutely
  everything so that absolutely everyone can
  understand it.
• Editor Its just been this problem of sometimes
  emotionalism that gets in the way.
• Author There is a real tension about who one is
  writing this for on the whole, when Im
  writing, I think Im writing this for other
  doctors.

Cochrane Pregnancy and Childbirth Group
13
Editor Author ResultsFavorable Assessment of
Process

• Dedicated and skillful coordinator works well
  with consumers and professionals
• Helpful to have summaries that highlight key
  points, along with full detailed collated
  feedback

Cochrane Pregnancy and Childbirth Group
14
Editor Author ResultsFavorable Assessment of
Process

• Author I think the consumers take it very
  seriously. theyre not destructive.
• Author Its good to actually get a lot of
  criticism.
• Author We do get very constructive comments
  by and large, and also a good summary from
  coordinator, and its generally quite easy to
  respond to.
• Author The feedback was very professional, and
  professionally handled.

Cochrane Pregnancy and Childbirth Group
15
Editor Author ResultsFavorable Assessment of
Process

• Editor It works well because weve got a really
  good coordinator. a dedicated person that can
  coordinate work with consumers, who understands
  what they are saying at the same understands what
  the review needs.
• Editor Feedback is very ably summarized. We
  have a covering note that reminds us that even
  though we might have a long list, its about the
  following, of issues that need to be considered
  dealt with. Teasing out from all the
  information there what are really the key points
  is helpful for a reviewer editorial process.

Cochrane Pregnancy and Childbirth Group
16
Editor Author ResultsConcerns About Process

• Volume of full collated feedback can be
  overwhelming
• Some feedback difficult to address if first
  available later in process
• Inadequate input from low-income countries

Cochrane Pregnancy and Childbirth Group
17
Editor Author ResultsConcerns About Process

• Author I think its simply my memory of these
  18 pages arriving and just thinking, Cant do
  this.
• Editor If the consumer said at the protocol
  stage, there are a number of important things
  missing I would ask the reviewers to think
  seriously about including them. If it was at
  the review stage, I probably wouldnt draw
  attention to that.
• Editor I know its been a struggle all along to
  get input from women in low-income countries.

Cochrane Pregnancy and Childbirth Group
18
Phase 1 Evaluation Summary

• Apart from helping to identify review topics,
  appears to contribute to Handbook objectives
• Skill mix of Principal Consumer Coordinator
  central to success, including provision of
  summary cover note, diplomacy and content
  knowledge
• Large volume of feedback can be both strength and
  challenge
• Need to develop better outreach to consumers in
  low- middle-income countries

Cochrane Pregnancy and Childbirth Group
19
Consumer Panel EvaluationPhase II

• Content analysis to assess whether consumer input
• adds value to input from authors peer referees
• improves quality of final review
• RCTs evaluating different models of consumer
  participation

Cochrane Pregnancy and Childbirth Group
20
For Information, To Participate, or To Obtain
Copy of Slides

• Gill Gyte
• PCG Principal Consumer Coordinator ( Europe,
  low- middle-income countries)
• ggyte_at_cochrane.co.uk
• Carol Sakala
• PCG Consumer Coordinator for North America
• sakala_at_maternitywise.org
• Dell Horey
• PCG Consumer Coordinator for Australia
• dhorey_at_ozemail.com.au

Cochrane Pregnancy and Childbirth Group