Use of Systematic Reviews for Policy Development

1
Use of Systematic Reviews for Policy Development

• Dr Jeremy Grimshaw
• Canadian Cochrane Network and Centre
• University of Ottawa

2
Outline of presentation

• Introductions
• How can research support policy making?
• Coffee
• What evidence is available to support policy
  making (using changing physician behaviour as
  example)?
• Summary

3
Introductions

• Name
• Background
• Previous experience of systematic reviews
• What you want to get out of the training?

4
How can research support policy making?
5
Group task (15 minutes)

• In groups of 4-6
• Identify one policy decision faced by one of
  group (preferably relating to a policy change)
• Discuss what sort of information sources that you
  consider when formulating policy
• Discuss how research evidence and what sort of
  research evidence might support formulating
  policy

6
Background

• What is the place for research evidence in
  management and policymaking?
• Helps to get problems on the agenda (i.e., what
  issue should I focus on?)
• Helps to think about problems and solutions
  differently (i.e., how should I begin to approach
  this issue?)
• Helps to solve particular problems at hand (i.e.,
  what program or policy should I support?)
• Helps to justify a decision made for other
  reasons (i.e., how can I sell the position Ive
  taken?)

7
Background

• Types of research evidence that could support
  management and policymaking.
• Does changing X change Y? (effectiveness)
• Is X associated with Y? (relationships)
• How/why does changing X change Y (mechanisms)
• How are X or Y viewed or experienced? (meanings)

8
Background

• Desirable attributes of studies evaluating
  whether changing X changes Y?
• Validity (can we trust the results?)
• Applicability (do the results apply to my
  setting?)

9
Background

• All policy options have potential benefits and
  harms
• Research attempts to identify the benefits and
  harms of alternate options (eg policy A vs policy
  B, policy A vs status quo)
• Important to use methods that allow a fair
  comparison of the policy options so that the
  results truly reflect differences between policy
  options and not other factors

10
Background

• What types of research designs can be used to
  evaluate whether changing X changes Y?

11
Background

• What types of research designs can be used to
  evaluate whether changing X changes Y?
• (Observational)
• Quasi experimental
• Experimental
• (Systematic reviews)

12
Quasi experimental designs

• Before and after studies
• Controlled before and after studies
• Interrupted time series

13
Quasi experimental designs

• Common threats to internal validity
• Secular trends
• Concurrent intervention
• Regression to the mean

14
Before and after studies

• Changes in outcome (e.g. compliance with
  guidelines) measured before and after
  intervention
• Difference assumed to be due to intervention
• What are the strengths and weaknesses of this
  design?

15
Before and after studies
Compliance
Intervention
Time
16
Concurrent intervention
Compliance
Intervention
?
Time
17
Secular trends
Compliance
Intervention
Time
18
Secular trends
Compliance
Intervention
Time
19
Random variation
Compliance
Intervention
Time
20
Delayed effects
Compliance
Intervention
Time
21
Controlled before and after studies

• In controlled before and after studies, the
  researcher attempts
• to identify a control population which has
  similar characteristics and performance to study
  population
• data collected in both populations before and
  after the intervention is applied to study
  population

22
Controlled before and after studies

• difference between study and control performance
  following intervention assumed to be due to
  intervention
• What are the strengths and weaknesses of this
  design?

23
Controlled before and after designs
Compliance
Time
24
Controlled before and after designs
Compliance
Time
25
Interrupted time series analysis

• Attempt to detect whether an intervention has had
  an effect significantly greater than the
  underlying trend
• Data collected at multiple time points before and
  after intervention
• What are the strengths and weaknesses of this
  design?

26
Interrupted time series analysis

• Change in slope

Before
After
27
Interrupted time series analysis

• Step change

Before
After
28
Randomised controlled trials

• Participants are allocated (randomised) to study
  or control group by chance (eg flip of a coin)
• Participants
• Study Control
• (50) (50)

29
Randomised controlled trials

• Use of randomisation should ensure that the two
  groups are equal in all known and unknown factors
  that might influence their response to treatment
• Differences observed between groups due to
  intervention
• Randomised trials should only be used when there
  is genuine uncertainty about the benefits and
  harms of treatment options

30
Buyer beware!

• Studies of all designs commonly suffer bias
  (threats to internal validity) important for
  research users to critically appraise all
  research evidence
• Individual studies are often misleading
• Individual studies provide limited opportunity to
  explore how contextual factors modify outcomes
• Individual studies rarely by themselves provide
  sufficient evidence for policy or practice changes

31
Barriers to use of evidence in policy making

• What are some of the key barriers to using
  evidence in policy making?

32
Barriers to use of evidence in policy making

• What are some of the key barriers to using
  evidence in policy making?
• Over 20,000 health journals published per year
• Published research of variable quality and
  relevance
• Research users (consumers, health care
  professionals and policy makers) often poorly
  trained in critical appraisal skills
• Average time professionals have available to read
  lt1 hour/week

33
Supporting use of evidence in policy making

• Users Guides to the Medical Literature
• We now recommend that resolving a clinical
  problem begins with a search for a valid
  systematic review or practice guideline as the
  most efficient method of deciding on the best
  patient care.
• Guyatt GH, Rennie D (1994). JAMA.

34
Supporting use of evidence in policy making

• Systematic reviews use rigorous scientific
  methods to identify, assess and synthesise the
  worldwide available evidence
• Systematic reviews are an efficient scientific
  approach to identifying and summarising evidence
  ... that allow the generalisability and
  consistency of research findings to be assessed
  and data inconsistencies to be explored.
• Mulrow CD (1994) British Medical Journal

35
Supporting use of evidence in policy making

• Systematic reviews involve
• stating the objectives of the research
• defining eligibility criteria for studies to be
  included
• identifying (all) potentially eligible studies
• applying eligibility criteria
• assembling the most complete dataset feasible
• analysing this dataset, using statistical
  synthesis and sensitivity analyses, if
  appropriate and possible
• preparing a structured report of the research.

36
Supporting use of evidence in policy making

• Systematic reviews of research evidence
• Reduce the likelihood that managers
  policymakers will be misled by research (by being
  more systematic and transparent in the
  identification, selection, appraisal and
  synthesis of studies)
• Increase confidence among managers policymakers
  about what can be expected from an intervention
  (by increasing number of units for study)

37
Supporting use of evidence in policy making

• Systematic reviews of research evidence
• Allow managers, civil servants and political
  staff to focus on appraising the local
  applicability of systematic reviews and on
  collecting and synthesizing other types of
  evidence, such as evidence about political
  acceptability and feasibility i.e., allow them
  to focus on the apex of the research knowledge
  pyramid while doing the rest of their jobs
• Allow for more constructive contestation of
  research evidence by stakeholders

38
Supporting use of evidence in policy making

• Actionable messages
• Systematic reviews of research
• Individual studies, articles, and reports
• Basic, theoretical and methodological innovations

39
Supporting use of evidence in policy making

• Users of systematic reviews still need to
  consider applicability of systematic review
  findings to their setting
• Could it work in my jurisdiction?
• Will it work?
• What would it take to make it work?
• Is it worth it?

40
Supporting use of evidence in policy making

• The Cochrane Library 2007 Issue 2
• Cochrane Reviews
• Full text systematic reviews - ongoing and
  completed (currently 3,094 reviews and 1,707
  protocols)
• Other summaries
• Non Cochrane reviews (6,113 summaries)
• Clinical Trials (495,002 summaries)
• Technology assessments (6,187 summaries)
• Economic evaluations (21,149 summaries)

41
Supporting use of evidence in policy making

• No coordinated effort akin to The Cochrane
  Collaboration has been undertaken to address the
  other types of questions asked by managers
  policymakers
• No one-stop shopping portal has been developed to
  facilitate access to these reviews

42
Supporting use of evidence in policy making

• Key skill is to frame research question clearly
  to facilitate identification of potentially
  relevant systematic reviews
• Information specialists have expertise in helping
  frame questions

43
Supporting use of evidence in policy making

• One popular approach is
• P Participants
• I Intervention
• C Comparison
• O Outcome
• Eg For (P) elderly surgical patients do (I) early
  discharge hospital at home schemes reduce (O)
  total length of stay compared to (C) current
  practice

44
Group task (15 mins)

• Using your group example, try to frame review
  question in terms of
• Participants
• Intervention
• Comparison
• Outcome

45
Summary

• Research evidence has potential to inform policy
  making
• Significant challenges relating to knowledge
  management
• Systematic reviews summarise global evidence and
  allow the influence of context on applicability
  of findings to be explored

46
Summary

• Cochrane Collaboration is worldwide organisation
  undertaking and maintaining systematic reviews of
  what works questions
• Products of The Cochrane Collaboration (and other
  resources) are published in The Cochrane Library
  a one stop shop for what works evidence

47
Summary

• For health system managers and policymakers who
  are interested in systematic reviews
• Develop the skills to acquire, assess, adapt and
  apply systematic reviews
• Become involved in the review process or at least
  push for the profiling of decision-relevant
  information in reviews

48

• COFFEE!

49
Evidence based implementation of evidence based
guidance
50
Towards evidence based implementation

• Most approaches to changing clinical practice are
  more often based on beliefs than on scientific
  evidence
• Evidence based medicine should be complemented
  by evidence based implementation
• Grol (1997). British Medical Journal.

51
Towards evidence based implementation

• If stake holders are to make decisions about
  implementation strategies informed by research
  evidence, they need information on
• Likely effectiveness of interventions (direction
  of effect, predicted effect size of intervention
  and relative effectiveness of different
  interventions)
• likely effect modifiers (context, type of
  targeted behaviour, barriers to change)
• resources need to deliver interventions.

52
Towards evidence based implementation

• What evidence is available to support decision
  makers?

53
Cochrane Effective Practice and Organisation of
Care (EPOC) Group

• EPOC aims to undertake systematic reviews of
  interventions to improve health care delivery and
  health care systems including
• professional interventions (e.g. continuing
  medical education, audit and feedback)
• Financial interventions (e.g. professional
  incentives)
• Organisational interventions (e.g. the expanded
  role of pharmacists)
• Regulatory interventions
• Bero, Eccles, Grilli, Grimshaw, Gruen, Mayhew,
  Oxman, Zwarenstein (2006). Cochrane Library.

54
Cochrane Effective Practice and Organisation of
Care (EPOC) Group

• Progress to date - register and reviews
• Register of 5000 primary studies
• 39 reviews, 39 protocols
• Collaborating with over 300 researchers globally
• Bero, Eccles, Grilli, Grimshaw, Gruen, Mayhew,
  Oxman, Shepperd, Tavender, Zwarenstein (2007).
  Cochrane Library.

55
Overview of reviews of professional behaviour
change strategies

• Identified over 150 systematic reviews of
  professional behaviour change interventions
• For COMPUS, we summarised approx 50 systematic
  reviews judged to be likely highest quality and
  most up-to-date
• Available at http//www.cadth.ca/index.php/en/com
  pus/optimal-ther-resources/interventions

56
Overview of reviews

• Prescribing general (10)
• Prescribing - safety (2)
• Changing roles nursing (1)
• Changing roles pharmacy (7)
• Financial (4)
• Regulatory (1)
• General (10)

• Educational materials (1)
• Educational meetings (1)
• Educational outreach (1)
• Audit and feedback (2)
• Opinion leaders (1)
• Mass media (1)
• Reminders general (4)
• Reminders Computer assisted drug dosage (3)
• Reminders CPOE (1)
• Tailored interventions (1)
• Multifaceted interventions (1)

57
Educational materials

• Distribution of published or printed
  recommendations for clinical care, including
  clinical practice guidelines, audio-visual
  materials and electronic publications. The
  materials may have been delivered personally or
  through mass mailings.
• Target knowledge, skills barriers at individual
  health care professional/peer group level
• Relatively low cost, feasible

58
Educational materials

• Farmer (2007) Cochrane Library (in preparation)
• High quality review
• 21 studies (RCTs, CCTs, CBAs, ITS)
• 9 studies included prescribing data
• Distribution of education materials may be
  effective for appropriate care.
• (Median effect across 6 RCTs 4.9 absolute
  improvement)

59
Educational meetings

• Health care providers who have participated in
  conferences, lectures, workshops or traineeships
• Didactic meetings largely target knowledge
  barriers at individual health care
  professional/peer group level
• Interactive educational meetings can also
  target skills (if simulation/rehearsal involved)
  and attitudes at individual health care
  professional/peer group level

60
Educational meetings

• Thomson OBrien (2001) Cochrane Library
• High quality review
• 32 studies (RCT, CCT)
• Interactive workshops and mixed
  interactive-dogmatic activities were generally
  effective for improving appropriate care. Mixed
  effects were observed for didactic sessions.

61
Educational outreach

• Use of a trained person who met with providers in
  their practice settings to give information with
  the intent of changing the providers practice.
  The information given may have included feedback
  on the performance of the provider(s).

62
Educational outreach

• Derives from social marketing approach
• Use social persuasion methods to target
  individuals knowledge and attitudes
• Typically aim to get maximum of 3 messages across
  in 10-15 minutes using approach tailored to
  individual health care provider
• Typically use additional strategies to reinforce
  approach
• Typically focus on relatively simple behaviours
  in control of individual physician eg choice of
  drugs to prescribe

63
Educational outreach

• Relatively expensive although may still be
  efficient
• May be less effective for complex behaviours
  requiring team or system change

64
Educational outreach

• Thomson OBrien (1997) Cochrane Library
• Medium quality review
• 18 studies (RCT, CCT)
• Multifaceted educational outreach visits were
  generally effective for improving appropriate
  care including prescribing
• (Grimshaw 2004 median effect across 13 RCTs of
  multifacted educational outreahc interventions
  6.0)

65
Local opinion leaders

• Use of providers nominated by their colleagues as
  educationally influential. The investigators
  must have explicitly stated that their colleagues
  identified the opinion leaders.
• Target peer group knowledge, attitudes
• Resources required include survey of target
  group, resources to recruit and support opinion
  leaders.

66
Local opinion leaders

• Doumit (2007) Cochrane Library
• High quality review
• 12 studies (RCT, CCT)
• Generally effective for improving appropriate
  care.
• Median effect across studies 10 absolute
  improvement

67
Local opinion leaders

• Appear to be condition specific
• Likely coverage of target group difficult to
  assess
• Grimshaw et al (2006). Implementation Science
• Stability over time uncertain Doumit
  re-surveyed surgeons 2 years after initial survey
  to identify opinion leaders. Only 4/16 original
  opinion leaders re-identified
• Doumit (2006) Masters thesis

68
Audit and feedback

• Any summary of clinical performance of health
  care over a specified period of time. The
  summary may also have included recommendations
  for clinical action. The information may have
  been obtained from medical records, computerised
  databases, or observations from patients.

69
Audit and feedback

• Target health care provider/peer groups
  perceptions of current performance levels
• Adams et al demonstrated that self reported
  behaviour likely to overestimate actual
  performance by 27
• Adams et al (1999) Int Journal for Quality in
  Health Care
• Aim to develop cognitive dissonance to motivate
  physicians to change

70
Audit and feedback

• Resources required include data abstraction and
  analysis costs, dissemination costs (postal or
  personal)
• Feasibility may depend on availability of
  meaningful routine administrative data for
  feedback

71
Audit and feedback

• Jamvedt (2005) Cochrane Library
• High quality review
• 118 studies (RCT, CCT)
• Audit and feedback alone, audit and feedback with
  educational meetings, audit and feedback as part
  of multifaceted intervention generally effective.
• Median effect across studies 10 absolute
  improvement
• Larger effects were seen if baseline compliance
  was low.

72
Reminders

• Patient or encounter specific information,
  provided verbally, on paper or on a computer
  screen, which is designed or intended to prompt a
  health professional to recall information. This
  would usually be encountered through their
  general education in the medical records or
  through interactions with peers, and so remind
  them to perform or avoid some action to aid
  individual patient care. Computer aided decision
  support and drugs dosage are included.
• Focus on professional patient interaction,
  prompting professional to remember to do
  important items

73
Reminders

• Resources vary across deliver mechanism
• Increasing interest in computerised decision
  support but evidence tends to come from a few
  highly computerised US academic health science
  centres
• Insufficient knowledge about how to prioritise
  and optimise reminders

74
Reminders

• Garg (2005) JAMA
• Medium quality review
• 100 studies (RCT, CCT)
• Mixed effects were observed for computerised
  clinical decision support systems (CDSS) for
  appropriate care

75
Multi faceted interventions

• Any intervention including two or more components
• Multi-faceted interventions are more likely to
  target different barriers in the system
• Likely more costly than single interventions
• Need to carefully consider how components likely
  to interact to maximise benefits

76
Multifaceted interventions

• Grimshaw et al (2004). Health Technology
  Assessment

77
Summary

• Substantial (if incomplete) evidence base on
  effects of interventions to change professional
  behaviour
• Availability of systematic reviews, allows
  decision maker to focus on question of
  applicability of the results to her setting and
  question.

78
Contacts

• Canadian Cochrane Centre
• jgrimshaw_at_ohri.ca
• cochrane_at_uottawa.ca
• http//www.cochrane.uottawa.ca/
• The Cochrane Collaboration
• http//www.cochrane.org
• The Cochrane Consumer Network
• http//www.cochrane.org/consumers/homepage.htm