Critical Appraisal of Practice Guidelines

1
Critical Appraisal of Practice Guidelines

• Ciprian Jauca
• Vancouver, BC, Canada
• jauca_at_ti.ubc.ca

ISDB General Assembly, December 2008, Matagalpa,
Nicaragua
2
Outline of presentation

• GRADE and GRADE PROfiler
• AGREE
• ADAPTE
• GIN Guidelines International Network
• Case study treatment of hypertension

3
How can we judge how sure we are that adherence
to a recommendation (our good intentions) will do
more good than harm?
4
Professional good intentions and plausible
theories are insufficient for selecting policies
and practices for protecting, promoting and
restoring health.
Iain Chalmers
5
Humility and uncertainty are preconditions for
unbiased assessments of the effects of the
prescriptions and proscriptions of policy makers
and practitioners for other people.
Iain Chalmers
6
We will serve the public more responsibly and
ethically when research designed to reduce the
likelihood that we will be misled by bias and the
play of chance has become an expected element of
professional and policy making practice, not an
optional add-on.
Iain Chalmers
7
Grading of Recommendations Assessment,
Development, and Evaluation (GRADE) Working Group

• www.gradeworkinggroup.org

8
What do you know about GRADE?

• Have prepared a guideline
• Read the BMJ paper
• Have prepared a systematic review and a summary
  of findings table
• Have attended a GRADE meeting, workshop or talk

9
About GRADE

• Began as informal working group in 2000
• Researchers/guideline developers with interest in
  methodology
• Aim to develop a common system for grading the
  quality of evidence and the strength of
  recommendations that is sensible and to explore
  the range of interventions and contexts for which
  it might be useful
• 13 meetings (10 35 attendants)
• Evaluation of existing systems and reliability
• Workshops at Cochrane Colloquia, WHO, GIN and
  various conferences since 2000

10
Why bother about grading?

• People draw conclusions about the
• quality of evidence
• strength of recommendations
• Systematic and explicit approaches can help
• protect against errors
• resolve disagreements
• facilitate critical appraisal
• communicate information
• However, there is wide variation in currently
  used approaches

11
Grading the Evidence

• Evidence concepts
• scientific results that approximate truth
• size, accuracy, precision
• reliability, reproducibility, appropriateness,
  bias
• statistical descriptions
• trade-offs, limiting factors, cost
• Grade components
• Quality (Validity)
• The quality of evidence indicates the extent to
  which one can be confident that an estimate of
  effect is correct.
• Strength (Benefit/Risk)
• The strength of a recommendation indicates the
  extent to which one can be confident that
  adherence to the recommendation will do more good
  than harm.

12
How to Grade Recommendations

• Strong recommendations
• strong methods
• large precise effect
• few downsides of therapy
• expect non-variant clinician and patient behavior
• diminished role for clinical expertise
• focus on implementation barriers
• focused role of patient values and preferences
• emphasis on compliance and barriers
• Weak recommendations
• weak methods
• imprecise estimate
• small effect
• substantial downsides
• expect variability in clinician and patient
  actions
• clinical expertise important
• focus on decision-making and implementation
• patient values and preferences important
• focus on determining values and preferences
  relative to decision

Clinical Experience
Research Evidence
Patient Values and Preferences
from Holger Schünemann
13
Implications of a strong recommendation

• Patients Most people in your situation would
  want the recommended course of action and only a
  small proportion would not
• Clinicians Most patients should receive the
  recommended course of action
• Policy makers The recommendation can be adapted
  as a policy in most situations

14
Implications of a weak recommendation

• Patients The majority of people in your
  situation would want the recommended course of
  action, but many would not
• Clinicians Be prepared to help patients to make
  a decision that is consistent with their own
  values
• Policy makers There is a need for substantial
  debate and involvement of stakeholders

15
Grading the Evidence

• The 5 steps in the GRADE approach, which follow
  these judgments, are to make sequential judgments
  about
• Which outcomes are critical to a decision
• The quality of evidence across studies for each
  important outcome
• The overall quality of evidence across these
  critical outcomes
• The balance between benefits and harms
• The strength of recommendations

16
How to create / produce an evidence summary

• Choose critical outcomes first (define)
• Typically use an existing systematic review
  (alternatively can start with other evidence
  synthesis, or search for original literature, or
  supplement existing evidence summary with
  additional evidence about other outcomes)
• Specify population (subpopulation),
  interventions
• Complete an evidence summary
• GRADEpro facilitates completion of
  a summary of findings evidence table,
  with quality grading
• Having included all critical outcomes, it will be
  possible to judge balance of benefits and risks

17
Judgments about Recommendations
Uncertain balance Or Equal balance
Trade-Offs
Net Benefits Or Net Harms
Evidence Quality
High
Strong Recommendation
Evaluate Values Preferences
Strong Recommendation Evaluate Values
Preferences
Moderate
Low
Weak Recommendation
Weak Recommendation Evaluate Values
Preferences
Very Low
Expert Opinion
No Recommendation
No Recommendation
No Recommendation
from Jeff Andrews
18
Effect of Cost Example
1
Net Benefit
Zero Net
Benefits Absolute Benefit Increase X Utility
The strength of a recommendation indicates the
extent to which one can be confident that
adherence to the recommendation will do more good
than harm.
COST
Net Benefits
Net Burden
Net Burden
0
Risks Harms Burdens Absolute Risk Increase X
Utility (Value Factor)
1
0
from Jeff Andrews
19
GRADE Profiler
20
GRADE profiler (GRADEpro)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
GRADE Profile

• Excel, HTML, MS Word format
• Can be linked to RevMan

27
APPRAISING CLINICAL PRACTICE GUIDELINES
28
PURPOSE OF THE AGREE INSTRUMENT

• To provide a systematic framework for appraising
  the quality of clinical guidelines
• To help guideline developers follow a structured
  and rigorous methodology
• To help policymakers decide which guideline to
  recommend for use in practice
• To help health care providers assess guidelines
  before adopting recommendations in practice

29
DEFINITION

• Quality of clinical guidelines is the
  confidence that
• the potential biases of guideline development
  have been addressed adequately
• the recommendations are both internally and
  externally valid, and are feasible for practice

30
VALIDATION PROCESS (1)

• Phase 1
• The instrument was applied to 100 guidelines from
  11 countries
• All countries followed the same protocol
• Each guideline was independently assessed by 4
  appraisers (194 altogether)
• Instrument was used in English

31
VALIDATION PROCESS (2)

• Phase 2
• Refined instrument tested on 33 guidelines
  randomly selected from 1st study
• Same countries participated
• New set of appraisers recruited (70)
• Each country assessed 3 guidelines

32
STRUCTURE

Six domains

• 23 items
• 4-point Likert Scale

1. Scope purpose (3) 2. Stakeholder
involvement (4) 3. Rigour of development (7) 4.
Clarity presentation (4) 5. Applicability
(3) 6. Editorial independence (2)
Overall assessment
User guide
33
RESPONSE SCALE
34
DOMAIN 1.SCOPE AND PURPOSE

• 1. The overall objective(s) of the guideline
  is(are) specifically described.
• 2. The clinical question(s) covered by the
  guideline is(are) specifically described.
• 3. The patients to whom the guideline is meant to
  apply are specifically described.

35
DOMAIN 2.STAKEHOLDER INVOLVEMENT

• 4. The guideline development group includes
  individuals from all the relevant professional
  groups.
• 5. The patients views and preferences have been
  sought.
• 6. The target users of the guideline are clearly
  defined.
• 7. The guideline has been piloted among target
  users.

36
DOMAIN 3.RIGOUR OF DEVELOPMENT (1)

• 8. Systematic methods were used to search for
  evidence.
• 9. The criteria for selecting the evidence are
  clearly described.
• 10. The methods used for formulating the
  recommendations are clearly described.
• 11. The health benefits, side effects and risks
  have been considered in formulating the
  recommendations.

37
DOMAIN 3.RIGOUR OF DEVELOPMENT (2)

• 12. There is an explicit link between the
  recommendations and the supporting evidence.
• 13. The guideline has been externally reviewed by
  an expert panel prior to publication.
• 14. A procedure for updating the guideline is
  provided.

38
DOMAIN 4.CLARITY AND PRESENTATION

• 15. The recommendations are specific and
  unambiguous.
• 16. The different options for management of the
  condition are clearly presented.
• 17. Key recommendations are easily identifiable.
• 18. The guideline is supported with tools for
  application.

39
DOMAIN 5.APPLICABILITY

• 19. The potential organisational barriers in
  applying the guideline have been discussed.
• 20. The potential costs implications of applying
  the recommendations have been considered.
• 21. The guideline presents key review criteria
  for monitoring and/or audit purposes.

40
DOMAIN 6.EDITORIAL INDEPENDENCE

• 22. The guideline is editorially independent from
  the funding body.
• 23. Conflicts of interest of guideline
  development members have been recorded.

41
CALCULATING DOMAIN SCORES

• Standardised guideline domain scores are
  calculated by
• summing up all the scores of individual items in
  a domain
• and
• by standardising the total as a percentage of the
  maximum possible score for that domain

42
EXAMPLE DOMAIN SCORE (1)

Item 1

Item 2

Item 3

Totaal

Appraiser 1

2

3

3

8

Appraiser 2

3

3

4

10

Appraiser 3

2

4

3

9

Appraiser 4

2

3

4

9

Total

9

13

14

36


Max. possible score 4 (strongly agree) x 3
(items) x 4 (appraisers) 48



Min. possible score 1 (strongly disagree) x 3
(items) x 4 (appraisers) 12


43
OVERALL ASSESSMENT (1)

• Would you recommend these guideline for use in
  practice?
• Strongly recommend
• Recommend (with provisos or alterations)
• Would not recommend
• Unsure

44
OVERALL ASSESSMENT (2)

• Do not aggregate the six domain scores into a
  single quality score!
• Take each appraisal criteria into account
• Use common sense as well

45
CONCLUSIONS

• AGREE is the first appraisal instrument for
  clinical guidelines to be developed and tested
  internationally
• It can be used consistently by a wide range of
  professionals from different cultural backgrounds
• Need several appraisers to assess one guideline
• Domain scores should not be aggregated

46
Website address
http//www.agreecollaboration.org
47
ADAPTE Framework

• The ADAPTE framework is a systematic approach to
  aid in the adaptation of guidelines produced in
  one setting to be used in a different cultural
  and/or organizational context.
• The ADAPTE framework is intended to be used by
  different groups, i.e. guideline developers,
  health care providers and policy makers at the
  local, national and international level.

48
ADAPTE Framework Core Principles

• Respect of evidence-based principles for
  guideline development
• Reliable and consistent methods to ensure the
  quality of the adapted guideline
• Participation of key stakeholders to foster
  acceptance and ownership of the adapted
  guideline
• Explicit consideration of context to ensure
  relevance for local practice and policies
• Transparent reporting to promote confidence in
  the recommendations of the adapted guideline
• Flexible format to accommodate specific needs and
  circumstances
• Respect for and acknowledgement of source
  guidelines material.The ADAPTE framework is
  supported by training resources, in particular a
  Manual and a Technical guide which includes tools
  that describe and facilitate guideline adaptation.

49
(No Transcript)
50
The ADAPTE Manual

• The ADAPTE collaboration has developed a Manual
  for guideline adaptation as well as a Resource
  Toolkit.
• These are being made freely available
  atwww.adapte.org

51
Guidelines International Network

• The Guidelines International Network (G-I-N) is
  an international not-for-profit association of
  organizations and individuals involved in the
  development and use of clinical practice
  guidelines.
• G-I-N seeks to improve the quality of health care
  by promoting systematic development of clinical
  practice guidelines and their application into
  practice, through supporting international
  collaboration.
• Founded in 2002, G-I-N has grown to 89
  organizational members and partners representing
  38 countries from Africa, America, Asia, Europe
  and Oceania. A number of international
  organizations such as the WHO are also members.

52
Guidelines International Network

• The Network has the world's largest Guideline
  Library and is regularly updated with the latest
  information about guidelines of the G-I-N
  membership.
• As at December 2008 more than 5,500 documents are
  available on their website
• www.g-i-n.net

53
Case Study

• Male or female, 65 years old.
• Smoker, BMI 28, plays golf 3 times per week.
• Recent diagnosis, Type 2 Diabetes managed by
  diet.
• Average resting sitting BP measured by doctor
  172/89 mmHg.
• Standing BP 166/92 mmHg.

54
Middle age is when broadness of the mind and
narrowness of the waist change places.
55
Canadian Hypertension Education Program (CHEP)
Guidelines

• Updated and published yearly.
• Most members have financial conflicts of
  interest.
• Program and guideline meetings are funded by the
  Drug Companies who market antihypertensive drugs.
  

56
2008 CHEP Guidelines

• Treat to target. In general, blood pressure
  should be lowered to less than 140/90 mmHg and in
  those with diabetes or chronic kidney disease, to
  less than 130/80 mmHg.
• Combinations of therapies (both drug and
  lifestyle) are generally necessary to achieve
  target blood pressures.
• Most patients require more than one
  antihypertensive drug and lifestyle changes to
  achieve recommended blood pressure targets.

57
2008 CHEP Guidelines (cont)

• Monitor patients whose blood pressure is above
  target regularly and increase the intensity of
  treatment until the targets are achieved.
• Regular follow-up and titration of therapy is
  required to achieve blood pressure targets.
• Focus on adherence. Non-adherence to therapy is
  an important cause of poor blood pressure
  control.
• Patient adherence to therapy should be assessed
  on each visit and interventions made to improve
  adherence should be a part of clinical routine.

58
Guideline treatment of patient

• Smoking cessation advice.
• Low sodium diet, high in fruits and vegetables
  and high in low fat dairy products.
• Regular cardiovascular exercise.

59
Guideline treatment of patient

• Target lt 130/80 mmHg.
• First-line thiazide, ACEI, CCB, ARB or
  combination.
• If fail to reach target, maximize dose and add
  other drugs.
• Add statin and low dose ASA.

60
Likely scenario at 2 years

• Patients resting sitting BP is 152/64 mmHg.
• Drugs thiazide, ACE inhibitor, CCB, beta
  blocker, alpha blocker, statin, low dose ASA,
  metformin, etc.
• Patient has unpleasant side effects.
• Patient and doctor are frustrated and
  dissatisfied.

61
2008 Evidence-based approach

• Best available evidence in 2008
• Mulrow et al systematic review in Cochrane
  Library 1998.
• Patients gt 60 years old.
• RCTs treatment vs. placebo or no treatment.
  gt20,000 patients.
• Mortality benefit, RR 0.88 0.82 0.94

62
2008 EBM approach (cont)

• Total cardiovascular events, RR 0.73 0.68
  0.77, ARR 5, NNT 20 for 5 years.
• Maximum number of drugs 3 and 40 of patients did
  not achieve target of lt160 mmHg systolic or lt90
  mmHg diastolic.
• Included patients with diabetes and likely is
  relevant to our patient.

63
(No Transcript)
64
Where does the target evidence come from?

• In people with elevated blood pressure does a
  lower BP target as compared to a traditional BP
  target change mortality and morbidity in RCTs?
• Traditional BP target lt160/90 mmHg.

65
What evidence is required to answer this question?

• Trials in which patients with hypertension are
  randomized to the traditional target and one or
  more lower blood pressure targets.
• Must at least report one of the desired outcomes.

66
What outcomes are desired?

• Total mortality
• Total people with at least one serious adverse
  event (including cardiovascular SAEs and all
  other SAEs)
• Achieved differences in blood pressure.

67
What evidence is available?

• A systematic review conducted by Dr. Jose
  Arguedas
• No published RCTs assess systolic BP targets?
• 7 RCTs assessed two or more diastolic or mean BP
  targets?

68
Achieved BP differences in 6 trials

• Systolic -3.9 -4.3, -3.5 mmHg
• Diastolic -3.4 -3.6, -3.2 mmHg

69
Morbidity and mortality

• None of the trials reported total SAEs
• Total mortality in 6 RCTs
  RR 0.99 0.82, 1.21

70
RevMan Forest Plot
71
HOT subgroup analysesMortality

• Men RR 1.10 0.97-1.24
• Women RR 0.98 0.83-1.15
• Smokers RR 1.32 1.08-1.62
• Non-smokers RR 0.98 0.86-1.09

72
Hot subgroup analysesMortality

• IHD RR 1.03 0.85-1.24
• No IHD RR 1.06 0.94-1.19
• Diabetes RR 0.77 0.58-1.02
• No Diabetes RR 1.10 0.99-1.22

73
(No Transcript)
74
Case

• Male or female, 65 years old.
• Smoker, BMI 28, plays golf 3 times per week.
• Recent diagnosis, Type 2 Diabetes managed by
  diet.
• Average resting sitting BP measured by doctor
  172/89 mmHg.

75
2008 EBM treatment of patient

• Smoking cessation advice.
• Low sodium diet, high in fruits and vegetables
  and high in low fat dairy products.
• Regular cardiovascular exercise.

76
2008 EBM treatment of patient

• Start with low dose thiazide. In Canada
  hydrochlorothiazide 12.5 mg daily (cost 2
  cents/day).
• If resting systolic remains gt 160 mmHg, increase
  to 25 mg daily (cost 4 cents/day)
• If resting systolic remains gt 160 mmHg add
  ramipril 2.5 mg and increase to maximum of 5 mg
  (cost lt 50 cents/day).
• Alternatively perindopril 2 mg and 4 mg (use half
  tablets to keep cost lt 50 cents/day)

77
2008 EBM treatment of patient

• Do not prescribe statin.
• Do not prescribe low dose ASA.

78
Likely scenario at 2 years

• Patients resting sitting BP is 152/64 mmHg.
• Drugs hydrochlorothiazide 25 mg, ramipril 5 mg,
  metformin.
• Patient has no side effects.
• Patient and doctor are satisfied.

79
Implication for researchneeded Target BP RCT

• Outcomes that measure both benefit and harm
  total mortality and SAEs.
• Use a BP range 140 150 mmHg versus 130 139
  mmHg systolic.
• Use traditional antihypertensive drugs in both
  comparison groups.
• Require two measurements on separate dates for
  change in medication.

80
Other needed RCTs

• Compare CHEP guideline approach to EBM approach.
• One drug versus two drug versus 3 or more drug
  approaches.

81
Eat well, stay fit, die anyway.