Faculty of Public Health

1
Faculty of Public Health

• Developing Quality and Risk Management Within an
  Evaluative Culture

2
Housekeeping

• Fire Exits
• Fire Alarms
• Etc

3
Format

• 10.00 - 11.00 Session 1
• Break
• 11.30 12.30 Session 2
• Lunch
• 1.30 2.30 Session 3
• 2.45 3.30 Review and Wind Up

4
Required

• KH24 Assessment of evidence of effectiveness of
  services, programmes and interventions, which
  impact on health.
• KH25 The different ways of assessing outcomes
  from a range of perspectives e.g. patient
  satisfaction qualitative outcomes, patient
  acceptability, quality of life
• KH26 The principles and methods of evaluation,
  audit, research, development and standard setting
  as applied to improving quality

5
This Section

• KH24
• Assessment of evidence of effectiveness of
  services, programmes and interventions, which
  impact on health.
• What are the important factors here?

6
Exercise 1

• On the sheet
• Evidence
• What do you define as health?
• What services, programmes or interventions would
  you like evidence on?
• Name 3 types of evidence
• Name 4 places you would look for it
• Define Effectiveness

7
What is Health?

• WHO definition
• "a state of complete physical, mental and social
  well being and not merely the absence of disease
  or infirmity."
• How much of this is the NHS responsible for?

8
Effectiveness

• 'Doing the right thing in the right way for the
  right patient at the right time. 
• This involves getting evidence of what works into
  everyday clinical practice and evaluating its
  effect on patient care'

9
Effectiveness

• The conscientious, explicit, and judicious use
  of current best evidence in making decisions
  about the care of individual patients (and
  patients in general) patients. The practice of
  evidence based medicine means integrating
  individual clinical expertise with the best
  available external clinical evidence from
  systematic research." (BMJ, 13 Jan 96, Sackett,
  David L.).

10
Services, programmes or interventions?

• Clinical interventions (operations or drugs)
• Public Health Interventions such as stop smoking,
  weight management, etc
• You say!

11
Types of Evidence
12
Types of Evidence

• Hierarchy of Evidence
• Covered in Critical Review Course
• Often quoted in terms of Operations, Drugs, etc
  etc, but not for public health
• Handout

13
Pragmatic vs Perfect

• Is anything perfect?
• What is the value of perfect information?
• Diminishing returns
• Critical Overview?
• Summary of best knowledge at moment
• Just good enough?
• Just in time?

14
Where to Look for It

• NICE
• PubMed
• Centre for Reviews and Dissemination
• Google
• Grey Literature
• Use of Citation Software
• Emailing people to ask for unpublished work
• Etc
• Check out http//www.hullpublichealth.org
• Paper on CD how to write an evidence based
  clinical review article

15
Checklist

• See paper on CD
• Internal Validity
• Extent to which paper/evaluation (not only RCT)
  avoids biases . See list and discuss
• External Validity
• Generalisability How good is this evidence if
  applied to my patients/clients

16
Internal vs External

• Internal validity important but in public health
  evidence we need to consider carefully the
  external validity and what this really will do to
  our patients.

17
Systematic Overview

• Examples of our overviews

18
Economic Evaluation

• What does this all mean to people (links to
  external validity)
• Medical outcome measures
• Improvement in visual acuity scales
• Reduction in hba1c
• Reduction in blood pressure
• Are they always meaningful to the patient
• Is it possible to do something to people which
  makes their overall quality of life worse?
• Views?

19
Economic Evaluation

• Possible to do something very expensive to people
  which may only have marginal health related
  quality of life benefits but may be very
  expensive
• Eg new drugs
• Will money spent on this mean that other people
  will not be treated
• Finite budgets and unlimited demand?

20
Economic Evaluation

• spending on health now means less spending on
  education, crime control, infrastructure,
  defence, etc.
• Most important economists want to maximise the
  health benefit for patients (individually and as
  a whole.
• Pareto Optimum - Society seen as a sum of
  individual benefits

21
Economic Evaluation

• Utilitarian viewpoint
• the greatest good for the greatest number
• hedonic calculus
• Attempt to measure health related quality of
  life

22
Economic Evaluation

• Economists want to measure Costs and Benefits
• Costs
• Drugs
• Nursing and doctoring
• Infrastructure
• Costs to patients important though
• 2 perspectives usually
• NHS Payer Perspective
• Societal Perspective (takes account of impact on
  patients and society. E.g. travel cost, lost
  wages, forgone leisure, lost productivity not
  considered not a slave society

23
Economic Evaluation

• Benefits
• Mortality. How many years of life saved by a
  certain drug/intervention
• With screening may use number of cases found as
  the outcome measure.
• Morbidity. How much health improvement might be
  brought about by the intervention (e.g. how much
  is hearing improved by a hearing aid or how much
  is blood pressure reduced per unit of a drug
  (which costs so much)
•  Treatment of hypertension mmHG blood pressure
  reduction
• Diagnosis of deep vein thrombosis - Cases
  detected
• Asthma - No of attack free days
• Thrombolysis - Years of lives gained

24
Economic Evaluation

• Decision Rule Between Programs
• Whichever gives the most benefit to a group of
  patients.
• Problem with measures as identified in previous
  slides is that you cant compare one with another
  (apples and pears)

25
Economic Evaluation

• Cost Utility Analysis
• Handout

26
Cost Benefit Analysis

• Costs and benefits in monetary units
• Willingness to pay
• (Contingent valuation and conjoint analysis)
• Is WTP an appropriate measure of the value of a
  medical intervention?
• Tends to be accepted by economists who accept
  welfare economics / Pareto criterion (Little?)
• Mark Pauly (USA)
• Ability to pay?
• Value of a (statistical life)
• Equity

27
Cost-Utility Analysis

• How to measure utility?
• Standard gamble favourite method
• Cardinal state value elicitation has not yet
  reached the level of methodological
  sophistication required.
• But has the ability to enable more informed
  choices to be made which incorporate patient
  views in the decision making process.

28
Cost-Utility Analysis

• Costs measured in usual way
• Benefits measured in estimated utility gains
• Attempts to measure utility using a normalised
  cardinal utility function
• Values each conceivable health state on a scale
  between death and full health
• Calculate a Quality Adjusted Life Year (QALY)

29
Cost-Utility Analysis

• QALY combines utility of health with time (1
  year)
• Use Cost per QALY as an outcome measure
• Decision rule is to start at the lowest cost per
  QALY and continue up the rankings until all money
  is spent
• (See Weinstein in Sloan, Ch 5)

30
QALY
Health related Q of L / Utility
With Treatment
Without treatment
Time
31
Cost-Effectiveness Analysis

• utility

Utility gain is gap between lines
With treatment
Without treatment
0
time
32
Cost-Utility Analysis

• Questionnaire based to find out how people value
  health states
• Euroquol attempt to produce a QALY measurement
  which is acceptable throughout Europe
• Health Utility Index (HUI and HUI2) (McMasters
  University Canada)
• SF36 and Brazier algorithm

33
Cost-Utility Analysis

• QALY
• Advantages
• One off measure of health outcome
• Allows comparison between different health
  interventions/treatments
• Takes account of wishes and utility of patients
• (some treatments may extend life but worsen
  quality of life)
• What is quality of life of people in pain

34
Cost-Utility Analysis

• But whose utilities should be the guiding ones
  Patient Group or Society as a Whole
• Complaints from some groups that society
  misunderstands their utility levels (deaf)
• Are Utility measures applicable between
  countries?
• Are QALY measures sufficiently robust and
  comparable?

35
Exercises and look at Questionnaires

• SF36
• Visual Analogue Scale
• Euroqol (EQ5D)
• Health Utilities Index

36
Policy

• Operating Framework
• Outcomes (utility values) compulsory must be
  collected for some operations, eg hip
  replacements, hernia, cataracts.
• How do we measure outcomes more comprehensively
• Opinions?

37
Scales

• HUI
• Good discrimination,
• Not as widely used
• Cost involved

38
Scales

• Euroqol
• Only useful for people who are ill
• Surveys show majority of people come out as
  perfect health
• Simple to use
• Quick
• Cheap

39
Scales

• SF36 and Brazier algorithm
• Very widely used
• Cost, but modest
• Slightly technical to work out
• Can analyse retrospective data

40
Reduction Mammaplasty

• Handout
• Discussion of paper
• Flaws
• Strengths
• Use of SF36 and Brazier algorithm

41
Exercises

• Now we know what a QALY is
• We know what the cost is
• QALY Exercises (if time) on handout
• If no time look at later

42
Food for thought

• Deaths brought forward in environmental
  literature
• What about deaths delayed?

43
Break?
44
Different Perspectives

• Already identified two main perspectives in
  economic evaluation societal and nhs payer
  perspective
• Evaluators want to measure other things other
  than the pure impact of health interventions on
  health related quality of life.

45
Patient Satisfaction

• What satisfies patients
• Are patients good judge of what they want
• be nice to them and give a crap service
• Doctors there to amuse us as we get better of our
  own accord (Voltaire)
• Questions often on scale asking how satisfied
  were you with
• Scales 4 or 5 point can be analysed using
  logistic regression.
• What about expectations? Isnt satisfaction
  rooted in changing expectations.

46
Qualitative Outcomes

• Helen?
• Many sophisticated methods available
• Action Research - improving services
• Social Marketing
• 4 ps (5 for us)
• Product
• Price
• Place
• Promotion
• Policy

47
Evaluative Culture

• Evaluative Public Health
• Muir grey Handout
• Need to use the tools we have to continually
  evaluate new and existing public health
  interventions
• NIHCE can help, more coming out and most say that
  public health interventions such as smoking
  cessation and weight management are very
  cost-effective.
• Often wider benefits
• Handout
• Stay Healthy Live Longer Evaluation

48
Qualitative

• Focus Groups
• Reflector Groups
• Delphi methods
• Inductive vs Deductive Methodology
• Virtuous circle

49
Evaluation

• Decide on what outcomes you want to measure
• Decide when you want to measure them

First presentation/diagnosis
At start of intervention
After Intervention
Follow-up
50
Evaluation

• Smoking Cessation? Is evaluation satisfactory?
• Weight management initiatives - Is evaluation
  satisfactory?
• What about cost per quitter?
• Cost per kilo lost?

51
Example

• Stay Healthy Live Longer
• SF36
• Weight Lost (LAA targets)
• Blood pressure
• Other outcomes

52
Evaluation

• Outcomes often wider than just weight loss
• Mental health
• Social health
• Wider picture will it always be important/
• Perspective again.

53
Audit

• (Or research)
• Should still consider impact on people
• Audit checks that what is done is ok
• Doesnt always consider outcomes to patients,
  merely intermediate measures such as processes.
• Need to go further with PH service provision
• Incomplete contracts?
• Good SLAs important
• Contestability important

54
Audit

• 21 Clinical Pathways
• How to Audit?
• Clear criteria
• Element of surprise keeps others on their toes.

55
Evidence Based Management

• Muir Grey says we dont consider the impact of
  management decisions

56
Evidence Based Evaluation
Identify Topic
Evidence
Set Standards Or Guidelines
Re-Audit to make sure change effective
Implement Change
Measure Practice Against guidelines
Identify change needed
57
Overall

• Aim is to improve quality
• Give patients what they need
• Give patients what they want
• Bring about good health
• Bring about an end to Eminence Based medicine

58
(No Transcript)
59
Decision Making in Pathways Development

• Dr Andrew J Taylor
• Assistant Director Public Health Science, Hull
  Teaching PCT

60
Slides On Website

• http//www.hullpublichealth.org

61
Decision Making in Pathways Development

• Objectives
• Describe an Economic view of Decision Making in
  Health Care
• Offer a potential approach
• Describe simple approach
• Pull together main recommendations

62
Economists Perspective

• More Demands on resources than there are
  resources available
• A view not always accepted by participants in NHS
• Patients needs must come first!
• Which patients needs come first?
• Cardiac Pathway ?
• Cancer Pathway ?

63
Economists Perspective2 key economic principles

• Opportunity Cost
• Investing resources in one way or on one group of
  patients means that resources not available for
  another

• The Margin
• About changes to the resource mix
• If budget increases how to spend new resources
• If budget decreases where to reduce resources
• If budget stays constant how to reconfigure for
  improved service

64
Marginal Analysis

• Aim to allocate resources to the point when no
  further change will result in a health gain
• Without explicit adherence to the two economic
  principles, resources will be unlikely to be
  allocated in the best possible configuration.
• Opportunity Cost - The Margin

65
Program Budgeting Marginal Analysis (PBMA)

• Approach used in UK Australia, New Zealand and
  Canada
• Hull is participant in National PBMA Pilot with
  York Observatory - Diabetes
• Similar principles to economic evaluation but
  suggested to be more pragmatic and applicable to
  various levels through organisation
• Key is to ensure that opportunity cost and the
  margin are used in decision making process

66
PBMA Steps

• Determine aim and scope of decision process
• Define budget (map current activity and spending)
• Form marginal analysis advisory panel
  (representative)
• Determine locally relevant decision criteria
• input from decision makers, board, public, etc
• Advisory Panel identify
• Areas for service growth
• Areas for savings through using less resources
  for same service level
• Areas for resource release through scaling back /
  stopping services
• Advisory Panel make recommendations in terms of
• Funding growth areas from new resources
• Moving resources from 5b to 5a
• Trade off decisions to move resources from 5c to
  5a if considered to be an improvement
• Validity checks with stakeholders

67
Determine Scope

• Aims need to be clearly stated
• Might be
• To determine optimum investment across all
  programs within Trust (PCT or Provider)
• Or
• To identify priorities for new technologies
• Both important for Pathways Development
• Map current activity and spending
• Starting point for decisions

68
Advisory Panel (Pathways Group)

• Representative group but not too large
• Group dependent on question under consideration
  and scope of exercise
• Necessary to restate economic principles

69
Decision Making Criteria

• Need to clearly state dependent criteria and make
  explicit (some examples)
• Cost-effectiveness/Clinical efficacy
• Deprivation/Equity
• Innovation
• Ethics
• Political Imperatives
• May be put into a decision making pro-forma to
  ensure fair consideration and avoid the loudest
  voice dominating
• Or use Qualitative (Delphi?) methods

70
Identify options for change

• Stage 5 in the PBMA process
• Prioritised list of service growth options
• Examine current expenditure to improve
  operational efficiency
• Identify options for service reductions or
  scaling back

71
Rating Options for Change

• Part 6 in the PBMA process
• Each planning group member to explicitly rate
  each option from stage 5 according to the
  pre-defined criteria
• All decision matrix forms summed and total score
  averaged to give a score per option51121

72
Ranking and Decisions

• If, for example service growth items A, B and C
  get scores of 900, 800 and 700 and service
  reduction options X, Y and Z get average scores
  of 850, 750 and 650 respectively
• Ranking of options is then A X B Y C Z
• Service A should be financed from lowest ranked
  service reduction (Z) still getting funding
• Continue comparing growth and service reduction
  until no further gains can be had by switching
  resources between options

73
Why and What Challenges

• Notion of freeing resources through improving
  operational efficiency commonplace in health
  organisations
• But Still
• Challenge to obtain resource releases through
  service reductions / disinvestments
• Need good information

74
Information and PBMA

• Economic Evaluations
• Systematic Overviews
• Outcome studies
• Health technology assessments
• Quality reports
• DoH or other guidelines
• Local utilisation data
• Informal input
• Expert opinion
• Business cases
• Etc as relevant

75
Other Challenges

• Data Requirements - Cost of Information
• Need to make decisions when there is little
  evidence (Bounded rationality)
• Identifying areas for resource release (sacred
  cows and protection of interests)
• Might be better approached by informal
  consultation with key personnel or by anonymous
  questionnaire
• How to incorporate patient views without
  distorting a rational or dispassionate view

76
Barriers and Facilitators

• Barriers
• Lack of trust between facilitators
• Physicians not on board
• Advisory panel lacking in health economics and/or
  allocation experience
• Politics preventing program evolution
• Discontinuity of personnel
• Too many admin demands leaving priority setting
  as a low priority activity

• Facilitators
• Senior level managerial/clinical champions
• Strong leadership
• Culture to learn and change
• Integrated budgets
• Resources earmarked for PBMA process itself
• Built in incentives for appropriate and efficient
  spending

77
Advantages of PBMA

• Explicit recognition of opportunity cost and the
  margin
• Transparent process
• Stakeholder consultation
• Enables public input
• Can be coupled with an ethical framework to
  examine the fairness of the process
• Improvement on common unstructured decision
  processes

78
Decision Tree Approach
Square Decision Node Circle Chance Node with
probability
79
Decision Trees1 Year Timescale in this case for
simplicity
Die
A
Live
Decision
Die
B
Live
80
Decision Trees ProbabilitiesMust Sum to 1
Die
0.05
A
Live
0.95
Decision
Die
0.1
B
Live
0.9
81
Decision Trees OutcomesMeasured using HUI SF36
Euroqol etc
Die
0
0.05
A
Live
0.85
0.95
Decision
Die
0
0.1
B
Live
0.80
0.9
82
Decision Trees CostImportant in every Pathway
Die
0
0.05
A
Live
0.85
0.95
1000
Decision
Die
0
0.1
100
B
Live
0.80
0.9
83
Decision Trees OutcomesMeasured using HUI SF36
Euroqol etc
Die
0
0.05
Outcome 0.81
A
Live
0.85
0.95
1000
new treatment
Decision
Die
0
Existing treatment
0.1
100
B
Live
0.80
Outcome 0.72
0.9
84
Decision Trees OutcomesMeasured using HUI SF36
Euroqol etc
Die
0
0.05
Outcome 0.81
A
Live
0.85
0.95
1000
Expensive Treatment
Decision
Cheap Treatment
Die
0
0.1
100
B
Live
0.80
Outcome 0.72
0.9
85
Decision Trees OutcomesMeasured using HUI SF36
Euroqol etc
Die
0
0.05
Outcome 0.81
A
Live
0.85
0.95
1000
Expensive Treatment
1 Patient Or 1000 ?
Decision
Die
0
Cheap Treatment
0.1
100
B
Live
0.80
Outcome 0.72
0.9
86
What Next??

• Difference in Health Related Quality of Life
  Outcome 0.09 on scale (between 0 death and 1 full
  health
• Cost difference 900
• Cost per QALY ?
• 900 for 0.09 improvement in Quality of life
• 9,999 cost per QALY
• ????? NIHCE Boundary 30,000 per QALY
• 9999.9999999

87
What Next?? Whats important?

• PBMA - Allows consideration and input of
  pathway for all members of group avoid dominance
  by any group
• Finite Resources Probably shrinking in real
  terms due to increases in costs
• Evidence Based From Studies and Measured Local
  Outcomes
• Costed Essential, but not always considered but
  relatively easy?
• Gold Standard, Silver Standard or Bronze ? -
  Based on Maximisation of benefit to patients

88
What Next?? Whats important?

• Cost of Information Not always possible to meet
  all the important points but should attempt to
  find ways where we can
• Innovation and Change? Can it be accommodated
  in a tariff based system?
• Others?

89

• References
• Farrar S, Ryan M, Ross D, Ludbrook A Using
  discrete choice modelling in priority setting an
  application to clinical service
  developments.Social
• Science and Medicine 2000, 5063-75.
• Lomas J, Woods J, Veenstra G Devolving authority
  for health care in Canada's provinces 1. An
  introduction to the issues.Canadian Medical
  Association Journal 1997, 156(3)371-377.
• Mitton C, Donaldson C Setting priorities in
  Canadian regional health authorities a survey of
  key decision makers.Health Policy 2002,
  60(1)39-58.
• Birch S, Chambers S To each according to need a
  community-based approach to allocating health
  care resources.Canadian Medical Association
  Journal 1993, 149607-612.
•  Donaldson C, Mitton C, Currie G Managing
  Medicare the pre-requisite to spending or
  reform.The Health Papers No. 157. Toronto CD
  Howe Institute 2002.  
• Mitton C, Donaldson C Tools of the trade a
  comparative analysis of approaches to priority
  setting in health care.Health Services
  Management Research 2003, 1696-105.
•  Mooney G, Gerard K, Donaldson C, Farrar S
  Priority Setting in Purchasing Some Practical
  Guidelines.(Research paper number 6) Scotland
  National Association of Health Authorities and
  Trusts 1992.
• Cohen D Marginal analysis in practice an
  alternative to needs assessment for contracting
  health care.British Medical Journal 1994,
  309781-785.
• Shackley P, Ryan M Involving consumers in health
  care decision making.Health Care Analysis 1995,
  3(3)196-204.
• Mabin V, King G, Menzies M, Joyce K Public
  sector priority setting using decision support
  tools.Australian Journal of Public
  Administration 2001, 60(2)44-59.  
•  

90

• References
• Astley J, Wake-Dyster W Evidence-based priority
  setting.
• Australian Health Review 2001, 24(2)32-39.
• Peacock S An Evaluation of Program Budgeting and
  Marginal Analysis Applied in South Australian
  Hospitals.Melbourne Center for Health Program
  Evaluation, Monash University 1998.
• Daniels N Accountability for reasonableness.Brit
  ish Medical Journal 2000, 321(7272)1300-1.
• Singer P, Martin D, Giacomini M, Purdy L
  Priority setting for new technologies in
  medicine qualitative case study.British Medical
  Journal 2000, 3211316-1318.
• Ruta DA, Donaldson C, Gilray I Economics, public
  health and health care purchasing the Tayside
  experience of programme budgeting and marginal
  analysis.
• J Health Serv Res Policy 1996, 1(4)185-193.
  Donaldson C, Mooney G Needs assessment, priority
  setting, and contracts for health care an
  economic view.British Medical Journal 1991,
  3031529-1530.
• McIver S, Baines D, Ham C, McLeod H Setting
  Priorities and managing demand in the
  NHS.Birmingham Health Services Management
  Centre 2001.
•  Ratcliffe J, Donaldson C, Macphee S Programme
  budgeting and marginal analysis a case study of
  maternity services.Journal of Public Health
  medicine 1996, 18(2)175-182.  
•  Litva A, Coast J, Donovan J, Eyles J, Shepherd
  M, Tacchi J, Abelson J, Morgan K 'The public is
  too subjective' public involvement at different
  levels of health-care decision making.Social
  Science and Medicine 2002, 541825-37.
•  Mitton C, Donaldson C Setting priorities and
  allocating resources in health regions lessons
  from a project evaluating program budgeting and
  marginal analysis (PBMA).Health Policy 2003,
  64335-348.