Performance Evaluation in the Healthcare

1
Performance Evaluation in the Healthcare

• Heesuk Yun
• Korea Development Institute
• March 17, 2004

2
Need for Performance Evaluation
in the Health Field

• Expenditure for public health is rapidly
  increasing in Korea
• Targets of public health actions is expanding
  beyond infectious diseases
• Increasing emphasis on prevention, but efficacy 
  is hard to prove
• Pressure of cost containment and Importance of
  accountability of resources invested in health
  field
• The need for rigorous evaluation will increase

3
This paper attempts to

• Summarize the essential elements and special
  characteristics of performance evaluation in the
  health field
• Introduce the U.S. experience of devising
  indicators in the health field
• Review standards for effective evaluation of
  health interventions
• Extract  implications for establishing a
  performance evaluation in the health field in
  Korea

4
Objectives of Public Health Intervention

• Multiplicity objectives range from the most
  general such as reducing mortality to the very
  specific such as reading a health pamphlet.
• Objectives and sub-objectives can be hypothesized
  corresponding to the steps or actions of a
  program.
• A continuous series of events ? divided into a
  hierarchy of sub-goals for evaluation purposes
  Each is  the result of successfully achieving the
  preceding goal, and a precondition to the next
  higher goal.

5
Cumulative chain of objectives
6
Needs to prove the intervening
assumptions through careful research

• An assumption needed  whenever one moves from a
  higher-order objective to a lower one. Every
  lower-level objective must assume all of the
  assumptions made for all of the objectives above
  it.
• Any program based on a false set of assumptions
  cannot be justified, even if sound evaluations
  are completed for each of the other objectives
  individually.

7
Special Characteristics of Performance
Evaluation in the Health Field
Scientific rigor

• Intervention logic an explanation of what the
  public action is supposed to achieve and how it
  is supposed to achieve it
• Causal assumptions are hidden in the logic
• Involves scientific theory  
• Identifying the hidden assumptions and
  investigating their uncertainties are more
  complicating

8
Smoking and Low Birth Weight
9
Addictive behaviors such as smoking and
alcohol consumption have
direct effect on low birth weight

• Economic stress has direct effects on family
  stress and social support
• Family stress and social support have direct
  effects on addictive behavior
• The three stress variables have indirect effects
  on low birth weight
• The standardized regression coefficients indicate
  the size of the change from various factors

10
Special Characteristics of Performance
Evaluation in the Health Field
Contextual Complexity

• Complicated contexts including historical,
  geographical settings, political, social and
  economic conditions, and influences of related or
  competing organizations
• Need to consider the needs of the target
  population and the particular problems
• Difficult to identify what effects are genuinely
  caused by a program and to separate these effects
  from other influences on the socio-economic
  problems
• Needs to understand these intertwined factors, to
  design a context-sensitive evaluation, to
  interpret findings accurately and to assess the
  generalizability of the findings.

11
Political Context

• Health Care interventions are often created to
  reduce health disparities across socio-economic
  groups
• Evaluations must consider political aspects among
  income groups, politicians, and interest groups
• Evaluations must be conducted in a political
  context where groups compete in their own
  interests.

12
Methods in Performance Evaluation for
Health Care Intervention Customizing
Tools for Evaluation

• Health care's  special features due to scientific
  hypotheses or the contextual particularity
• Need to understand the intervention specific
  components and process
• Golden rule there is no single evaluation
  methodology which is universally applicable for
  the entire health field. The choice of method
  determined by the particular evaluation problems

13
Prerequisites for Starting
Evaluation Infrastructure

• (1) Construction of Representative
  Frameworks for Evaluating Public Healthcare
  Intervention
• Standardized framework supports a practical
  approach based on steps and standards applicable
  in public health settings.
• Provides a guide for designing and conducting
  specific evaluation projects
• Can be used as a template to create or enhance
  program-specific evaluation guidelines that
  further operationalize the step sand standards in
  ways that are appropriate for each intervention

14
Guided by the steps and standards
in the framework

• Program planning will also evolve
• Integrated information systems will support a
  more systematic measurement
• Lessons learned from evaluations can be used more
  effectively to guide changes in public health
  strategies.
• General framework can encourage evaluation
  research to be integrated with routine program
  operations.

15
Prerequisites for Starting

• (2) Constructing Indicators for Major
  Healthcare Evaluation Types
• System of criteria to measure the effects of an
  intervention
• Where intervention logic is grounded on
  scientific knowledge and all the political,
  social, and economic interests, Indicators should
  function as the guiding rules
• Should translate general concepts regarding the
  intervention, its context, and its expected
  effects into specific measures that can be
  interpreted, while providing a basis for
  collecting evidence

16
Developing indicators should be based
on intervention logic

• For each step of the model, qualitative and
  quantitative indicators  developed to suit the
  concept in question, linking assumption, the
  information available, and the planned usage of
  data.
• By relating indicators to intervention logic,
• Detect changes in performance faster than when
  relying on a single outcome as the only
  performance measure
• Lines of responsibility and accountability are
  clarified as the measures are aligned with each
  step of the program strategy
• Detect the consequences of intermediate effects
  on health outcomes of the program.

17
Prerequisites for Starting

• (3) Preparing Date Sources for Evaluation
• When an indicator is proposed for use in
  performance monitoring, appropriate data must be
  available to support an indicator
• Data should be collected from the specific
  population of interest, within the relevant time
  frame, using valid, reliable, and responsive
  measures
• Often, performance evaluations have to rely on
  data collected for another purpose
• Evaluators must understand limitations on the
  applicability of the data.

18
Examining Indicators for Healthcare
Evaluation Experience of U.S.

• Panel on Performance Measures and Data for Public
  Health Performance Partnership Grants
• Recommend performance measures in ten areas
  chronic disease sexually transmitted disease
  (STD), human immunodeficiency virus (HIV)
  infection, and tuberculosis mental health
  immunization substance abuse  sexual assault,
  disabilities, and emergency medical services

19
Panel's Framework for Assessing
Suggested Indicators

• Health outcome measures are widely used but
  insufficient
• Many measures of health outcomes are affected by
  various factors that are not under the health
  intervention's control ? changes in outcomes
  cannot be attributed only to specific program
  effectiveness.
• Many important public health objectives, such as
  lowering the incidence of cancer and HIV
  infection, cannot be achieved over short periods
  of time to derive an outcome measure.

20
Health outcome change in the health of
a defined population related to an
intervention.

• Risk status (intermediate outcome) change in the
  risk demonstrated or assumed to be associated
  with health status.
• Process what is done for the defined groups as
  part of the delivery of services, such as
  performing a test or procedure or offering an
  educational service.
• Capacity the ability to provide specific
  services, such as clinical screening and disease
  surveillance

21
Example CHRONIC DISEALSE

• Prevention of chronic disease is the primary goal
  of many health programs,
• But chronic disease incidence and mortality data
  are not useful because
• The expected time period between  prevention
  activities and the effect exceeds the time that
  health departments are generally willing to wait
  to assess the intervention's effectiveness
• Potential chronic disease measures are focused on
  risk reduction and screening, and process
  measures.

22
Potential Risk Status Measure

• Percentage of (a) persons aged 18-24 and (b)
  persons aged 25 and older currently smoking
  tobacco
• Who eat five or more servings of fruits and
  vegetables per day
• Who do not engage in physical activity or
  exercise
• Who had their blood pressure checked within past
  2 years

23
Examples of Process Measure

• Nutrition Program Strategy Percentage of schools
  with menus that meet dietary guidelines
• Physical Activity Program Strategy Percentage of
  worksites with worksite wellness programs
• Smoking Program Strategy Percentage of vendors
  who illegally sell smoking tobacco to minors
• Screening Program Strategy Percentage of persons
  with diabetes receiving diabetes health education

24
Examples of Capacity Measures

• Resources Number of full-time health department
  employees for chronic disease prevention
• Proficiencies Number of key surveillance systems
  and data sets that are establish and maintained
• Planning Percentage of population served by
  systematic community planning process, with
  leadership provided by the official health agency
  
• Community Involvement Proportion of health
  department programs that operate within the
  framework of a community coalition or have a
  community advisory group

25
Summary

• Performance evaluation in the health field is an
  increasingly important area
• Evaluation of public health action use of
  rigorous scientific methods and contextual
  complexity requiring socio-economic and political
  awareness.
• Setting up an evaluation system requires building
  infrastructure
• 1) Framework for evaluating public health
  intervention needs to be devised 
• 2) Indicators need to be collected and assessed
  for future use
• 3) Appropriate data sources for each indicator
  should be gathered and organized. and following
  up data collection should be made within a
  reasonable time.