Developing Public Health Performance Measures: Benefits and Challenges

1
Developing Public Health Performance Measures
Benefits and Challenges

• Mare Schumacher
• Maricopa County Department of Public Health
• APHA Meetings October 2001

2
About Us

• Maricopa County Department of Public Health in
  Phoenix, AZ
• 440 employees
• 47 distinct programs (from childhood
  immunizations to HIV education)
• 85 grant funded

3
Developing Measures
4
History

• Pre-1998 Collected some metrics (mostly grants)
• 1998-99 Dept. leaders request collection of
  measures for all programs
• 2000 Countywide initiative
• 2000-01 Complete family of measures for all 47
  programs

5
Developing Measures

• One employee dedicated to process coordination
  (about ½ time)
• Trained managers/supervisors on
• Types of pms (e.g. output, outcome)
• Proper collection methods
• Proper documentation

6
Developing Measures

• Supervisors sat down with staff to determine
  goals
• When needed, coordinator worked individually with
  programs
• Biggest challenge the psychology

7
Stages of PM Process

• Denial This is flavor of the month. If I
  ignore it, it will go away shortly.
• Anger I havent got time for this!
• Panic/Freeze PMA (Performance Measure Anxiety)
• Depression I cant wait to retire.
• Acceptance We have to do this.

Apologies to Elisabeth Kubler-Ross
8
Two Sales Options
Cheerleader Model Telling employees how
super-fantastic these performance measures will
be!!!
Empathy (I Feel Your Pain) Model Acknowledging
hard work that will be required by employees.
Helping them through difficulties.
9
Collection and Maintenance

• Maintain PMs in central document/ electronic
  location w/gatekeeper
• Important to keep narrative explaining
  significant shifts in numbers
• Updated semi-annually, now quarterly

10
Example Immunizations

• Result Coverage level - of clients receiving
  entire series
• Output of immunizations, of clients, of
  clients 0-2 years old
• Efficiency Cost perimmunization

11
Example Nutrition Education

• Result Increase in knowledge, attitudes,
  behavioral intent from pre- to post-test

• Output of students
• Efficiency Cost per student

12
Using and Abusing Measures
13
Philosophy
Set Goals
Funding Decisions (Accountability)
Run Program
Measure and Analyze Program Results
14
Benefits

• Long term resource planning
• Staff
• Budget
• A goal to shoot for, rally around
• Sticking with priorities are we using our
  energies for our top priority?

15
Do we really measure performance?
16
Performance Measurement Experiment
County X
Maricopa County
Goal for Fiscal Year 01
Goal for Fiscal Year 01
25
25
Actual Performance
Actual Performance
17
Performance Measurement Experiment
At this point in the presentation, two people
representing each of the counties must fill a
cup with marbles. Maricopa County uses a spoon
and County X has to use chopsticks. The result
is that County X scores lower on its performance
measure. Without knowing that County X had
chopsticks, we might conclude that it exhibits
poor performance.
County X
Maricopa County
Goal for Fiscal Year 01
Goal for Fiscal Year 01
25
25
Actual Performance
Actual Performance
18
The Chopstick Effect

• Funding
• Staff shortages
• Change in grant requirements
• Marlboro has big ad/promotion
• And so on

19
Issues

• May not measure program effort or potential
  effectiveness
• Widget Factory model often not applicable in PH
• Cost of collecting data some-times not worth it
• Goal setting as art

20
Alternatives

• Process evaluation
• Up front research (usually from Academia)
• Statistical modeling
• Standard research practicesTest/Control, etc.
• National Public Health Performance Standards
  (self-assessment)

21
Conclusions

• You can develop measures for your organization
• But dont forget the psychological component.
• Use performance measures wisely better for
  planning than evaluating.
• Lets find other ways to accurately determine
  success (for true accountability).

22
Information

• Mare SchumacherMaricopa County Department of
  PH1845 E. Roosevelt, Phoenix, AZ 85006
• Mareschumacher_at_mail.maricopa.gov
• Resource Taking Stock of Performance
  Measurement Information Resources for Public
  Managers, Government Finance Review, April 2001
• Resource Performance Measures Basics
  (presentation for program staff). Send e-mail to
  Mare Schumacher for copy.

23
The following is a presentation I used to educate
program staff on performance measures at our
Department. (It was not presented at the APHA.)
Feel free to use it or alter it for your own
purposes.I ask nothing in return except that you
think of me for a moment every time you use it

• Mare Schumacher
• Department of Public Health
• Maricopa County, Arizona (Phoenix)

24
Performance Measures

• Presentation to Public Health
• Strategic Planning Team
• March 1999

25
The Past

• The night before due, make up s to put in as
  performance measures
• No one looks at them
• No one checks to see if you measured
• No one thinks about it for 11 months and 27 days
• Next year, repeat process

26
The Future

• County Administration will be looking closely at
  performance measures
• Depts./Programs will be judged on performance
  measures
• Budget may be tied to performance measures

27
Making the Best of It
1.
2.

• Weve got two choices

• The night before due, make up s
• Live with consequences

• Come up with performance measures that are
  meaningful to County and you

28
Suggested Process

• May be other options...

29
Meet with Staff

• In a meeting with all or many staff
• Helps set realistic goals
• Employees know what is expected of them

What gets measured gets done. -- Rose Howe,
December 1998
30
Decide Program Mission

• Why do we exist?
• What contribution to we make to the people of MC?
• NOT What do we do?

31
Example HIV Planning

• What we do
• Have meetings
• Contact coalition members
• Prepare reports
• Get trained in how to build teams

• Why we exist
• To create a comprehensive HIV prevention plan
  that will reduce the spread of HIV

32
Just talk this

• Were not measuring what we do
• We are measuring our contribution to the public
  health
• Doesnt mean what we do is not important, just
  not the thing were measuring

33
Create Output Measurement

• Output measurement
• Activities
• Things we do

• Examples
• Number of applications processed
• Number of clients
• Number of immunizations
• Number of coalition meetings

34
Example Immunizations

• Conducted 7500 immunizations in 97/98, etc.
• Expect to administer 9500 in 99/00
• Reasons Population growing by 5, will have
  increase in staff in 99/00

All numbers made up -- demonstration purposes
only
35
Create Efficiency Measurement

• Efficiency measurement
• Cost per output
• Resources used to produce the activity

• Examples
• Cost per client
• Time spent per client
• Cost per client compared to certain standard(s)

36
Example Immunizations

• Clients per hour 4 in 97/98
• Expect 5 by 2000/01 due to new methods
• Cost per client 1.22
• Expect cost 1.45 by 00/01 due to increase cost
  of vaccine.

37
Create Impact (Outcome) Measurement

• Impact (outcome) measurement
• Indicate the impact of service on the community
• Should flow directly from mission

• Examples
• Percentage of clients rehabilitated
• Reduction in incidence of disease

38
Example Immunizations

• 90 coverage of school-aged children by 00/01
• 57 coverage of clients at public health
  immunization sites
• Possibly 400 cases of measles avoided

39
All Measures

• What data will we use to measure it?
• If none available now, how will we get it?

40
Goals (Future Years)

• Make them specific
• Make them realistic - how much control do we have
  over that measure?
• What are the reasons for expecting the goal to be
  higher, lower, etc.
• Document why you picked that goal? How did you
  calculate it?

41
Timing

• Develop measures with staff
• Draft meeting with John/Mare
• Final draft meeting
• Final measures due

• End of March/April
• End of April (sign-up sheet)
• May 11 SP Meeting
• June 1