Performance Measurement: Making it Work

1
Performance Measurement Making it Work

• Sponsored by Office of Training
• Conducted by
• Bureau of the Budget
• State Planning Office
• June 20, 2001

2
Workshop Developed by CAP

• Established by American Society for Public
  Administration
• Mission to improve the practice of public service
  by helping public administrators to acquire
  knowledge, skills, and resources to manage for
  results

3
Workshop Objectives

• Developed for state government employees with a
  basic understanding of performance government
  concepts
• Participants will understand the elements of an
  overall performance measurement system
• Participants will learn the importance of
  managing the performance measurement system
• Participants will be provided with a realistic
  approach to performance measurement

4
Todays Agenda

• Morning
• Welcome Introductions
• Setting the Right Stage
• Avoiding Common Pitfalls

• Afternoon
• Managing Performance Data
• Elected Officials use of Performance Data
• Wrap-up Evaluations

Lunch on your own
5
Introductions

• Who are you?
• What is your involvement with performance
  measurement?
• What is the one most important thing you want to
  learn today?

6
Exercise 1

• So What?

7
Setting the Right Stage

• Performance measures must be aligned to a
  strategic framework

8
Setting the Right Stage

• Clarify Public Purposes
• Align Measures to Goals and Objectives
• Hierarchy of Measures Distinguish between policy
  and operations

9
Clarify Public Purposes

• Performance measures inform government
  decision-makers about how well they are achieving
  their public purposes.

10
Clarify Public Purposes

• What is legislative intent?
• What problem(s) are you intended to address?
• What benefits do you provide to citizens?

11
Clarify Public Purposes

• Enabling statutes and amendments
• Constitution
• Legislative History
• Grant Requirements
• Local/State/Federal Requirements
• Stakeholders

12
Align Measures to Goals and Objectives

• A Strategic Plan
• Translates the agencys public purpose into
  mission and goals
• Defines what the agency wants to achieve
• Defines the outcome the agency should focus upon
• Helps the agency make decisions consistent with
  its public purposes

13
Avoid this Situation

• Theres no sense in developing a performance
  measurement system if you dont know what you
  want to accomplish

No, but were making good time!
Do you know where were going?
14
Giving Measures a Strategic Context

• Provide basis for selecting measures
• Make sure progress is related to public purpose
• Prevent measuring for sake of measuring
• Prevent mission drift
• Make measures meaningful to policy-makers and
  citizens

15
Example Align Measures to Goals and Objectives

• Performance Measures for Youth Recreation
  Services
• Percent of school-age youth participating in
  community recreation programs
• Satisfaction rate of parents and youth with
  programs
• Avg. cost per program participant

Measures lack a strategic context
16
Example Align Measures to Goals and Objectives

• What if survey results show that parents are
  dissatisfied with community recreation programs
  because there are not enough programs for adults.

What might the agency do?
17
Example Giving Measures a Strategic Context

• Goal Ensure that the young people in our
  community thrive.
• Objective Reduce the percentage of school-aged
  youth who participate in dangerous activities.

• Percent of school-age youth participating in
  community recreation programs
• Satisfaction rate of parents and youth with
  programs
• Avg. cost per program participant

Do the measures relate to the agencys public
purpose?
18
Example Giving Measures a Strategic Context

• Revised Performance Measures for Youth Recreation
  Services
• Juvenile Crime Rate
• Crime rate of youth participating in recreation
  programs
• Percent of school-aged youth who say they have
  nothing to do after school
• Percent of school-age youth participating in
  community recreation programs

19
Most Commonly Used Measures

• Output - What is the activity or goods/services?
• Efficiency - How are resources used? What is the
  cost per unit of output?
• Service - What do customers think?
• Outcome - What has changed for the customer?

20
Levels of Outcome

• Short-term outcome
• Intermediate outcome
• Long-term outcome

• Number of people that quit smoking in response to
  agency campaign
• Percent of people in our state who smoke
• Percent of people hospitalized with
  smoking-related diseases

21
Hierarchy of Measures So-That Chain
Ultimate Goal
So that...
Output
So that
Short-term Outcome
Degree of Agency Influence
Long-term Outcome
Source State of Washington, Public Knowledge,
Inc.
22
Hierarchy of Measures

• Performance measures inform decision-making at
  all levels.
• A good performance measurement system tracks both
  policy and programmatic measurement data, but
  recognizes their different uses within and
  external to the agency.

23
Hierarchy of Measures
Policy-makers
Managers
Getting to Our Outcomes
24
Hierarchy of Measures

• Policy/Strategic Is the program achieving its
  public purpose?
• Goals and objectives define your long-term and
  intermediate outcomes
• Program/Operational Is the agency performing
  well?
• Performance measures track your short-term
  outcomes and outputs

25
Example Hierarchy of Measures within an
Organization
Goal Enhance the safety of all citizens and
visitors engaged in outdoor recreational
activities.

• Individual Employee
• Articulate contributions to goals and objectives
• Employee Performance Expectation
• Percent of trainees who rate the training as
  helping them to hunt more safely

• Agency or
• Department
• Improve program performance
• Short-term Outcome
• Number of hunter safety violations per licensed
  hunter

• Program Unit or External Vendor
• Make staffing and operational decisions
• Output Measure
• Number of licensed hunters receiving safety
  training

• Organization or Community-wide
• Informs Policy Dialogue
• Intermediate Outcome
• Hunting accident rate per licensed hunter

26
Example Program LogicFlorida Fire Marshal
Long-term Outcome Reduce Losses from Fires
Factors Contributing to Fires Intermediate
Outcomes
Inadequately Trained Emergency Personnel
Arson
Building Defects
Investigations
Building Codes
Public Education
Certification
Program Interventions Outputs
Fire College
Prosecution
Expert witness
Emergency Response Exercises
Research
27
Exercise 2

• Hierarchy of Measures Develop a Program Design
  Model
• Identify Program Outputs/Outcomes

28
Avoiding Common Pitfalls

• Considerations in Selecting Performance Measures

29
Considerations in Selecting Measures

• You can measure outcomes (and you should!)
• Provide a family of measures
• Select a few, key measures
• Watch out for unintended consequences

30
You Can Measure Outcomes

• Distinguish between long-term trends/indicators
  and short-term accountability
• Rate the degree of agencys influence on the
  outcome
• Provide a range of values
• Include explanatory information
• Remember --performance measurement is not program
  evaluation

31
Example You Can Measure Outcomes
Outcome Reduce rabies infection in people

• Possible Performance Measures
• Percent of rabies infection (reported incidences)
  in areas where public education program is
  conducted vs. in areas where there is no
  education program
• Percent of pet owners who believe the agency is
  effective at controlling rabies
• Rate of rabies in wild animals (our agency
  exercises less than 50 control over this
  outcome)
• Rate of rabies in domestic animals in our
  community (explanatory measure rate of rabies in
  domestic animals statewide)

32
A Family of Measures

• No single measure can capture an entire program
• Using only one type of measure is inappropriate
• Outputs are often only explanatory
• Improved performance may require trade-offs

33
Example A Family of Measures

• Goal Ensure the safety of our citizens
• Objective Reduce the number of fire-related
  deaths.
• Performance Measures
• Number of fires investigated
• Number of assists to fire/police agencies
• Number of safety programs offered

Are the measures useful?
34
Example A Family of Measures

• Goal Ensure the safety of our citizens
• Objective Reduce the number of fire-related
  deaths.

What would you want to know?
35
Example A Family of Measures

• Percent of fires caused by building defects
  (intermediate outcome)
• Percent of fires in houses with building defects
  compared to fires in houses that meet fire codes
  (intermediate outcome)
• Percent of buildings that meet fire codes
  (short-term outcome)
• Number of buildings inspected by the Office
  (output)
• Avg. number of staff hours per fire inspection
  (productivity/efficiency)
• Satisfaction rate of homeowners with inspection
  process (service)

36
Select a Few, Key Measures

• Three Tests
• The Policy Test Are we achieving our stated
  public purpose?
• The Communication Test How well are we doing?
• The Data Test Are performance data accessible
  and economical?

37
The Policy Test

• If we perform well in areas being measured, can
  we reasonably expect to make progress toward the
  agencys goals and objectives?
• Are the measures important to policy-makers and
  citizens?
• Are the measures related to a major spending
  category in the budget?

38
Example The Policy Test

• Goal Promote traffic safety and civil order
• Objective Increase the percentage of people who
  feel safe.

• Performance Measures
• Number of arrests
• Number of licenses issued
• Number of vehicles inspected

Do the activities being measured contribute to
achieving the goal and objective?
39
Example The Policy Test

• Goal Promote traffic safety and civil order
• Objective Increase the percentage of people who
  feel safe.

• Performance Measures
• clearance rate for criminal offenses
• motor vehicle crash rate per 100,000 miles driven
• average time to respond to calls

40
The Communication Test

• How effective is the program or service?
• What is the impact of our actions?
• What are our key accomplishments?

41
Example The Communication Test

• Goal Ensure clean air
• Objective Increase the percentage of citizens
  that live in areas that meet clean air standards.

• Performance Measures
• Percent of facilities inspected
• Number of permits issued
• Avg. cost per inspection

Do the performance measures tell us how
effective we are?
42
Example The Communication Test

• Performance Measures
• number of ozone exceedance days/year
• tons of pollutants emitted
• compliance rate for inspected facilities

• Goal Ensure clean air
• Objective Increase the percentage of citizens
  that live in areas that meet clean air standards.

43
Data Is it Accessible and Economical?

• Is the data readily available?
• Can it be collected in a cost-effective manner?
• Does data provide sufficient value to warrant the
  cost?

44
Measurement Consequences

• Measures focus attention on whats being measured
• Measures increase visibility of program
  performance
• Measures may cause some cheating or sabotage
• Measures have unintended consequences

45
Example Measurement Consequence
Lets see, I need exactly 39,586.14
GOVERNOR WELD WANTS EX-CONS TO PAY COST OF
PRISON STAY
46
(No Transcript)
47
(No Transcript)
48
Examples Measurement Consequences

• Rate of achievement on standardized tests
• Number of traffic citations written
• Average sale per customer (State-owned Liquor
  Operations)
• Number of complaints received
• Number of clients seen by a case worker

Can you anticipate the unintended consequence?
49
Exercise 3

• Measurement Consequences Spot the Potential
  Problems

50
Managing Performance Data

• Successful performance measurement systems need
  regular, ongoing management attention

51
Erroneous Assumptions

• Performance measurement systems manage themselves
• Computer staff understand program data
• Technical staff will make sure data are accurate
  and complete
• Clerks and interns will correct errors when they
  enter performance information
• Computerized data are error-free
• Management can wait for annual performance
  reports to tell them how things are going

52
Lessons Learned

• Need clearly-defined, useful measures
• Accurate data is essential
• Management oversight should be ongoing
• Frequent reporting (more than once/year) prevents
  surprises
• Staff need to know what, why, and how
• Report honestly, no matter how bad things are
• Data should be used regularly to identify needs
  and improvement opportunities

53
Elected Officials Use of Performance Data

• Are You Ready?

54
Use of Performance Data

• The Budget Process - allocates resources to
  achieve goals and objectives
• Policy-making and Evaluation - ensures that goals
  and objectives are being met

55
The Budget Process

• Increasing the Fact Content

56
Performance Budgeting
57
Characteristics of Performance Budgeting

• Focuses on the quantitative relationship between
  inputs and outputs and outcomes
• Looks at outcomes expected by way of results in
  the years beyond those covered by the proposed
  budget
• Aligns spending with goals and priorities

58
Budgeting to Outputs vs. Outcomes

• Estimating the amount of future outcomes that
  will occur given specific amounts of input is
  much more difficult than tracking past
  performance

• lack of experience
• lack of cost data
• uncertainty about future outcomes

59
Linking Inputs to Performance

• The relationship between inputs and outputs is
  often predictable
• We are likely to be able to only make rough
  projections as to the relationship between inputs
  and intermediate outcomes
• While extremely important, the relationship
  between inputs and long-term outcomes may be best
  considered qualitatively

60
Inputs and Outcomes Qualitative Relationship

• Evaluate the relationship between outputs and
  outcomes - does it make sense?
• Use program evaluation and in-depth studies
• Make plausible connections
• Rely on judgement and policy-makers priorities

61
In the end...

• Performance budgeting, even qualitative
  discussion of outcomes, encourages those making
  budget decisions to focus on what is most
  important to achieve

62
Performance Budgeting Enhances Communication

• Are program investments the most important to
  achieve goals and objectives?
• Where might resources be shifted to better
  achieve the desired results?
• What programs or activities might be pared or
  eliminated?

63
Investment Decision Matrix
64
Legislators Questions

• Why are we providing this service?
• Who benefits from the service?
• Is everyone that is eligible for the service
  receiving it?
• What value do we get for the tax dollars spent?

• Does the benefit to customers and citizens
  outweigh the costs?
• Could the resources provide greater benefit to
  another program?
• Would more money improve results?

65
Preparing for a Legislative Budget Briefing

• Describe why or why not the agency has met its
  performance targets
• Provide a cost-benefit analysis of the program
  funding
• Be prepared to describe the affect of
  greater/reduced funding levels on performance

66
Linking Measures to Part II

• Describe how Part II funds will impact your
  performance targets
• Discuss the incremental improvement that will be
  achieved
• If there is no measurable impact, describe what
  else the funds help you accomplish

67
Activity-based Costing

• Computes costs for activities and activity
  outputs, rather than for line-items
• Provides information about how resources are
  spent, by activity
• Allocates indirect costs to the activity
• Activity-based costing supports performance
  budgeting

68
Policy-making Oversight

• Legislative Oversight
• Auditors Role
• Agency Program Evaluation

69
Legislative Oversight

• Establish policy goals and priorities
• Determine expected levels of performance (set
  targets)
• Allocate resources to achieve desired results
• Evaluate audit results
• Evaluate program accomplishments/impacts
• Compare performance to policy mandates
• Make policy changes for program improvement

70
Legislative Oversight What you want to know

• What are policy expectations of legislators?
• Are there disconnects between policy expectations
  of legislators vs. agency vs stakeholders
• What level of performance is acceptable with the
  funding provided?

71
Government Evaluation Review Act

• Periodic review of agencies
• Evaluate efficiency and performance in carrying
  out legislative mandates
• Agency program evaluation report
• Public review
• Committee findings and recommendations
• 3 MRSA, Sec 956

72
Preparing for a Legislative Evaluation Briefing

• Provide overview of program public purpose
  (goals/obj)
• Describe how program achievements relate to the
  agencys goals and objectives?
• Describe what performance data means
• Explain reasons for performance what else
  impacts performance
• Describe efforts to meet performance targets
• Describe whats on the horizon new strategies,
  possible changes to measures, budget implications

73
Performance Auditing

• Options
• Legislative auditor
• Independent government auditor
• Academia
• Contracted service

74
The Auditors Role

• Verify measures accuracy, validity, and
  reliability
• Verify program performance using performance
  measurement information
• Determine if performance targets were met
• Make recommendations for program improvement
• Issue audit findings to Agency Legislature

75
The Audit Process

• Data collection that supports performance
  measurement
• Independent and objective assessment
• A public report with findings
• Agency response to findings
• Public airing of report
• Agency and/or legislative action

76
A Typical Audit Finding

• Criteria or Performance Measure
• Condition
• Cause
• Effect
• Recommendation

77
Example Audit Finding

• Criteria Buses, on average, shall arrive on time
  96 of the time
• Condition Buses are late 25 of the time
• Cause Frequent bus breakdowns inadequate fleet
  in operation. 3 months behind in maintenance
  schedule. Management did not plan for aging fleet
• Effect Passengers cannot rely on schedule,
  customer service expectations not met
• Recommendation Eliminate maintenance backlog
  establish monitoring system develop replacement
  schedule capital plan

78
Preparing for a Performance Audit

• Document performance measurement definitions
• Document agency processes for data collection
  steps to ensure accuracy
• (In writing), explain how measures are
  calculated, data sources, and any problem with
  data
• Document performance from multiple sources if
  possible
• Be open, forthcoming

79
Not a Substitute for Program Evaluation

• Performance Measures do not tell you
• why performance is at the level it is
• how to improve performance
• whether program effort and outcomes are
  correlated

80
Agency Role in Evaluation

• Supplement performance measures with program
  evaluation, cause/effect analysis, customer
  satisfaction surveys, process evaluation or
  reengineering techniques
• Make program improvements
• Make policy and program recommendations to the
  Legislature
• Follow-up and monitor progress

81
Evaluation Tools

• Compliance Verification
• Research Design Evaluation
• Statistical Testing
• Cost-Benefit Analysis

82
Compliance Verification

• Determine if actions and results are consistent
  with mandates and policies

• Example
• City Council ordinance mandates recycling and
  solid waste reduction to reduce use of landfill.
• Conduct field work to determine if landfill usage
  has declined

83
Research Design Evaluation

• Determine result of program/service and and what
  interventions/activities caused the result

• Example
• Evaluate impact of student readiness on student
  scores.
• Evaluate what improved student readiness

84
Statistical Testing

• Determine if there is a relationship between
  program inputs and outputs or outcomes achieved

• Example
• Determine what factors predict the number of bus
  violations a driver will incur

85
Cost-benefit Analysis

• Determine if the benefits of the program outweigh
  the associated costs over the life of the program
  

• Example
• What is the cost benefit of drug treatment
  programs that have an annual cost of 100,000 per
  client and a success rate of 5?

86
Exercise 5

• Legislative Oversight Evaluate Results

87
Making it Work The Basics

• Start with a Strategic Plan
• Focus on Results
• Chose a few, key measures
• Remember, the overall objective is accountability

88
Making it Work Implementation

• Think about long-term outcomes as well as
  short-term outputs
• Use outcomes as guidance, interpretation is the
  key
• Communicate, communicate, communicate
• Assess unintended consequences
• Manage the data and data collection process
• Publish results in a user-friendly way

89
Making it Work Using Performance Measures

• Link measures to existing management systems
• Use performance measures regularly and visibly in
  decision-making
• Assess program performance and make improvements
  where needed