Performance Measurement

1
Performance Measurement
2
Government Performance and Results Act (GPRA)

• Requires agency to
• Set multiyear strategic goals and corresponding
  annual goals
• Measure performance toward the achievement of
  those goals
• Publicly report on their progress

3
GPRA is to

• improve congressional decisionmaking by
  providing more objective information on achieving
  statutory objectives, and on the relative
  effectiveness and efficiency of federal programs
  and spending.
• Shift focus from preoccupation with staffing and
  activity levels to a broader focus on the results
  or outcomes of federal programs
• Answer the defining question Do the programs
  contribute to a real difference in the quality
  and security of citizens lives?
• Source David Walker

4
Federal agencies ask themselves

• What is our mission?
• What are our goals?
• How can we achieve them?
• How can we measure our performance?
• How will we use performance information to make
  improvements and other management decisions?

5
Continuing Challenges

• Expansion of reliance on state and local
  governments and other third parties to deliver
  services adds complications to process of
  reaching consensus as to strategic goals
• Predominance of entitlement spending clouds
  ability to hold agencies accountable for outcomes
  by shifting attention from broad goals (e.g.,
  assuring certain standard of living) to specific
  processes (ensuring correct and timely payments
  to individuals).
• Achievement of outcomes not necessarily under
  control of a single agency.
• Some outcomes can only be achieved over many
  years.

6
More challenges

• Level of agency at which performance planning
  and measurement should appropriately occur may
  not match the program activities that OMB and
  Congress is used to working with.
• Legislative staff not comfortable with use of
  outcome measurements for oversightwould rather
  have information that they are used to.
• GPRA cant eliminate conflict inherent in
  budgetingit is a political process of resource
  allocations.
•  

7
Difference between Outcome and Output Measures
Outcome An assessment of the results of the
program compared to its intended
purpose. Output A tabulation, calculation or
recording activity or effort that can be
expressed in a quantitative or qualitative manner.
As defined by GPRA
8
Outcome or Output?

• 40 of welfare recipients receiving job training
  are employed three months after its completion.
• Completely eradicate tropical spastic
  paraparesis.
• Annually inspect 3200 grain elevators.
• Ensure that 99.5 of payment vouchers are paid
  within 30 days or receipt.
• Update earning records for 45 million employee
  contributors to Social Security Trust Fund.
• Manufacture and deliver 35,000 rounds of
  armor-piercing 120mm projectile shells in 2001.

9
Example of Measurement Framework(State of Texas)

• Goal We will establish and implement policies
  governing purchasing and public works contracting
  that foster meaningful and substantive inclusion
  of historically underutilized businesses (HUBs)
• Objective To include historically underutilized
  businesses in at least ___ of total value of
  contracts and subcontracts awarded annually by FY
  2002
• Outcome measure Percentage of total dollar value
  of purchasing and public works contracts and
  subcontracts awarded to HUBs.
• Output measures
• of HUB contractors and subcontractors
  contacted for bid proposals
• Number of HUB contracts and subcontracts
  awarded,
• Dollar value of HUB contracts and subcontractors
  awarded

10
Example of FrameworkState of Minnesota

• Target population adolescent parents involved
  with child neglect or abuse
• Desired outcome Minor parents will parent their
  children appropriately
• Outcome indicator ratio of minor parents who
  have an acceptable score on parenting praoces to
  total of minor parents receiving services
• Method assessment instrument for parenting
  practices
• Performance target certain of minor parents
  who will parent appropriately by a particular year

11
Example of Framework San Francisco Muni Service
Standards

• Standard Missed service due to either
  insufficient vehicles or driver unavailability as
  a percentage of scheduled service hours
• GOAL No greater than 1.5 on July 1, 2004
• Purpose To measure service hours through
  available operators and available equipment.
• Definition of Measurement Monthly measurement
  of percent of total available hours missed for
  service measuring operators and equipment and
  percentage of equipment available daily.
• Method of Measurement Both operators and
  equipment are measured as to the total number of
  hours in service missed as a percentage of the
  total scheduled hours. Data come from the online
  Dispatching System and will be reported to the
  board on a quarterly basis.

12
San Francisco Muni (cont.)
Milestones
Report for the FY99/00 through February 2000
13
Measuring Citizen SatisfactionSan Francisco
14
Using BenchmarksCity of Long Beach
15
Developing Performance MeasuresHarry HatryThe
Urban Institute
16
Creating a Mission Statement

• Form
• To Identify the basic objectives (results) the
  program seeks and any major negative consequences
  the program should avoid.
• By Identify the basic way the services is
  provided. But dont constrain the options for
  ways to provide the service.
• Example Social Security Administration
• To Promote the economic security of the nations
  people
• By Shaping and managing Americas social
  security programs

17
Identifying key categories of customers

• Key Questions
• Who benefits?
• Who might be hurt?
• What other persons are affected?
• Which groups are particularly affected?
• Is the public likely to have a major interest?
• Consider both intermediate and end customers
• Example Child immunization program children,
  parents, taxpayers

18
Identify end and intermediate objectives

• Example Student Support Program (RISE)
• End objectives
• Better health in broader communities
• Research addressing needs of broader communities
  
• Greater number of underrepresented (UR) groups
  pursuing biomedical research careers
• Intermediate Objectives
• Greater retention of UR students in science
  majors
• UR students more competitive for grad programs
• More UR students interested in biomedical careers

19
Identify candidate outcomes

• Outcomes should
• Produce the results sought by the program
• Minimize undesirable effects
• Improve the quality of service delivery
• Reduce the amount of unmet need
• Produce benefits that accrue to the general
  population from providing effective services to
  the specific customer groups
• Provide equitable outcomes to customer groups

20
Design outcome sequence charts
Example RISE student support program
Students get into PhDprograms
Academic supportactivities
Students participate in activities
Better health
Students get bettergrades
Pursue biomed careers
End outcomes
Activity/output
Intermediate outcomes
21
Select outcome indicators

• Criteria
• Relevance
• Importance
• Understandability
• Program influence over outcome

• Feasibility
• Uniqueness
• Manipulability
• Comprehensiveness

• Example National Office of Drug Policy
• Objective Educate parents to help youth reject
  drugs
• Outcome Every youth has parent who provides good
  reasons for rejecting drugs
• Indicators
• Proportion of adults who disagree with
  statement What I say will have little influence
  on whether my child tries marijuana.
• The proportion of adults who report having
  discussed drugs with children.

22
Be careful what you measure!
23
Identify Data Collection Methods

• Possible sources
• Administrative data
• Customer surveys
• Trained observer ratings
• Special technical equipment

Examples Outcome Every youth has parent who
provides good reasons for rejecting
drugs Indicator Proportion of adults who
disagree with statement What I say will have
little influence on whether my child tries
marijuana. Data source National Survey on
American attitudes Outcome Break the cycle of
drugs and crime Indicator Proportion of inmates
who test positive for drugs Data SourceCensus of
jails
24
Preparing the Report
What breakouts would you use? Examples
organizational unit, type and amount of
serviceprovided, customer characteristics
What benchmarks/baselines would you
use? Examples previous performance, performance
of other jurisdictions, previously established
targets.
25
Examples of Reporting Formats
Example Fire Department