Performance Auditing

1
Performance Auditing

• Alta Fölscher
• CABRI/WBI PFM Training Seminar
• 18-20 June 2007

2
What is a Performance Audit

• Differs from regularity audit
• Different approach, more ad hoc, less rigid,
  different skills
• But same insofar an independent, objective
  assessment
• Audit of 3 Es
• Economy
• Efficiency
• Effectiveness
• All three or any combination of three
• Of wide range of issues
• Aim to lead to improvements
• Two dominant approaches
• Performance oriented
• Problem oriented

3
An example of a performance audit

• Does the Drug Abuse Resistance Education (DARE)
  Program Work?
• DARE is a drug abuse prevention program that aims
  to reduce drug use among school-age children. The
  performance audit of the program measured the
  extent to which it was achieving its goals. The
  audit used an experimental design, comparing
  juvenile arrest rates for youth in the program
  (the intervention group) with those who did not
  (the control group). The demographic profiles
  (ethnicity, income levels, age) were identical.
  The audit found that students who participated in
  the program were arrested more frequently than
  the control group, for both drug-related and
  non-drug-related offences. The design of this
  audit was heavily dependent on the existence of
  sufficient reliable data for determining student
  involvement in the program and identifying their
  arrest information in the local juvenile
  correctional system. These conditions are often
  difficult to meet, unless such comparisons have
  been planned from the start.

Source Location fictionalized from the 1994
audit of the Austin DARE Program, Office of the
City Auditor, Austin, Texas as reported in Waring
C and Morgan S, 2007. Public Sector Performance
Auditing in Shah, A (ed) Performance
Accountability and Combating Corruption, WBI,
Washington DC
4
Why is it important?

• Based on need for public accountability
• Insight into activities of public agency on
  behalf of stakeholders
• Reliable independent information on performance
  strengthens legitimacy and trust
• Basis for future decisions
• Incentives for change
• Better government spending, better services,
  better public accountability and better public
  management
• Are things done in the right way are the right
  things being done?

5
The 3 Es

• For same level of quality
• Economy
• Is about keeping the cost low
• Minimising the cost of resources used or required
• Are means chosen most economical use of
  resources?
• Have human, financial and material resources been
  used economically?
• Is management of programme in line with sound
  admin and management?
• Spending less
• Efficiency
• Relationship between outputs and the goods,
  services and resources used to produce them
• Can I produce more from given resources, can I
  use less resources for a level of outputs?
• In practice often difficult to distinguish from
  economy
• Spending well

6
3 Es

• Effectiveness
• Relationship between intended and achieved
  results of public spending
• Are aims being met by the means employed, outputs
  produced and impacts?
• Are the impacts really result of policy rather
  than other circumstances?
• Spending wisely

7
3Es

• Assessment always relative to audit criteria
  (standards against which actual performance
  measured)
• Important choice in audit design
• Standards used to determine whether programme
  meets or exceeds expectations
• Audit scope and design interprets economy,
  efficiency and effectiveness concretely
• Criteria should be relevant, reasonable,
  attainable eg
• Historical Similar programmes or other
  locations Legislation Accepted standard
  Professional opinion
• Assessment of effectiveness particularly tough
• Starts with programme design and clear, concrete
  goals
• Ideally measurement before and after control
  group
• In practice difficult
• Less ambitious
• Assess plausibility of assumptions in programme
  design and achievement of goals as set down (goal
  achievement analysis)
• Have objectives been reached, target groups
  reached, what is level of performance?

8
Mandate of Performance Auditor

• Important underpinning for work
• Mandate and goals should be defined in legal
  framework
• Specific regulations
• Auditor must be free to select audit areas within
  mandate and refuse commissions
• Professional quality of work important
  sufficient resources to acquire skills
• Differs across countries
• Coverage of mandate
• Scope of audits
• Approach

9
Input Output chain
Programme dimension
Performance Auditors domain
Commitment
Output
Outcome
Action/production
Input
Purpose defined
Activity undertaken
Resources assigned
Goods and services provided
Objectives met
Auditor should be careful not to stray into
political terrain But, audit findings can include
reflections on policies
Performance dimension

• Economy
• Physical and financial
• Amount
• Timing
• Quantity
• Quality

Efficiency Productivity Unit cost Operating ratios
Output effectiveness Level Quantity Timeliness Qua
lity Price Cost Customer satisfaction
Outcome effectiveness Mission and goal
achievement Financial viability Cost benefit Cost
effectiveness
Source Waring C and Morgan S, 2007. Public
Sector Performance Auditing in Shah, A (ed)
Performance Accountability and Combating
Corruption, WBI, Washington DC
Cross-cutting performance aspects
Compliance with laws and regulations validity,
reliability and availability of information
maintaining underlying values continuous
improvement
10
What to audit?

• Determine selection criteria country by country
• Added value important
• Important problem / problem area
• Risk
• Substantial budgets, sudden increase in budgets
• Traditional risk areas
• New, urgent activities
• Complex management arrangements
• Performance information not updated

11
Standards for Performance Auditing

• Well thought out plan very important
• Pre-study
• Audit proposal
• Identify issue and audit objectives (what will be
  audited and why)
• Develop scope and design of audit
• What issues, what are hypotheses, what type of
  study?
• Is issue auditable and worth auditing?
• Understanding audited programme
• Defining audit criteria
• Methodological planning
• Quality assurance, timetable, audit team and
  resources
• Field standards
• Communicative and analytical process
• Sufficiency, validity, reliability, relevance,
  reasonableness of evidence
• Scepticism
• Record-keeping and oversight
• Accessible, comprehensive reporting

12
Performance auditing chain
Audit cause the programme (intervention)
Audit criteria what should be
Audit evidence (condition) what is
Audit finding what is compared with what should
be
Determine the causes and effects of the finding
Develop audit conclusions and recommendations
Estimate likely impact of recommendations
13
Examples of audit findings
Type Elements Example
Descriptive Condition only Annual cost to incarcerate a prisoner was 67,800 in 2005.
Normative Condition and criteria Annual cost to incarcerate a prisoner was 67,800 in 2005, compared with 52,000 at comparable prison
Traditional/causal Criteria, condition, cause, and effect Annual cost to incarcerate a prisoner was 67,800 in 2005. Budget authorized 58,000, resulting in a deficit of 17.8 million. The additional costs were caused primarily by increase in labour and benefit costs following the May 2005 union contract.
Impact Condition with cause compared with condition without cause Recidivism rates among alcohol-dependent inmates who participated in the alcohol treatment before release were significantly lower than rates among alcohol-dependent inmates who did not receive the treatment.
Source Waring C and Morgan S, 2007. Public
Sector Performance Auditing in Shah, A (ed)
Performance Accountability and Combating
Corruption, WBI, Washington DC
14
Performance auditing in Sub-Saharan African

• Performance auditing can help build foundations
  for performance system
• But
• Keep it simple at start
• Basics should be in place (rule of law, clearly
  defined institutions with understood roles and
  responsibilities, budget systems and accounting
  systems)
• Financial audit function in place first
• Organisational independence, legal mandate,
  sufficient funding, strong leadership, ability to
  recruit and retain skilled staff, stakeholder
  support and professional audit standards
• Credibility
• Support at highest levels of government