The National Audit Office's evaluation of RIAs: Experiences from the Pilot Year

1
The National Audit Office's evaluation of RIAs
Experiences from the Pilot Year

• Ed Humpherson
• 26 November 2003

2
Presentation structure

• The NAOs 2001 report on Better Regulation
• The emerging findings of our 2003 evaluation of
  RIAs
• The benefits of the NAOs evaluation process
• The risks of the NAOs evaluation process

3
Auditing regulation A short history

• Developed from privatisation audit
• 1990s reports on sectoral regulators
• Late 1990s applying the lessons learned to
  other regulators (eg OFT, Gaming Board)
• Late 1990s, growing interest in Better
  Regulation, leading to.
• .2000s NAO work on Better Regulation (2001
  report)

4
The NAOs 2001 report on Better Regulation

• The aim of the report was to highlight good
  practice examples
• We evaluated 23 RIAs from a wide range of
  Departments and agencies

5
Key messages from 2001 report

• Considering regulatory impacts from an early
  stage requires a culture change
• Effective consultation is not easy
• Guidance needed on Small Business Litmus Test
• Policy makers unfamiliar with costing techniques
• Evaluating benefits is important but difficult
  

6
2003 Evaluating the quality of RIAs

• Cabinet Office asked the NAO to carry out an
  annual evaluation of RIAs
• First annual evaluation in 2003
• Unlike 2001 report the annual evaluation draws
  out both good and bad learning points

7
2003 Evaluating the quality of RIAs

• The purpose of our evaluation of RIAs is to
• raise the profile of RIAs within government
• identify and disseminate best practice and
• enhance the credibility of RIAs through robust
  accountability

8
2003 Evaluating the quality of RIAs

• Scope
• Evaluate quality of RIA process for sample of 10
  RIAs
• Methodology
• Framework of six high level questions
• Output
• Unpublished reports to departments
• Published Compendium Report in early 2004

9
2003 Preliminary Findings
10
2003 Preliminary Findings in context

• Sample focused on 2002 RIAs
• Drawn from Task Force recommendations a skewed
  sample?
• Expect to see ongoing improvements in quality

11
The benefits of the NAOs work

• The accountability benefit
• The beneficial change benefit

12
The accountability benefit

• The NAO is independent of Government
• Within Government raising the profile
• Outside Government adding credibility
• Challenging bureaucratic dispositions (drafting
  and deterministic thinking)

13
The beneficial change benefit

• Encouraging transfer of technically strong
  elements of RIAs across Government
• Detailed work on each individual RIA based on
  documentary evidence
• Focus on learning points
• Highlighting the full potential of the RIA

14
The risks of the NAOs evaluation

• Narrow (and retrospective) focus an ex post
  evaluation of an ex ante tool
• Meaning of quality
• The null hypothesis does the RIA make any
  difference at all?
• The audit society inhibiting change

15
The National Audit Office's evaluation of RIAs
Experiences from the Pilot Year

• Ed Humpherson
• 26 November 2003