ResultsBased Governance: Governance through Measurement

1
Results-Based GovernanceGovernance through
Measurement?
In Exploring Evaluation and Collaborative
Government

• John Mayne
• Office of the Auditor General of Canada
• john.mayne_at_oag-bvg.gc.ca

2
Outline

• challenges to traditional governance
• elements of new governance
• monitoring outcomes as governance
• some implications

3
challenges to governance

• complexity in society
• information flood, interconnectedness, better
  educated citizens, rapid feedback
• collaborative arrangements becoming the norm
• to deal with complexity and reduced capacity of
  government
• citizen engagement, public reporting and
  accountability
• demands for and requirements for greater direct
  involvement of citizens in governance

4
challenges to governance

• traditional organizations cant cope, need for
  learning an adaptability
• traditional accountability concepts
  incompatible with the need for learning and
  adapting
• traditional characteristics of effective
  leadership, control, evaluation/audit may add to
  rather than help solve the problems

5
elements of new governance

• resilient organizations and arrangements (3Ds
  focus rather than 3Es)
• living with uncertainty and embracing errors
• government as a vehicle for societal learning
• focus on sustainability
• accountability as learning

6
governing (learning) with outcomes

• results-based management a start, but ...
• need to go beyond the current measurement trend
  in public administration
• using the setting, agreeing on, monitoring and
  reporting outcomes as a means of governance in
  complex, collaborative situations
• several potential benefits

7
potential benefits

• sorting out roles and responsibilities
• coordination among governments and citizens
• encouragement of local and citizen involvement
• encouragement of learning
• enhancement of accountability

8
examples of early outcome-based governance

• Australia Committee for the Review of
  Commonwealth/State Service Provision
• Canada Securing the Social Union
• Canada Outcome-based national standards as an
  instrument of federalism (Labour Market
  Agreements)

9
Some implications

• for accountability
• for reporting
• for evaluation and audit

10
some implications for accountability

• re-articulation of accountability
• accountability for outcomes
• shared accountability

11
Traditional accountability

• Accountability is the obligation to answer for a
  responsibility conferred
• is exclusively hierarchical
• apportions blame
• aims at punishing wrongdoing
• focuses on compliance with procedures
• loved by auditors.

12
Modernizing accountability

• The obligation to demonstrate and take
  responsibility for performance in light of agreed
  expectations.
• works in collaborative government
• focuses on results not compliance
• supports learning
• http//www.tbs-sct.gc.ca/rma/account/OAGTBS_E.html

13
Accountability for results

• Requires
• credibly demonstrating
• results achieved
• contribution made (dealing with attribution)
• what has been learned
• taking responsibility for the strategies used

14
Shared Accountability

• Requires reporting on
• overall performance of the arrangement
• the federal contribution
• what has been learned
• Requires taking responsibility for
• the arrangement framework
• federal actions and strategies

15
some implications for reporting

• electronic publishing
• data bases replacing reporting
• comparative reporting
• providing audit assurance
• transparency replacing audit

16
some implications for evaluation

• requires fuzzy measurement/evaluation
• measurement as argument (Majone 1989) or even
  performance story telling (Mayne 1999)
• measurement as reducing not eliminating
  uncertainty
• measurement as learning not control
• measurement of new things (trust, influence)

17
New criteria of success
Success as adaptability and continued relevance

• Diagnosis
• identification and redefinition of problems
• Design
• formulation of new solutions, structures
  strategies
• Development
• implementation of solutions as a learning process

vs

• Economy, Efficiency, Effectiveness

backup
18
traditional evaluation still needed

• Evaluation is also needed to measure
• attribution/contribution/influence
• unintended effects, including those related to
  measuring and moving towards results-based
  governance
• the level and nature of learning going on in
  public organizations

19
Contribution Analysis
The results chain
inputs
under program control
I/O analysis
outputs
Information obtainable
immediate outcomes

• Contribution analysis
• program evaluation
• confirmation of logic
• evidence on
• outcomes occurring
• relationships between outcomes
• impact of other factors
• identification of knowledge gaps

outside factors
intermediate outcomes
final outcomes
20
Measurement strategy

• agree on the vision/logic of the program
• agree on what ought to be measured to know how
  well you are doing
• identify which can be handled through ongoing
  measures (indicators)
• program evaluations are needed identify which
  evaluations need to be carried out

21
Theory-based Evaluation

• Reconstructing the theory of the program
• Assess/test the credibility of the micro-steps in
  the theory (links in the program logic/story)
• Developing confirming the results achieved by
  the program
• References
• Chen (1990). Theory-Driven Evaluation A
  Comprehensive Perspective. Sage.
• Chen (1994). Papers from a Panel. Evaluation
  Practice. Vol. 1
• Weiss (1997). Theory-based Evaluation Past,
  Present and Future. New Directions for
  Evaluation. No. 76, Winter.

22
Evaluators

• Foster the use of
• logic charts/results chains/performance
  frameworks to understand manage performance
• contribution analysis to examine attribution
• evaluations to assess attribution
• measurement strategies
• Consider using theory-based approaches to
  evaluation

23
some implications for performance audit

• perhaps even more of a challenge here
• audit does not deal well with uncertainty nor
  errors it is expected to tell the truth
• can audits have fuzzy conclusions?
• principle-based audit

24
governance through results-based learning