Evaluation Policy Renewal Overview

1
Evaluation Policy Renewal Overview

• March 22, 2007
• RDIMS

2
Getting started on evaluation policy renewal

• Drivers for change
• Diagnostic
• Renewed evaluation policy proposed directions
• Renewed evaluation policy what will this mean
  for DPRs?
• Next steps

3
Key drivers for renewing the evaluation policy

• our governments approach to spending control
  is based on the following three principles
• government programs should focus on results and
  value for money
• government programs must be consistent with
  federal responsibilities and
• programs that no longer serve the purpose for
  which they were created should be eliminated.
• With those principles in mind, the Government
  is launching a review of its expenditure
  management system.
• The Budget Speech 2006 (p.18)

• The Federal Accountability Act requires all
  Transfer Payment Programs to be evaluated
  (relevance and effectiveness) over a five-year
  cycle
• Budget 2006 and the Economic and Fiscal Update
  commits to using results and value-for-money to
  inform priority setting and decision-making
  through a renewed EMS system
• Evaluation will be critical to deliver on this
  commitment
• Current evaluation policy is not meeting the
  needs of Deputy Heads and does not provide the
  support evaluators want to address government
  information needs (Breen Report, 2005)
• OAG has consistently noted the need for the
  evaluation of ongoing programs

4
Diagnostic of Evaluation Function Evaluation
products not always timely and of high quality
There are serious deficiencies
Months to Undertake an Evaluation (Does not
include time for approval publication)
(N128, 2004-05)

• 23 of evaluations are of poor quality relative
  to our 2001 policy criteria
• Deputies think that evaluations take too long to
  complete and are difficult to understand
• 47 of evaluations take more than a year to
  complete (N128)

SOURCES EKOS Assessment of Federal Evaluation
Quality (2004-05) Breen Report, 2004-05
CEE Annual Survey of Heads of Evaluation,
200-06
5
Diagnostic of Evaluation Function Narrow Focus
of Current Evaluations
Most evaluations focus on small programs
Size of Programs Evaluated (N129, 2004-05)

• Half of evaluation studies focus on programs of
  4 million or less
• 12 of evaluations are of programs less than
  500K
• Deputies have said they want higher-level
  evaluations. They want to know departmental
  performance of policy areas or higher level
  programs in the PAA
• The OAG has consistently encouraged evaluations
  of more important programs of interest to
  Parliamentarians

SOURCES Annual CEE Survey of Departments, 2005
Breen Report, 2005 Report of the Auditor
General of Canada - May - Chapter 3, 1996
Report of the Auditor General of Canada 2000,
December - Chapter 20 - Appendix D
6
Diagnostic of Evaluation Function Evaluation is
often not addressing the right questions
Majority of evaluations focus on program
improvement
Evaluation Issues Addressed (200 Evaluation
Reports)

• Convincing assessments of relevance and
  value-for-money are rare
• Majority of evaluations focus primarily on
  program improvement (72)
• Need to find a balance between program
  improvement and assessing poor performers

SOURCE TBS Evaluation data-base (ERIC) CEE,
2004 Consultations
7
Overview Proposed Policy Objective and Focus
Policy Objective Ensure that evaluation
information is available to Ministers,
departments and central agencies to support
evidence-based decision-making on policy,
expenditure management and program improvements

• The proposed policy and its supporting directive
  focus on the following areas
• Re-focus evaluation on value-for-money and
  strengthen accountability for performance of the
  function
• Expanded evaluation coverage
• Strengthened governance and usage
• Appropriate evaluation competencies and standards
  for evaluators
• Strengthened TBS capacity to lead, monitor and
  use evaluation information
• Address small agency evaluation needs

8
Proposed Policy Directions Re-focus Evaluation
on Value for Money

• Balance evaluation findings to support program
  improvement and the assessment of program
  performance (identification of opportunities for
  investment and reallocation)

• Policy Proposals
• Refocus evaluation on results and value-for-money
  (i.e., relevance and program performance)
• Clear expectation as to what constitutes an
  evaluation report and who can undertake an
  evaluation
• Ensure those evaluations used to support
  decision-making provide conclusions on the
  relevance and effectiveness of programs
• Introduce new evaluation approaches to support
  the timeliness and rigor of evaluation linking
  complexity of evaluation with the risks
  associated with a program

• Proposed Suite of Evaluation Approaches__________
  _______
• Strategic Policy Evaluation
• Impact Evaluation
• Targeted Evaluations
• Implementation Evaluation

9
Proposed Policy Directions Expand Evaluation
Coverage

• Federal Accountability Act expectation is 100
  coverage of transfer payment programs over five
  years
• EMS renewal could extend coverage beyond GsCs to
  involve a review of all direct program
  expenditures over five years

• Policy Proposals
• Expectation of 100 coverage of all program
  expenditures (Direct Program Spending) over a
  five-year cycle achievable through a mix of
• Re-orienting existing management reviews toward
  value-for-money issues
• Create efficiencies by introducing a suite of
  flexible evaluation approaches based on size,
  complexity, and risk
• Investing adequate resources in the evaluation
  function
• Rolling (five year) departmental evaluation plans
  
• Plans would guide application of a broad suite of
  evaluation tools based on risk, scale and impact
• Introduce a TB Government of Canada Evaluation
  Plan that links departmental plans and directs
  horizontal reviews

10
Proposed Policy Directions - Clear Accountability
and Governance
Policy Proposals

• Evaluation is used to inform management
  decisionmaking
• Deputy Heads and their teams are primary users
  (i.e. provide leadership and ensure usage)
• Departmental Evaluation Committee to review
  reports
• Clarifies roles and responsibilities of the Head
  of Evaluation
• Evaluation results reported directly to Deputies
• Ensuring results orientation of new spending
  initiatives (i.e., allocation)
• Ensuring evaluative information available to
  support expenditure management (i.e.,
  reallocation)

11
Proposed Policy Directions Clear Links to
Performance Measurement

• Role of evaluation
• review and provide advice on performance
  measurement strategies embedded in the
  organizations Management Resources and Results
  Structure
• review and provide advice on ongoing performance
  measurement strategies, including RMAFs and
  performance provisions in Cabinet documents
• assess departmental performance measurement
• report annually on the state of performance
  measurement of programs
• informed, annual discussion by departmental
  evaluation committees on the state of performance
  measurement activities
• Program managers are responsible for on-going
  performance monitoring
• develop and implement ongoing performance
  measurement strategies
• consult with the head of evaluation on the
  performance measurement strategies of all new and
  ongoing program spending

12
Strengthening DPRs integrating evaluation
findings

• Evaluation is the key source of credible and
  neutral information on the relevance, success and
  cost-effectiveness of programs and policies
• Improve DPRs by integrating evaluation
  information on outcomes
• Provide a complete picture of the performance
  story
• Evaluation information on all direct program
  spending (over a five-year cycle)
• with clear links to the PAA
• Consultations with the evaluation units are key
• can provide advice and input into DPRs