Phase 2 Evaluation of the PD

1
Phase 2 Evaluation of the PD

• Key Issues for
• Evaluation Design
• Reference Group Workshop/Meeting 11 13
  February 2009

2
Evaluation Design

• Many choices must be made when finalising an
  evaluation design, for example, we have to
  choose
• The main focus and purpose
• The Unit of Analysis what are the
  investigations located
• Methods and techniques
• Information sources
• Management and governance etc.
• All of these are discussed at this workshop

3
Evaluation Design

• In this session we want to concentrate on two
  important design choices
• The evaluation questions that should be
  prioritised in the evaluation
• Some propositions (or hypotheses) that follow
  from the characteristics of the PD itself

4
Evaluation Questions

• Evaluation questions are at the centre of any
  evaluation what are they?
• At simple level they are questions we want
  answers to and we are likely to be able to use
  the results to inform policy or practice
• Evaluation questions also concern effects and
  imply judgement placing a value on these
  effects

5
Evaluation Questions

• So an evaluation question
• Concerns an effect or result external to the
  interventions of the programme or policy if a
  question is just about the internal working of
  the policy its better dealt with by monitoring
  or auditing!
• For example

6
Evaluation Questions

• An evaluation question also implies a judgement
  criterion
• Some evaluation questions at least should help us
  understand what constitutes success,
  achievement or good performance
• For example

7
Evaluation Questions

• Evaluation questions lead to certain types of
  enquiry. They can be
• Descriptive so that the evaluation describes
  what happens in a situation or over time
• Explanatory so that the evaluation investigates
  causal relationships
• Normative so that the evaluation focuses on how
  to bring about desired change
• For example

8
Evaluation Questions

• In the Approach Paper the linkages study
  different types of evaluation questions are
  identified. The main types are
• Results questions. For example How has the
  adoption of PD principles and commitments
  improved development results
• And
• Counterfactual questions. For example Are there
  other ways not following PD principles that
  could lead to similar results more effectively
  efficiently

9
Evaluation Questions

• The emphasis on results follows from the focus of
  the Phase 2 evaluation.
• Of course we can ask process and implementation
  questions as well if these processes help us
  understand results

10
Evaluation Questions

• But these are examples only in the group
  discussions after this presentation you will have
  the chance to put forward what you think are the
  most important questions for the evaluation to
  answer

11
Characteristics of the PD

• Many evaluation design choices arent choices at
  all they follow from the character of the
  policy or programme being evaluated! For example
  the linkages study identifies some of the special
  qualities of the PD that make this evaluation
  very challenging

12
Characteristics of the PD

• For example we know that
• The PD is closely connected to other aid and
  development policies how to separate out PD
  effects?
• Development actors (governments, civil society
  etc.) will use PD arrangements differently
  there wont be one standard model
• The PD is not a single once-only intervention it
  is a development and learning process
• Not all results of the PD will be evident by
  2010!

13
Characteristics of the PD

• Evaluation will be difficult but possible! The
  linkages study identifies 21 propositions (or
  hypothesis) that research suggests could be
  tested in the Phase 2 evaluation. These are on
  Page 16 of the Approach Paper.
• These propositions come from research rather
  than practice. This is why it would be useful for
  people here who are familiar with aid
  development policy to consider and review these
  propositions

14
Discussion Tasks

• The two tasks for the discussion groups will
  concentrate on these two design choices
• Evaluation Questions
• Evaluation Propositions
• But first are there any questions or points of
  clarification..?