Impact Assessment and ICT4D

1
Impact Assessment and ICT4D

• David Souter

2
What is impact assessment?

• Impact assessment is the systematic analysis of
  the lasting or significant changes positive or
  negative, intended or not in peoples lives
  brought about by a given action or series of
  actions.
• Novib/Oxfam research project, C. Roche, Impact
  Assessment for Development Agencies
• Impact assessment refers to an evaluation of
  how, and to what extent, development
  interventions cause sustainable changes in living
  conditions and behaviour of beneficiaries and the
  differential effects of these on women and men.
• Peter Oakley et al., Outcomes and Impact

3
Why do it?

• Proof of effectiveness is not the main goal of
  impact assessment it is concerned more with
  understanding, and learning from, the processes
  and approaches that have led to achieving a
  particular impact so that such knowledge and
  experience could be used in comparable
  situations.

4
Four levels of results
5
The complexity of context
Impact
Outcomes or Effects
Outputs
Activities
Increasing influence of context (social,
economic, political, etc.)
Inputs
Source Roche
Baseline
6
An example - 1
Urban centre
X
7
An example - 2
Urban centre
X
8
An example 3
A
B
Urban centre
X
D
9
An example - 4
A
B
Urban centre
X
D
10
The principal challenges

• The challenge of the baseline
• The challenge of timescale
• The challenge of attribution
• The challenge of aggregation
• The challenge of the unexpected
• The challenge of redistribution

11
The challenge of the baseline

• without some sort of baseline, it is not
  really possible to assess impact.
• unless detailed and continuous ME takes place
  at programme and project level, it will be
  impossible to build up the body of knowledge
  which is necessary to understand the wider
  outcomes.
• Measuring change requires understanding of the
  starting point from which change is measured
• This includes all areas in which potential impact
  is anticipated (those related to wider contextual
  issues)
• Most impact assessment suffers from the lack of
  baseline data
• Impact assessment needs to be built into
  processes from the beginning stage if it is to be
  undertaken meaningfully
• Measures can be taken to try to recover baseline,
  but these are difficult and inevitably weak

12
The challenge of timescale

• Impact varies over time
• short-term impacts often differ from long-term
  impacts
• Lasting and sustainable change only becomes
  apparent over longer timeframes
• Very few initiatives have seen retrospective
  impact assessments (returning to the scene a
  period after the end of the intervention)

13
The challenge of attribution

• Many factors influence change many people may
  wish to claim credit for it.
• Attributing responsibility for change is complex
  and contentious and ultimately uncertain.
• There is often a temptation to claim credit in
  order to emphasise the value of work done
• but this does not help to understand what has
  been (or not been) achieved, and so does not
  contribute to the quality of future work

14
The challenge of aggregation

• Individual interventions usually target specific
  aspects of change
• Other interventions target similar and related
  aspects of change
• Change may result from the cumulative impact of
  numerous interventions
• the sum is more than the total of the parts
• implying that it is the sum rather than the
  parts that drives impact and that impact
  assessment should look specifically at
  inter-intervention synergies

15
The challenge of the unexpected

• All initiatives have unexpected impacts
• These may be positive or negative
• It is as important to understand these
  unexpected impacts as to understand whether
  intended impacts are achieved.
• Negative impacts tend to be under-reported, with
  the result that important lessons are not learnt
  for future programme design and implementation

16
The challenge of redistribution

• Interventions are usually targeted at specific
  groups, but they affect wider communities
• Impact assessment needs to address impacts on all
  who are affected, and understand what
  distributional changes may have arisen
• Baselines and stakeholder relationships with
  non-target groups are usually weak but
  widespread and diverse stakeholder perception is
  crucial to understanding impact
• Non-target beneficiaries may be reticent about
  negative impacts on themselves
• e.g. women may be reticent about changes in the
  balance of power within households

17
Impact assessment and advocacy

• Assessing the impact of advocacy work involves
  two tiers of assessment
• did advocacy lead to policy change and
• did policy change lead to lasting and sustainable
  change in peoples lives
• Much advocacy work has long-term objectives and
  may have few short-term effects
• e.g. influencing the quality of journalism
  broadening the range of options considered within
  policy formulation increasing womens
  participation
• Advocacy impact often results from aggregated
  interventions

18
Oxfams tiers of advocacy impact
Heightened awareness
Contribution to debate
Change in opinions
Change in policy
Implementation of change in policy
Positive change in peoples lives
19
Impact assessment of ICD projects

• Impact assessment of ICD projects does not
  obviously differ in principle from impact
  assessment in other development areas
• It should therefore gain from sharing experience
  with the wider development community
• This experience emphasises
• development of baselines
• continuous monitoring and assessment throughout
  intervention cycle and beyond
• stakeholder participation
• exploration of unexpected and negative outcomes

20
Impact Assessment and ICT4D

• However, there are aspects of ICD projects which
  would exacerbate the challenges of impact
  assessment
• the lack of baseline data
• the very limited range of existing substantive
  impact assessment
• the unpredictability of change in ICT technology
  and behaviour
• uncertainty about the implications of different
  policy options

21
The BCO Impact Assessment

• The BCO Impact Assessment will look at three
  areas of specific experience
• Community radio, related media and political
  change in Nepal
• Agriculture, microfinance and ICTs in Ecuador
• Experience with networks in ICT4D work,
  particularly advocacy
• and include a specific review of impact
  assessment in ICD compared with other areas

22
Investigation 4 scope

• Purpose
• To contribute towards improved processes for
  monitoring, evaluation and impact assessment

• Scope
• To review the approaches to evaluation and impact
  assessment which are used by BCO partners and
  other ICD agencies
• To compare these with approaches to evaluation
  and impact assessment which are used in other
  development organisations, including both donor
  agencies and mainstream development NGOs
• To assess whether, how and where the evaluation
  and impact assessment requirements of ICD
  activities differ from those of other development
  contexts
• To identify measures which BCO partners and other
  ICD agencies can take to improve the quality and
  value of their monitoring, evaluation and impact
  assessment