Evaluation in Natural and Humanmade Crisis Situations

1
Evaluation in Natural and Human-made Crisis
Situations
Regional Workshop on UNDP Evaluation
Policy Bangkok, Thailand 16-19 October 2006
2
Outline

• Experience of two recent Evaluation Office
  evaluations
• Joint evaluation of International Response to
  Tsunami
• Joint Evaluation
• Approach and Methodology
• Key findings and recommendations
• Lessons from evaluation
• Evaluation of UNDP support in Conflict-affected
  countries
• Approach and Methodology
• Emerging findings
• Lessons from evaluation

3
Impact of International Response to Tsunami

• Background
• Tsunami of December 2004 affected 14 countries,
  killed 227,000 people, displaced over 1.7 million
  and caused damage of USD9.9billion. The response
  was also unprecedented involving mobilized
  resources around USD13.5 billion
• To hold actors accountable and to provide
  feedback to programming activities that are
  expected to continue for 3-5 years
• Joint evaluation involving over 40 agencies UN
  agencies, Bilateral donors, INGOs, NGOs (TEC)
• Five thematic areas Coordination, Funding,
  Impact on Local and National Capacities, Linking
  Relief and Rehabilitation to Development, Needs
  Assessment.

4
Architecture of Tsunami Evaluation Coalition
(TEC)

• Open structure participatory decision-making
  and voluntary
• Core-Management Group Chaired by Ocha, included
  UN agencies (UNDP, UNICEF,WHO and FAO), bilateral
  donors (Danida, Sida, SDC), IFRC, INGO (World
  Vision) and NGOs (AIDMI) Manage the overall
  evaluation process, produce the synthesis report,
  assure quality and conduct outreach activities
• TEC Secretariat To coordinate the five thematic
  evaluations and produce a synthesis report based
  on the five thematic evaluation.
• Peer Review panel of international experts
• Thematic Groups
• Steering Committee to assist the management of
  the process of each thematic evaluation
• Working group for each thematic area- to assure
  quality of Thematic Evaluation Reports

5
Approach Methodology

• Assess sector-wide performance (not individual
  agency performance) of tsunami response
• Performance Criteria International Codes
  including, Code of Conduct, Sphere, Good
  Humanitarian Donorship (GHD)- not simply speed
  and quantum of services delivered
• Basis for Thematic Report on Impact on Local and
  National Capacities
• Country case-studies (Indonesia, Maldives, Sri
  Lanka, Thailand) (focus group discussions and
  interviews- 550 interlocutors),
• Claimholder (Beneficiary) surveys in two
  countries (involving 2000 households)
• Desk review of secondary evidence
• Quality Assurance Peer reviews by working group,
  drafts of report reviewed by stakeholders, entry
  exit stakeholder workshops

6
Evaluation Constraints

• Scope was restricted to response during the first
  8-11 months (field studies from Sept Nov 2005)
• Limited duration of field studies (country
  missions from 10 days to three weeks)
• Dearth of documentary evidence (project
  documents, budget, etc. were difficult to obtain)
• The report presents prevalent systemic issues
  there are number of exceptions to the trends, all
  of which could not be recorded.

7
Select findings from capacities evaluation

• Weak to non-existent support to strengthening
  local/National ownership of relief and early
  recovery efforts
• Phased approach (disjointed relief, recovery and
  development) weakens sustained longer term
  capacity development
• Inadequate disaster preparedness and investment
  in disaster preparedness

8
Select Findings cont.

• Local capacity is key to saving lives and
  improving peoples resilience to disasters but
  unrecognized and undervalued by external actors
  (Claimholder Surveys)
• Consultations and information sharing with
  claim-holders were often non-existent or
  inadequate during initial phases (Claimholder
  Surveys)
• Pressure to spend has favored capital-intensive,
  high-profile programmes, subcontracting and
  poaching, to the detriment of sustainability
  and local ownership
• International aid is likely to reinforce existing
  inequalities within community if response is not
  designed to address existing inequities and those
  likely to emerge due to response (e.g. gender
  inequalities, informal sector,etc).

9
Select Recommendations

• The international humanitarian community needs a
  fundamental reorientation from supplying aid to
  supporting and facilitating communities own
  relief and recovery priorities
• All actors should increase their disaster
  response capacities and improve the linkages and
  coherence between themselves and other actors
  including those from the affected countries
• The international relief system should establish
  an accreditation and certification system for
  humanitarian work
• All actors need to make the current funding
  system impartial, and more efficient, flexible,
  transparent and better aligned with principles
• Engage with local and national capacities at the
  very early stages - Require local partnerships in
  recovery programming while working against the
  prevalent practice of Poaching of
  local/national capacities
• Prioritize acquiring analytical capacity based on
  nuanced understanding of national and local
  context.
• Provide incentives and guidelines for
  claim-holder consultations, information sharing
  and inclusive decision making

10
Lessons

• High profile evaluation
• Comprehensive approach
• Reduced transaction costs
• Higher quality
• Unclear accountability trail
• Difficult to coordinate follow-up

11
Evaluation of UNDP support to conflict-affected
countries

• Rationale
• Crisis Prevention and Recovery is one of the five
  practice areas of UNDP
• Nearly 40 of the UNDP global expenditure in 2005
  was in BCPR supported countries
• Requested by the UNDP Executive Board

12
Purpose

• To evaluate UNDP assistance to conflict-affected
  countries with special reference to human
  development and human security
• To look at UNDPs role in coordination and
  partnership
• To make forward looking recommendations

13
Conceptual Framework

• Human Security is about the security of
  individuals and communities. It is about both
  physical and material security.
• Represents a response to extreme vulnerabilities
  the downside risks faced by human beings.
  (Amartya Sen)
• About human development during conflict and
  crises

14
Evaluation Questions

• What are the trends in human security and
  conflict in the case study countries and other
  selected countries since 2000
• Since the absence of violent conflict is an
  important component of human security, what can
  we learn from our case studies about the
  character of conflict and can we identify
  underlying conditions that make violence more or
  less likely?
• In countries where the United Nations has
  intervened in before, during or after conflict,
  is it possible to conclude that the intervention
  as a whole has contributed to an improvement in
  human security? In particular, has the
  intervention addressed the conditions that are
  likely to exacerbate conflict?
• Is it possible to identify a specific UNDP role
  in contributing to human security? Was UNDP
  assistance targeted at human security needs as
  identified above?
• Did institutional arrangements within UNDP and
  with partners help or hinder UNDPs role in
  contributing to human security?
• What lessons can we learn for future strategy,
  institutional arrangements, and monitoring and
  evaluation?

15
Approach

• The trends in human security including the
  character of the conflict
• The overall role of the United Nations in
  contributing to human security and addressing the
  conditions conducive to conflict
• The contributions of UNDP assessed in terms of
• Relevance and Positioning.
• Results and Effectiveness.
• Management
• Coordination and partnerships
• Substantive Leadership and Credibility

16
Approach to Case Studies

• Afghanistan, DR Congo, Guatemala, Haiti, Sierra
  Leone, Tajikistan (six countries with Security
  Council Mandates)
• Survey of 24 Countries Receiving BCPR Assistance
• Stakeholder Interviews, Documentary research,
  Other Statistics

17
Methodology

• Four Indicators
• Battle Deaths,
• Human Rights Violations,
• Refugees and Displaced Persons, and
• Victims of Natural and Technological Disasters
• Emphasis on extreme vulnerabilities (rather than
  longer term measures of human development).
• Limitations Paucity of Data

18
Emerging findings

• International community helped to stabilise
  conflict but less successful at addressing every
  day human insecurity and the structural
  conditions conducive to conflict
• Insufficient humanitarian and development voice
  in integrated UN missions
• UNDP has unique mandate and role to address
  structural conditions conducive to conflict
• UNDP needs to build substantive, including
  analytical, capacity in core areas, improve
  effectiveness of implementation, and enhance
  coordination and partnerships

19
Lessons

• Methodological issues around use of human
  security as evaluative lens
• Analytical issues of situating UNDP within the
  international community
• Value of focus group discussions in country case
  studies
• Broader implications of implementing
  recommendations

20
Thank you!