Humanitarian aid evaluation at Medecins sans Frontieres

1
Humanitarian aid evaluation at Medecins sans
Frontieres

• Sabine Kampmüller, MIH
• MSF Vienna Evaluation Unit
• http//evaluation.msf.at

2
Dimensions of evaluation
Source SDC (2002)
3
Evaluation post-disaster Haiti earthquake
response
Evaluation conducted 5 10/2010
4
Evaluation process

• 1st time (!) all section Review
• 6 specific reviews Global/operational,
  medical/surg, log/supply, Comms, FR, HR
• Quantitative qualitative methods
• data, field visits, interviews, qualitative
  research - beneficiaries, web survey -staff
• Limitations
• Incomplete and non-uniform data
• Recall problems

5
(No Transcript)
6
Patients / Communitys Perception

• Very positive perception of MSFs interventions
  and services
• Fear/experience of stigmatisation, increase of
  violence in the camps 
• Lack of space for occupational and educational
  activities.  
• More attention to socio-cultural and spiritual
  needs

7
Haiti evaluation findings

• MSF was one of the biggest emergency health
  actors
• Operational choices in line with emergency needs
  less coherent over time
• Common data collection difficult to impossible
• International platforms / working groups are
  underused / undervalued
• Successful adocacy on some issues, complicated
  decision making prevented more

8
Main recommendations

1. Organize inter-section capacity for assessment
   and monitoring of evolving needs and assistance
2. Define strategy on mass casualty
3. Revise emergency supply strategies
4. Ensure uniform data collection
5. Strengthen technical working groups
6. Focus on organisational learning
7. Engage with national and international actors

9
Thematic Evaluation Response to displacement
Evaluation conducted 2009/2010
10
Evaluation process

• Comparison of 6 case studies (urban/rural,
  low/middle income, etc)
• Quantitative qualitative methods
• Literature research, document/tools data
  review, field visits, interviews,
• Limitations
• Availability of data and key people
• Poor documentation

11
Particularity of open settings

• Lack of clear boundaries
• Geographical spread, invisibility, needs
  difficult to identify and measure, protection
  issues
• Displaced settle in environment with available
  resources and existing health system
• Better survival capacities, but deterioration
  likely
• Protracted, chronic or intermittent character
• Mortality near-normal levels (pre-emergency),
  might rise slowly over time

12
Findings on assessment

• Complexity of open settings requires more
  attention to assessment
• Critical aspects of information missing
• Quantitative information difficult to obtain
• Health system issues, access barriers little
  addressed
• Concerns on the use of surveys
• Views of displaced and host often omitted
• Capacities and vulnerabilities not assessed

13
Recommendations on assessment

• Develop innovative assessment approaches for
  inaccessible areas
• Distance assessment
• Promote systematic use of qualitative methods
• Adopt concept of continual assessment
• Develop a frame to assess vulnerabilities,
  capacities and coping
• Provide better support and guidance
• assessment toolbox, experts, training

14
Findings on intervention

• Engagement with the health system a main
  challenge
• Outreach workers invaluable, set ups improvable
• Non-medical assistance is marginal
• Overambitious targets of coverage
• Strategy adapted to the level of emergency

15
Recommendations on intervention

• Need for new intervention frameworks
• Adopt existing models
• Generalize the practice to cover both displaced
  and host where appropriate
• Define the criteria / scope of light support
• Develop community based strategies

16
Evaluation criteria (donors)

• Appropriateness
• (Connectedness)
• (Coherence)
• Timeliness
• Coordination
• Coverage

• Relevance
• Effectiveness
• Efficiency
• Impact
• Sustainability

Adopted from Hallam, A. Good practice review
(ODI), 1998
17
Ways to ensure UTILISATION

1. Get original purpose very clear
2. Ensure ownership, participatory process
3. Get key stakeholders behind
4. Share findings in the field
5. Allow debate with evaluators
6. Ensure good readibility of report
7. Disseminate info widely / on all levels
8. Promote credibility of evaluators
9. Use in training
10. Formulate recommendations clear and realistically

18
10 Steps in the process

1. Defining purpose and scope
2. Writing the terms of reference
3. Analysing stakeholders
4. Choosing the methodology
5. Deciding the budget for evalution
6. Selecting the evaluation team
7. Preparing the field
8. Research phase
9. Reporting and dissemination
10. Managment follow-up