Assessing Humanitarian Performance: Where are we now?

1
Assessing Humanitarian Performance Where are
we now?

• 24th Biannual Meeting
• Berlin, 3rd December 2008

2
Various strands of ALNAP work are working towards
assessing system-wide performance

• Various components of RHA - evaluation
  synthesis, meta-evaluation (especially on joint
  evaluations) and themed chapters
• Facilitation of TEC and discussions on
  recommendations
• HPP - data mapping and exploratory analysis of
  how to assess system-wide performance

3
What have we learned from the RHA?

• Evaluation synthesis useful but, on its own, not
  able to assess performance
• Component parts of RHA are good but the final
  product is probably less than the sum of its
  parts
• Need to strengthen methodology and produce a more
  coherent whole

4
What have we learned from the TEC?

• joint evaluations better than single-agency
  evaluations in providing system-wide snapshot
• system-wide joint evaluations provide one off
  picture only
• utilisation and take up of recommendations very
  difficult to achieve in practice

5
What have we learned from HPP?

• Lots of data collected, but of different types,
  from different sources with different uses.
• Many methodological and conceptual difficulties
• Majority data gathered in needs assessment phase
• Very little effort given to seeking the views of
  affected populations/ recipients of aid

6
What have we learned from the Madrid biannual?

• Be realistic as to what can be achieved now, use
  existing evidence to assess performance, with a
  special emphasis on impact
• Explore the use of beneficiary surveys in
  assessment of impact and performance
• Develop a pilot to test these ideas
• Continue mapping and do not lose sight of
  developing a more precise way of assessing
  performance

7
What are we going to do?

• The three-track approach
•  
• Track one - fast trackState of the System
  pilot
• Track two - medium track Learn more about use of
  beneficiary surveys and impact assessment and
  feed this into future State of the System
  reports
• Track three - slow track Continue mapping and
  work on developing key performance indicators

8
Track One State of the System Report What is
it for?

• Overall goal is to assess overall humanitarian
  performance against agreed criteria
• Pilot will provide a base-line to track future
  performance

9
 What problems will we face?

• Analysing a system that is not strictly a system
  (i.e., not systematic)
• Lack of data relating to outcomes and indicators

10
How will we address the problems?

• a) Break down the system into different units of
  analysis. Disaggregate the data by looking at

• state of response in individual crisis
• state of response in particular sectors
  (clusters)
• state of response in particular categories
  natural disasters, wars, high profile
  crisis, neglected crises
• state of response in relation to types of
  actors UN, NGOs, donors, governments etc

11

• b) For each unit of analysis, performance will
  be analysed in relation to OECD-DAC criteria

• Relevance/Appropriateness
• Connectedness
• Coherence
• Coverage
• Efficiency
• Effectiveness
• Impact

12
c) Need to identify indicators to apply to
OECD-DAC criteria. For example

• was coverage adequate are resources adequate
• global funding against needs (CAP, beneficiary
  surveys)
• funding across sectors and emergencies
• staffing coverage in key areas
• and so on..

13
What do we want the report to tell us?

• emerging themes
• trends how has sector y or response in
  emergency x changed over time
• innovations and changes
• performance indicators if/when they exist
• perceptions of informed stakeholders about
  effectiveness, impact etc

14
What methods shall we use?

• Building on the RHA
•  
• key informant interviews (NGOs, donors,
  government) aiming for mix of HQ and field
• financial data analysis (OECD-DAC and FTS)
• mapping of global footprint and across current
  emergencies
• key informant survey polling opinion about
  performance to provide base line
• evaluation synthesis
• literature review

15
What now?

• establish peer review advisory panel
• undertake preliminary interviews/ consultations
  or input on scope and objectives of pilot
• design detailed methodology and research plan
  in inception report to be peer reviewed