..Instead of writing reports or documenting experiences in ways that help us to learn, we spend a gr

1
Some thoughts on Monitoring and Evaluation..
2

• My brief provocative, challenging..
• Challenges we all face in ME
• Barriers to changing ME systems
• Ideas of potential ways to moving forward

3

• ..Instead of writing reports or documenting
  experiences in ways that help us to learn..
• we spend a great deal of time trying to please
  you and doing what you want
• Everjoice Win, Zimbabwean NGO, feminist (If it
  doesnt fit in the blue square its out! An open
  letter to a donor friend Christine 2004)

4

• What comes to mind when we think of
• Monitoring and Evaluation?
• A light-hearted look.

5
Data, bureaucracy
Is the monitoring or the evaluation used?
What learning is derived from it?
6
Formats/form filling
7
Processes that we do to them in order to
provide information upwards
8
Preset indicators monitoring against initial
project docs. Monitoring is often about
documenting what we and our partners expect to
happen..
9
Other words that spring to mind..

• Measurement
• not everything that can be counted counts
• and not everything that counts can be counted
  Albert Einstein
• Often NGOs are working on..
• Rights enabling people to access their basic
  rights hold decision makers accountable
• Control/choice
• Empowerment
• Confidence
• Freedom from abuse..

10
Challenges to monitoring and evaluation

• Introducing Learning cycles.
• Local NGO and community learning. Monitoring to
  influence change action?
• (b) NZ NGO using the learning from
  projects/programmes to influence on-going annual
  plans and funding decisions.
• Consolidated overview of impact?
• Assessments of problems/learning?
• ME influencing funding decisions?

CDRA James Taylor
11

• NGO quote..
• ..most of our organizations have effective means
  of ignoring findings which challenge us. ..
• This suggests that the challenge is as much to
  look at how open we are to dealing with difficult
  information about our performance, as it is to
  develop sophisticated methods.
• which produce results which we subsequently
  ignore..
• Head of Programmes,
• Oxfam Australia (2005)
• C.Roche Author of Impact Assessment for
  Development Agencies (1999)

12

• Challenge 2 - Introducing Participation
• Participation is often limited to communities
  participating in initial project ideas
• V. little participatory monitoring. V. little
  participatory assessment of ( and -) changes by
  community groups beneficiaries
• V. little participatory evaluation (in one
  evaluation report participatory processes
  limited to consultation with a handful of
  beneficiaries)
• Often hear the reality at the end evaluation

13
Challenge 3 better understanding of impact

• Hear a lot about activities - outputsbutV.
  little about impact
• Consolidated understanding of how (and if) the
  NZ NGO is making a difference
• Across the portfolio spending X then are we
  making a difference of Y?
• What is changing in peoples lives
• What are the problems and issues faced?
• What are we achieving against our strategy?
• Do our programmes/projects add up to a good way
  of tackling poverty?

14
Challenge 4 improving downward accountability
sharing knowledge/power

• Normally accountability systems for upward
  reporting? to Trustees, councils, donors
  because our system are devised to feed
  information upwards WE often fall into the
  pitfalls
• How often do we invite feedback? Invite
  criticism?
• How often do we ask what information our partners
  or people in communities may want from us?
• How often do we (or our partners) ask them what
  we could do differently to make their lives
  easier? And the work more appropriate?
• ..we can only respond to the questions raised if
  you can promise that you will not victimize us by
  canceling our project..
• (Concerns expressed by CBO member during AA
  Kenya PRRP 200)

15
What may be the barriers? 4

• The words e.g. Measurement
• ME often tries to simplify complex reality of
  social change often preset indicators
• Numbers/figures important but dont add up
  apples and pears
• Life is not linear Whats changed and what is
  significant is most interesting if it comes from
  beneficiaries themselves..

16
Possible barriers

• 2) Our own western conditioning/education our
  paradigm
• Ask for reports in English (but maybe 2nd or 3rd
  language)
• Maybe oral traditions (video, most significant
  change R.Davies, J.Dart)
• Rarely start with their concepts, but ask
  them to respond to our questions/expectations
  about what should be happening (James Taylor
  talks about starting from peoples own
  experience)
• Reality is often more complex.

17
Possible barriers..

• 3) Our own expectations of upward accountability
  to our funders to the NZ public..
• Rigour sometimes leads to inflexibility/reducti
  onism..
• Their expectations of us (DFID/PPA wanted
  fixed indicators rather than indicative
  outcomes).

18
Possible Barriers

• 4. Our own systemsways of working
• Financial systems Sometimes hard to get
  financial staff to work with programme staff
  (analysis of achievements v. costs each year).
  Sometimes work in silos
• Programme staff V busy. Sometimes hard to change
  practice.
• Often NGOs in NZ often dont have a dedicated
  person working on ME and impact assessment
• Changing ME systems ways of working takes a
  lot of hard work and effort

19
Potential ways forwardAssessment of contrasting
paradigms (Chambers Ideas for Devel 2005 Ros
Eyben Andrea Cornwall) Shifts in Development
thinking..
20
Shifts in Development thinking..
21
Final thoughts Participatory Review and
Reflections can be rigorous AA- Hear the
stories Understanding impact(participatory
Video Upper East Ghana Basket weavers for
fair trade market. - Exposure of the pressure of
the middle men (problems of being paid)-
problems they faced in getting materials for
weaving. - but full descriptions of the freedom
and the confidence that the work given them and
their ability to deal with the middle
men.Vietnam Ms Kieu Thi Tim of An Hai Commune
is a single parent with two girls. She was
formerly houseless and ranked as one of the
poorest in the village. With a small first loan
she built a home, which she repaid in a year.
With a second loan she started a breakfast bar
and began duck rearing. When asked what
difference the loans had made she replied ..I
can not compare the difference look at me
There was a deep sense of satisfaction in her
eyes and lots of confidence in her face.
22

• Uganda. Partner organisation working with street
  children who tell their stories
• A 13 year old girl narrates her story
• When my father died, I resorted to selling
  local brew at my sisters house in Jambula zone.
  I can not express the magnitude of the physical
  and mental torture I have gone through. My sister
  did not give me food despite the long hours of
  work serving drunkards, who instead of
  appreciating my work forced me into sex. I had no
  protection from my sister instead, she supported
  the men who harassed me because they paid her
  whenever they slept with me.
• When I got pregnant, my sister threw me out of
  her house I then went to my friends who gave me
  refuge until I delivered. The night I delivered,
  I was all by myself ... I screamed loudly,
  awakening the neighbours who found me lying in a
  pool of blood. The ordeal left me lame. I do not
  know if I will ever walk normally again
• Peoples testimonies put in context
• Numbers of street children
• Expenditure of partner NGO
• Long term changes to childrens lives and
  prospects.

23

• 1. How can we make (more) space for our partners
  to tell us about what is really going on? 2.
  How can we make (more) space to hear from
  beneficiaries about what difference the work is
  making to their lives from their own
  perspectives?
• 3. How can we give peoples analysis and
  perspective meaning by adding numbers and
  context?