AGF Working Group

1
Monitoring and evaluating capacity-building at
multiple levels
UNFCCC Meeting on monitoring and evaluating CB,
Rio, Brazil 6-7 November 2008 David
Watson Consultant
2
Scope of Presentation

• Background perspectives and approach
• The story so far ME of CB under UNFCCC
• How other global programmes tackle ME of CB
• The big picture ME of CB in international
  development
• Alternative paradigms esp. systems thinking
• Towards a tool-kit for practitioners

3
Background

• Sorry about late distribution of paper! (See
  Section X for a potted summary!)
• Personal points of departure building
  institutions and governance
• A touch of scepticism about ME practices
  pragmatism incrementalism practicality process
• Broadening debate constructively
• Not selling any particular approaches
• Cognisant of the seriousness of the challenges of
  slowing climate change, and of enhancing the
  effectiveness of CB to that end

4
Levels of Capacity and CB

• May be helpful to bear in mind two complementary
  dimensions
• Horizontal from individuals, through
  organisations, to social / national systems or
  networks
• Vertical from local, to regional, to
  national, to global.

5
Other Global Programmes Experience with ME of CB

• PRSPs
• GAVI / Roll Back Malaria
• UNAIDS
• Programme Based Approaches (PBAs)
• WFP
• Lessons pertinent to CB to address Climate Change

6
Lessons from other global progs.

• Several (GAVI and RBM) had specialised groups
  focussed on ME, plus Task Forces on specialised
  subjects (e.g. CB)
• At country level a Secretariat may have a ME
  Cell use existing ME structures where they
  exist keep it simple minimise number of
  indicators.
• Empirical evidence of enhanced capacities is
  often scarce
• Strategic incrementalism attracted PBA-ers
  attention pursue quick wins and peer-learning

7
..lessons continued

• Clarify objectives of CB, based on thorough needs
  assessments, as a basis for a simple results
  framework
• Try to avoid a missing middle indicators for
  the steps between outputs and outcomes
• Reflect on whether the ME process should feed
  more into national political processes, (engaging
  e.g. with parliaments, media, civil society) AND
  be a precondition of partnerships nationally and
  with the international community.
• Dont be naïve in your theory of political
  change participation is not enough.

8
A poignant quotation

• ME practices do not provide a framework for
  re-presenting (or making meaning from) the
  complex and multiple processes of institutional
  and individual learning and change that are
  sought in strengthening institutional capacity
• Source Unitar Challenges and Constraints in ME
  of Capacity Building presentation to first
  (Antigua) workshop Nov 2007

9
ME of CB recent insights

• Theme paper for ECDPM Study on Capacity Change
  and Performance 2003-8 (Baser Morgan Watson on
  Theme Paper)
• On ME of CD tended to agree with UNITAR!
• Often, performance improvement indicators are
  used as proxy for capacity increases
• not many e.g.s of capacity being monitored
• Accountability (to donors) the main driver
• Most public sector CB experience
  disappointingpolitical and institutional factors
  seem important
• Donors have a poor record on ME of CB
  ..including their own capacities and incentives

10
Capacitated organisations

• 1. ..carry out tasks effectively
• 2. ..engage, decide, act
• 3. ..relate, attract resources support
• 4. ..adapt and self-renew
• 5. ..balance diversity, and build coherence
  across the organisational system
• Capacity that emergent combination of
  individual competences and collective
  capabilities which enables a human system to
  create public value

11
..and implications for ME of CD..?

• Agree on the nature of capacity to be monitored!
• ..need to focus on more complex issues
• Information on change and progress should not be
  sent up but shared internally for purpose of
  learning
• Pay more attention to the contextand
  inter-relationships in capacity processes..
• to the changes taking place (intended or not) and
  their contributions to capacity
• ..i.e. need more participative approaches to ME
  and to learning and reflection..

12
Alternative Paradigms to ME (of CB)

• Critical reactions to monitoring abilities to
  performand meeting pre-determined objectives
• Wheatley measurement anathema (when looking
  at human behaviour)
• Cause and effect logic inapplicable
• ..change processes are complex..involve
  inter-dependenciesand are not linear or
  stable, nor necessarily visible
• ..but may well be long-term, and more political
  than technical

13
Systems Thinkings relevance

• Growth in interest in ST as an analytical
  framework for development and natural resource
  management
• Explicitly examined as part of ECDPM study helps
  address implications mentioned
• Human institutions seen as complex adaptive
  systems e.g. climate!
• See Box 1 for a summary of basic concepts
• Mess problems unbounded in scope, time and
  resources no clear agreement on optimal solution
  or how to achieve it goals and strategies are
  contested perspectives differ no clear cause
  and effect uncertainty prevails.

14
What about Capacity?

• Capacity is an emergent property of human
  systems (ECDPM Study)
• (Emergent properties are those which have no
  meaning in terms of parts making up the whole
  system. They are the outcome of system behaviour
  or synergy.)
• Two examples ENACT Jamaica (Box 2)
• And IUCN Asia (Box 3)

15
ENACT IUCN

• Formal performance monitoring system abandoned
• Empowerment of frontline staff for rapid response
• Absence of a model to assess performance
• Let partners adapt and adopt measures
• Donor modified its approach..more
  learning-friendly faced with diversity

• Unusually diverse membership
• Flexibility demonstrated by funding agencies
• Permitted experimentation, innovation and
  creativity
• Evolution of IUCN learning processes
• Teaming process
• Ownership by governments IUCN credibility /
  legitimacy
• CB continuous process no road map, only a goal

16
Cases commonalities c.f. ST

• Identification and recognition of goals
• Emphasis on values to be reflected
• Clarity and awareness of mission amongst clients
  too
• Leadership encourages experimentation
• Opportunities for learning from experience
  self-assessment and stories of positive
  experience or changes or errors
• Flexibility to adapt (thru new skills-building)
  to new needs / priorities, thru OJT hands-on
• Informality of ME systems responsive to needs
  of clients / network members
• Ability to learn from experience is crucial

17
BUT Reductionist approaches are still
relevantwhere

• It is possible to define required capabilities
  unambiguously and specifically
• ..and to assess existing ones gap
• Therefore easy to define indicators
• Where stakeholders able and willing to define
  their shortfalls and sign up
• Incentives exist to improve performance
• Leadership, and all above combine into
  ownership
• BUT ..this combination of circumstances is rare!
  (e.g. public financial management IMF and WB
  forged consensus)

18
Towards a Tool Box for ME of CD in Climate
Change

• Principles pragmatism acknowledge weaknesses in
  all ME systems main aim sense-making
• Seek out what ME exists, and what works,
  already
• Acknowledge that the best ME systems are
  customised adapted by participants, based on
  local conditions

19
Some existing frameworks

• GEF Resource Kit on ME
• GEF Indicators for 5 key capacities
• Engagement generate access to and use of
  information policy and legislation development
  management and implementation monitor and
  evaluate
• Scoring / indicator system for each
• Obliges actions / next steps / link to outcomes
  
• UNEP Lessons Trees
• Trying to improve quality of learning (especially
  about common problems) and application of
  learning to future programmes

20
Self-Assessment

• Examples from CB in research and development
  organisations
• SA workshops after applying qualitative and
  quantitative tools
• Managers, staff and stakeholders identify
  strengths and weaknesses, and set new directions
  .Advantages
• (1) those with knowledge of and interest in the
  organisation gain in-depth insights whats
  working and why..where improvements needed
• (2) Well-prepared to address the changes needed
• E.g. ME of past CB in Mekong Farming RD Systems
  Institute (Box 4)
• Preparation of work stories on past CB efforts
• interviews with key staff on changes and
  challenges

21
A Balanced Approach to ME of CB

• Framework generated to address ME of CB a la
  ECDPM Study dimensions of Capacity
• Piloted in PNG legal and judicial reform
• Time-consuming and exhaustive but national
  practitioners did find it helpful in
  conceptualising all dimensions of capacities
  being built up.
• See extra handout

22
Appreciative Enquiry in Formative Evaluation

• Evaluators form more of an understanding of the
  political, cultural and historical landscape
• Encouraging organisations to develop their
  relationships with primary stakeholders
• Community Development Corporations in US
• Seeking out what enabled effectiveness, and what
  hindered it
• Evaluator regularly listening to stories
• Regular visits and sharing of reports
• Recipients able to co-create the initiative and
  develop OWN capacity for assessment
• Evaluator chosen by NGO not donor NGO employed
  not as expert but for its ability to learn
  collectively

23
Most Significant Change (MSC)

• First applied in evaluation of a complex RD
  programme in Bangladesh
• Process managers identify domains of change which
  are important to evaluate
• Stories (descriptions of changes deemed
  significant with reasons why significant?)
  periodically collected from stakeholders
• Analysed and filtered up thru committees
• Criteria for choosing stories are collated and
  fed back to stakeholders
• Final selection made (annually?) with reasons
• Circulated to all
• Site visits to check deepen understanding of
  changes

24
MSC features

• Focuses attention and direction of work in
  programme towards valued directions
• Dialogue and deliberations crucial
• Takes place over time responsive to changing
  contexts
• Policy makers, funders, field engaged looking at
  the value of changes
• Stories help all relate to information
• Non-experts (the story-tellers) involved in
  evaluation
• Dialogue based on real experience and concrete
  outcomes not abstract indicators
• MSC positively evaluated as technique in Laos
  (Willetts 2004)

25
Annotated Bibliography

• ..use as a part of the tool-kit!
• Sections on
• Climate Change-related sources
• Broader CB literature and ME
• Other Global Programmes Materials
• Civil Society CB and ME
• Systems Thinking and Complexity literature and
  ME examples

26
Summary why ST has potential for ME of CB in
Climate Change

• Avoids pitfalls of logical framework in what is
  an amorphous field CB
• Emphasises clarity of objective-setting, AND of
  learning collectively from reality
• Generates, relates to, and values stories
• Potential to enhance team-work and
  inter-relationships of hitherto disparate
  institutions and groups in their ecosystem
  context
• Climate change context politically-charged,
  formal and informal, amenable to negotiation
• Evidence indicates ST ME approaches can work
  for and strengthen social change (Guijt 2007
  IDS)