Monitoring and Evaluation: Meeting Needs and Expectations

1
Monitoring and Evaluation Meeting Needs and
Expectations

• Ian Askew
• Cindy Geary

2
Presentation Overview

• Conceptual framework for M E
• Monitoring programme performance
• Evaluating programme effectiveness
• Challenges particular to ME for youth
  RH/HIV/AIDS programs

3
Conceptual Framework

• PROGRAMME
• (Programme design)
• Resources
• ?
• Activities
• ?
• Outputs
• ?
• Outcomes

• M E
• (Needs Assessment/Baseline)
• Cost analysis
• ?
• Process documentation
• ?
• Monitoring
• ?
• Evaluation

4
Needs Assessment

• Critical step prior to/during programme planning
• Gather information from and with youth and
  stakeholders
• Contribute information to participatory process
  for designing / refining activities
• If data collected on desired outcomes, can serve
  as baseline for evaluation
• Develop plans for ME

5
Cost Analysis andProcess Documentation

• Often forgotten elements of ME
• Process documentation critical for understanding
  how to implement activities when sustaining and
  scaling-up
• Cost analysis critical for understanding
  resources needed for sustainability and for
  scale-up if activities are feasible
• Plan for both and include in ME budget

6
Monitoring Programme Outputs

• Also known as process evaluation or performance
  monitoring
• Describes and measures outputs produced by
  programme activities
• Provides regular indication of programme progress
  for identifying strengths and weaknesses and
  making adjustments
• Requires routine recording and reporting of
  activity outputs, with regular review and use of
  information generated

7
Typical Output Indicators for Youth RH/HIV/AIDS
Programmes
8
But also need to indicators

• for monitoring activities to meet non-RH/HIV
  needs
• for monitoring activities to enhance livelihoods
• for monitoring activities for other structural
  interventions

9
Undertaking Output Monitoring and Process
Documentation

• Monitoring Outputs
• Standard formats for recording activity outputs
  as they happen
• Reporting summaries routinely to supervisor
• Synthesis of reports
• Feedback to
• Managers
• Staff
• Youth / stakeholders

• Process Documentation
• Observation of activities / services
• Reports from regular supervisory visits
• Occasional visits by external experts
• Focus groups with
• Staff
• Youth
• Stakeholders

10
Evaluating Programme Outcomes

• Assesses programme effectiveness in achieving
  pre-determined desired outcomes
• Outcomes expected to be directly caused by
  activities
• Measures changes among beneficiaries
• Knowledge increase
• Attitude changes
• Behaviour / practice sustained / improved
• Reduced vulnerability
• Experimental or quasi-experimental designs

11
Typical Outcome Variables for Youth RH/HIV/AIDS
Programmes
12
But need indicators for evaluating

• effect of structural intervention programmes on
• Social structures
• Youth vulnerability
• economic
• social
• emotional
• effect of youth empowerment programmes

13
Undertaking Outcome Evaluations

• State programme objectives as expected changes
  among beneficiaries
• Hypothesise expected effect of programme
  activities on outcomes
• Logical argument (cause-effect)
• Theoretical model
• Evaluation must be able to make comparisons
• Before and after, and/or
• With comparison group (not participating)
• Questionnaire surveys among beneficiaries most
  commonly used to enable statistical comparisons
• Qualitative data important to understand why
  changes did or did not happen

14
M E Challenges
15
Addressing Differing Needs

• If programmes are to meet needs of specific youth
  sub-groups
• then ME must
• Monitor whether activities reach intended
  sub-group(s)
• Evaluate whether programmes have differential
  effects on each sub-group

16
Evaluating a Feasible and Sustainable Programme

• Although evaluation should be planned and
  budgeted before a programme begins
• outcome evaluation should not be conducted
  unless a programme has been demonstrated to be
  feasible and sustainable
• Through process documentation
• Through monitoring outputs
• Through cost analysis
• Determine in advance decision time, criteria and
  responsibility for decision

17
ME for programmes addressing vulnerability

• Structural interventions Livelihoods, social
  isolation, gender-based violence, education
• Hypothesising effects of programme activities on
  RH/HIV outcomes
• Measuring outputs and outcomes
• Sustaining and expanding pilot projects

18
Planning and budgeting for ME

• Good ME is not cheap, but done well it is
  critical for achieving effective programming
• Assign sufficient funding to ME to be able to
• Routinely measure and know of activity outputs
• Assess whether the programme has generated
  significant changes among beneficiaries

19
Undertaking Routine Monitoring and Timely
Evaluation

• Allow sufficient time for a participatory needs
  assessment and for a baseline survey before
  starting activities
• Ensure outputs are recorded and reported
  routinely, and feedback communicated regularly
• Allow sufficient time for the activities to have
  an effect
• Allow enough time to undertake the endline survey
  within the programme period

20
Participation in ME byBeneficiaries and
Stakeholders

• Participatory methodologies are critical for
  valid assessments of programme activities, and
  for getting beneficiary support for evaluation
  results
• BUT
• Funders often impatient for quick results
• Primarily interested in objective quantifiable
  measures
• Many evaluators do not have training in
  participatory methods

21
Availability of ME skills

• Still limited, especially for evaluation
• Reliance on external (costly) expertise
• Misunderstood by managers and donors
• Maintains separation of ME from programming

22
Ethics of Research Among Youth

• Respect for individuals
• Autonomy in participation
• Protect the vulnerable
• Protect from harm
• Maximise benefits from participating
• Balance in benefits and risks
• Designs with control / comparison groups
• Designing and testing unsustainable approaches

23
Legal and professional implications

• Who has authority to grant permission for access
  to children and adolescents as participants?
• What will happen if the child or adolescent
  reveals information indicating legal or illegal
  activities that could bring harm to the child,
  family, or community?
• What is the responsibility of researchers who
  uncover serious problems or needs, such as abuse,
  neglect, or malnutrition?

24
So.are we meeting our needs and expectations?

• ME critical to enable donors to know how to
  allocate funding
• ME critical to enable programmes to know whats
  working, why and to what effect
• ME critical to enable beneficiaries to know if
  programme is meeting their needs
• BUT
• Do donors, programmes and beneficiaries get the
  information they need, and use it to meet their
  expectations?