Monitoring and Evaluation: Frameworks

1
Monitoring and EvaluationFrameworks
2
Learning Objectives

• At the end of the session, participants will be
  able to
• Identify and differentiate between conceptual
  frameworks, results and logical frameworks, and
  logic models
• Design goals and objectives for specific
  intervention programs
• Design frameworks for specific intervention
  programs
• Discuss how frameworks are used for ME planning

3
Module Outline

• Introduction to frameworks
• Conceptual Frameworks
• Goals and objectives
• Results and Logical Frameworks
• Logic models
• Exercises developing frameworks
• Interface between frameworks and ME planning
• Work on group project

4
Why are Frameworks Useful?

• Designing frameworks assist developing
• Clearly understood program/project goals and
  measurable, long-term, short-term, and
  intermediate objectives
• Clearly defined relationships between
  program/project inputs, processes, outputs, and
  outcomes, and between program/project activities
  and the external context (environmental factors)
• Sound implementation and ME plans

5
Types of Frameworks

• Conceptual frameworks
• Results frameworks
• Logical frameworks
• Logic models

6
Conceptual Frameworks(a.k.a., research or
theoretical frameworks)

• Diagram that identifies and illustrates the
  relationships between all relevant
• systemic, organizational, individual, or other
  salient factors that may influence
• program/project operation and the successful
  achievement of program or project
• goals.
• ME Purpose
• To show where program fits into wider context
• To clarify assumptions about causal relationships
• To show how program components will operate to
  influence outcomes
• To guide identification of indicators
• To guide impact analysis (causal pathways)
• Similar frameworks
• Proximate Determinants

7
Conceptual Frameworks
8
Conceptual Frameworks
National Tuberculosis (TB) Program

• External Factors
• Demographic
• (age, gender)
• Socio-economic (education, occupation)
• Environmental
• (war, weather)

TB infection

• Health Systems (DOTS)
• Availability
• Access
• Utilization
• Case Detection

• TB Morbidity
• Prevalence
• Incidence
• HIV co-infection
• MDR-TB

TBmortality

• Program Factors
• Political commitment
• Donor involvement
• National TB program

• Co-morbidity
• HIV
• Malnutrition
• Alcoholism
• Diabetes

• TB Knowledge
• Stigma
• Treatable
• Symptoms

Source MEASURE Evaluation 2004 (draft)
9
Conceptual Frameworks
Causes of malnutrition in society
Outcomes
Immediate causes
Underlying causes at household/ family level
Basic causes at societal level
Source UNICEF, State of the Worlds Children,
1998
10
Activity

• Get into groups of 4 or 5
• Develop conceptual framework for utilization of
  family planning/maternal and child health
  services
• Identify factors that influence health service
  utilization
• Arrange factors into a conceptual framework
• After 30 minutes, a member of each group will
  share the framework with all participants

11
Goals and Objectives

• Goal a broad statement of a desired, long-term
  outcome of the program
• Objectives statements of desired, specific,
  realisticand measurable program results
• SMART
• Specific identifies concrete events or actions
  that will take place
• Measurable quantifies the amount of resources,
  activity, or change to be expended and achieved
• Appropriate logically relates to the overall
  problem statement and desired effects of the
  program
• Realistic Provides a realistic dimension that
  can be achieved with the available resources and
  plans for implementation
• Time-based specifies a time within which the
  objective will be achieved

Source GAP 2003
12
Formulating Program Objectives
To action
The specify what knowledge, attitudes, skills, behaviors
Among specific population or segment
From to from baseline to desired level
or by x percent
or to specific level
By time frame
13
Goals and Objectives

• National AIDS Committee (NAC) 2005 Strategic Plan
• Goal Prevent the spread of HIV epidemic and
  minimise its impact on nation by 2009
• Objective 1 Reduction of HIV prevalence in
  nation
• Objective 2 Improved health quality of life
  of people infected affected by HIV/AIDS
• Objective 3 Strengthened capacity of NAC
  stakeholders to respond to the HIV/AIDS epidemic
  at all levels through improved research, ME and
  improved management coordination

14
Goals and Objectives Activity
Outcomes
Immediate causes
Underlying causes at household/ family level
Basic causes at societal level
Source UNICEF, State of the Worlds Children,
1998
15
Results Frameworks

• Diagrams that identify steps or levels of results
  and illustrate the causal relationships linking
  all levels of a programs objectives.
• Purposes
• Provides a clarified focus on the causal
  relationships that connect incremental
  achievement of results to the comprehensive
  program impact
• Clarifies project/program mechanics and factors
  relationships that suggest ways and means of
  objectively measuring the achievement of desired
  ends
• Other terms used
• Strategic frameworks

16
Results Frameworks
Source David Marsh, 1999
17
Results Frameworks
Tuberculosis (TB) Control Programs
SO1 Increase tuberculosis case detection rate to
70
IR1 Increased availabilityof quality services
IR2 Increased demand for quality services
IR2.1 Customerknowledge of TB improved
IR1.1 Services increased
IR1.2 Practitioners skillsand knowledge
increased
IR2.2 Social support for TB practices increased
IR1.3 Improved program management
Source MEASURE Evaluation, 2004 (draft)
18
Results Frameworks
Donor/USAID Reproductive Health Program
SO1 Increased Utilization of Family
Planning/Reproductive Health Services
IR1 Strengthened sustainability of FP/RH Program
IR2 Expansion of high quality FP/RH services in
the public and private sectors
IR2.1 Increased availability of postpartum and
postabortion FP services
IR1.1 Improved policy environment for the
provision of FP/RH services in the public and
private sectors
IR2.2 Increased accurate knowledgeof clients
about modern methodsand FP services
IR1.2 Strengthened NGO advocacy for FP program
IR2.3 Improved job performance of health
providers, trainers, and administrators
Source USAID/Turkey Performance Monitoring Plan,
1998-2001
19
Results Frameworks
FIVE-YEAR GOAL Foundation Established
for Reducing Famine Vulnerability, Hunger and
Poverty
SO 14 Human capacity and social resiliency
increased
IR 14.1 Use of high impact health, family
planning, and nutrition services, products, and
practices increased
IR 14.2 HIV/AIDS prevalence reduced and
mitigation of the impact of HIV/AIDS increased
IR 14.3 Use of quality primary education
services enhanced
IR 14.2.1 Reduced risk behavior
IR 14.3.1 Community participation in the
management and delivery of primary education
services strengthened
IR 14.1.1 Community support for high impact
health interventions increased
IR 14.2.2 Mother-to-child transmission of HIV
reduced
IR 14.1.2 Availability of key health services
and products improved
IR 14.3.2 Planning, mgmt and monitoring and
evaluation for delivery of primary education
services strengthened
IR 14.2.3 Access to care and treatment for
people living with HIV and AIDS increased
IR 14.1.3 Quality of key health services improved
IR 14.2.4 Care and support for orphans and
vulnerable children expanded
IR 14.3.3 Quality of primary education improved
IR 14.1.4 Health sector resources and systems
improved
IR 14.3.4 Equitable primary education services
strengthened
IR 14.2.5 A more supportive environment for
responding to HIV/AIDS
Source USAID/Ethiopia Integrated Strategic Plan,
February, 2004 (revised draft)
20
Logical Frameworks

• Present a standardized summary of the project and
  its logic.
• Purposes
• Summarizes what the project intends to do and how
• Summarizes key assumptions
• Summarizes outputs and outcomes that will be
  monitored and evaluated
• Other terms used
• Logframe matrix

21
Logical Frameworks
Project Description Performance Indicators Means of Verification Assumptions
Goal The broader development impact to which the project contributes - at a national and sectoral level. Measures of the extent to which a sustainable contribution to the goal has been made. Used during evaluation. Sources of information and methods used to collect and report it.
Purpose The development outcome expected at the end of the project. All components will contribute to this Conditions at the end of the project indicating that the Purpose has been achieved and that benefits are sustainable. Used for project completion and evaluation. Sources of information and methods used to collect and report it. Assumptions concerning the purpose/goal linkage.
Component Objectives The expected outcome of producing each component's outputs. Measures of the extent to which component objectives have been achieved and lead to sustainable benefits. Used during review and evaluation. Sources of information and methods used to collect and report it. Assumptions concerning the component objective/purpose linkage.
Outputs The direct measurable results (goods and services) of the project which are largely under project management's control Measures of the quantity and quality of outputs and the timing of their delivery. Used during monitoring and review. Sources of information and methods used to collect and report it. Assumptions concerning the output/component objective linkage.
Activities The tasks carried out to implement the project and deliver the identified outputs. Implementation/work program targets. Used during monitoring. Sources of information and methods used to collect and report it. Assumptions concerning the activity/output linkage.
22
Logical Frameworks
Taskforce on Communicable Disease Controlin the
Region X Tuberculosis
GOAL Reduced burden of TB to reach global average levels Further development of multi-drug resistant TB (MDR-TB) prevented   PERFORMANCE INDICATORS Notification rate B-1. Treatment outcome B-2. Prevalence of Multi-drug Resistance in new and previously treated TB patients MEANS OF VERIFICATION Annual notification reports (surveillance) B-1. Annual reports on outcome of treatment (cohort analysis) B-2. Periodic reports on surveillance of anti-TB drug resistance    ASSUMPTIONS   - A dual HIV/TB epidemic causing increase in TB incidence does not occur - Control of private practitioner and pharmaceutical sectors to prevent MDR - Prevalence of resistance to second line anti-TB drugs low enough at the outset so as not to seriously compromise treatment success ratio
23
Logical Frameworks
Taskforce on Communicable Disease Controlin the
Barents and Baltic Sea Regions Tuberculosis
PURPOSE   Implementing cost-effective measures for the prevention and control of TB operating within civil and penitentiary health services in the Task Force area PERFORMANCE INDICATORS   1. Coverage of TB programmes in line with international recommendations. 2. Proportion of patients defaulting out of patients treated. 3. Proportion of previously treated cases among all cases. 4. Proportion of patients on ambulatory treatment out of all patients treated. MEANS OF VERIFICATION Annual reports Annual reports National / local annual notification reports (surveillance) 4. Annual record reviews during site visits (consecutive series of patients) ASSUMPTIONS   - Stable political situation, sustained political commitment and financing - Sufficient numbers of competent health care personnel in the government sector
24
Logical Frameworks
Taskforce on Communicable Disease Controlin the
Barents and Baltic Sea Regions
Tuberculosis(only one output and related
activities shown here)
OUTPUTS   8. Measures to increase awareness of TB and its treatment among all members of the community developed and tested PERFORMANCE INDICATORS     8.1. Number of pamphlets / posters printed and distributed annually 8.2. Awareness of TB among target groups MEANS OF VERIFICATION 8.1. Material produced / distributed 8.2. KAP or other surveys (before/after) ASSUMPTIONS   Relevant persons motivated to participate Professional interest, sufficient financing Target groups interested in (their) health and able to participate
 ACTIVITIES   8.1 Identify groups at risk for TB 8.2 Develop advocacy material suitable for all target groups (not only risk groups) 8.3 Organize health education directed at all target groups 8.4 Involve the mass media  INPUTS   MEANS OF VERIFICATION   Financial management reports ASSUMPTIONS
25
Activity

• Return to small groups from previous activity
• Develop goals and objectives for a program aimed
  at increasing utilization of FP/MCH health
  services
• Develop a results or logical framework for this
  program
• In 45 minutes, a member of each group will share
  the goals, objectives and framework

26
Logic Models

• Diagrams that identify and illustrate the linear
  relationships flowing
• from program inputs, processes, outputs, and
  outcomes. Inputs or
• resources affect Processes or activities which
  produce immediate
• results or Outputs, ultimately leading to longer
  term or broader
• results, or Outcomes.
• Purposes
• Provides a streamlined interpretation of planned
  use of resources and desired ends
• Clarifies project/program assumptions about
  linear relationships between key factors relevant
  to desired ends
• Other terms used
• ME Frameworks, Logical Frameworks

27
Logic Models Training
INPUT
PROCESS
OUTPUT OUTCOME IMPACT
Develop clinical training curriculum
Conduct training events
Practitioners trained in new clinical techniques
Increase in clients served by (newly) trained
providers
Declining morbidity levels in target population
28
Logic Models
Voluntary Counseling and Testing (VCT)
Problem Statement HIV infection rates continue
to rise, underscoring the importance for people
to know their serostatus, develop personalized
risk-reduction strategies, and access care and
treatment services.
OUTPUTS
PROCESSES
OUTCOMES
IMPACTS
INPUTS
Supervision and training for VCT personnel
Risk behaviors decreased
Financial Resources
Clients (HIV and -) develop adhere to
personalized HIV risk-reduction
HIV transmission rates decreased
People know their HIV status
Demand for Services
Provide pre-test counseling
Increase in care, prevention, and treatment
services for HIV, HIV-, and discordant couples
Infrastructure
HIV Clients develop adhere to personalized HIV
care, support, and treatment plans
Provide HIV testing
HIV incidence decreased
HIV people are referred to appropriate services
VCT MIS
Provide post-test counseling
VCT protocols, guidelines, and training
documents
Refer HIV clients to appropriate services
(PMTCT, care and support, treatment)
HIV morbidity mortality decreased
Counseling and Testing Personnel
Health outcomes of HIV improve
Client records are available
Continuity of care is available
HIV test kits
Maintain patient records
Service reports are produced
Program and services are improved
Referral system for prevention Tx services
Complete reporting requirements
29
Results Framework
Tuberculosis (TB) Control Programs
SO1 Increase tuberculosis case detection rate to
70
IR1 Increased availabilityof quality services
IR2 Increased demand for quality services
IR2.1 Customerknowledge of TB improved
IR1.1 Services increased
IR1.2 Practitioners skillsand knowledge
increased
IR2.2 Social support for TB practices increased
IR1.3 Improved program management
Source MEASURE Evaluation, 2004 (draft)
30
Logic Model
Portion of model for tuberculosis control
relating to increasing demand for quality
services

• INPUT
• Human and financial resources to develop and
  print educational brochure

• PROCESS
• Distribute brochure to health facilities
• Meet with physicians to promote distribution of
  brochure

• OUTPUT
• Brochure distributed to clients of facilities

• OUTCOME
• Increased customer knowledge of TB transmission
  and treatment
• Increased demand for quality TB services

• IMPACT
• Decreased TB infection, morbidity and mortality

31
Activity

• Return to small groups from previous activities
• Develop logic models for one of the activities of
  your program to increase utilization of FP/MCH
  health services
• After 20 minutes, a member of each group will
  share one of the logical frameworks

32
Frameworks for ME Planning

• Purposes
• clarifying assumptions, goals, and
  interrelationships between factors relevant to
  the project or program
• defining objectives
• selecting activities
• defining levels of performance and desired
  results in terms of planned activities and
  realistic, objective impacts
• Monitoring and evaluation plans incorporate
• program managers assumptions and objectives, in
  a given context
• a schematic design displaying the directional
  linkages between key program elements and/or
  planned results, and other relevant factors

33
Summary of Frameworks
Type of Framework Brief Description Program Management Basis for Monitoring and Evaluation
Conceptual Interaction of various factors Determine which factors the program will influence No. Can help to explain results
Results Logically linked program objectives Shows the causal relationship between program objectives Yes at the objective level
Logical Logically linked program objectives, outputs, and activities Shows the causal relationship between activities and objectives Yes at the output and objective level
Logic model Logically links inputs, processes, outputs, and outcomes, Shows the causal relationship between inputs and the objectives Yes at all stages of the program from inputs to process to outputs to outcomes/ objectives
34
References

• AusGuide. The Logical Framework Approach.
  http//www.ausaid.gov.au/ausguide/ausguidelines/1-
  1-1.cfm
• Bertrand, Jane T., Magnani, Robert J, and
  Rutenberg, Naomi, 1996. Evaluating Family
  Planning Programs, with Adaptations for
  Reproductive Health, Chapel Hill, N.C. The
  EVALUATION Project.
• Global AIDS Program. (2003) Monitoring and
  Evaluation Capacity Building for Program
  Improvement Field Guide, Version 1. U.S. Centers
  for Disease Control and Prevention, Atlanta, GA.
• Marsh, David. 1999. Results Frameworks
  Performance Monitoring. A Refresher by David
  Marsh (ppt) http//www.childsurvival.com/tools/Mar
  sh/sld001.htm
• Tsui, Amy. 1998. Frameworks (ppt). Presented at
  the Summer Institute, University of North
  Carolina, Chapel Hill.
• Tsui, Amy. 1999. Frameworks (ppt). Presented at
  the Summer Institute, University of North
  Carolina, Chapel Hill.
• UNICEF. 1998. State of the Worlds Children.

35
Group Projects

• Form groups for your group projects
• For your project, develop
• Goals and objectives
• Conceptual framework
• One other framework results, logical or logic
  model