Evaluating CDC HIV Prevention Programs: Guidance and Data System

1
MODULE THREE DEVELOPING COORDINATED SI/ME
PLANS INDICATOR REPORTING
2
MODULE 3

• Panel Chairs

Planning for SI/ME, John Encandela, ORC
Macro Indicator Planning Reporting, Greet
Peersman, CDC
3
RELATIONSHIP TOCONCEPTUAL FRAMEWORKS
Public Health ME Key
Tools Questions Components

• Assessment
• Planning
• __________________________________________
• Situation Analysis
• Response Analysis
• Stakeholder
• Needs
• Resource
• Analysis
• Collaboration
• Plans
• Program
• Development
• Data

• Situation/
• Response
• Assess
• Program
• Logic Model
• ME Logic
• Model
• Stakeholder
• Framework
• Indicator
• Framework

• How can
• ME plan-
• ning be used to
• guide re-
• sponse to the
• epidemic?

4
MODULE THREE DEVELOPING COORDINATED SI/ME
PLANS INDICATOR REPORTING
5
MODULE 3

• Panel Members

Erin Eckert (Measure) Mary Freyder
(USAID) Margaret Janes (Measure) Tom Scialfa
(Rwanda) Shirl Smith (Synergy) Festus Ukwuani
(USAID)
6

• Common Planning Terminology
• ME Plan Components
• Bare bones National ME Plan Components
• Project-Level ME Plan Components
• Tools for ME Planning
• Integrated Planning Cycle Applied to ME
• Stakeholder Frameworks and Logic Models
• Synchronizing Sub-National with National ME
  Planning
• Balancing M with E (staircase evaluability
  assessment)
• Additional Planning Issues
• Planning for Data Management
• Discussion

PLANNING FOR SI/ME
7
ME Planning

• ME Plan Components

8
BARE BONES OUTLINE FOR A NATIONAL ME PLAN

• Introduction of the composite of interventions
• (Background Information e.g., ME
  resourcesfinancial, human, other roles and
  responsibilities)
• Logic model/Results Framework
• Problem statement
• Expected outcomes/impacts
• Indicators
• Multi-year targets (measurable objectives)
• Operational definitions, sources, frequency of
  indicator data, method of verification/validation
  
• Frequency, format, and content of reporting on
  indicators
• Frequency, format, and content of reporting on
  interventions and activities

9
ADDITIONAL OR OPTIONS COMPONENTS OF A NATIONAL
ME PLAN

• Work Plan
• Other Types of Evaluation and Operational Research

10
SI/ME OFFICERS VIS-À-VIS NATIONAL ME PLAN

• Get the National AIDS Plan (NAP) and ME Plan and
  read them
• Assess for your understanding, not necessarily
  for critique unless asked
• Is the ME Plan consistent with NAP?
• Does ME Plan have bare-bones components?
• Is the public health problem operationally
  defined? (NAP)
• Are the expected outcomes reasonable/attainable
  and relevant to the problem?
• Are the indicators appropriate? (Next session)
• Are targets or objectives measurable (meet the
  SMART criteria)?
• Plan for reporting on indicators sound and
  timely?
• Plan for reporting on activities and
  interventions reasonable and timely?

11
ME PLAN COMPONENTS

• Typically understood in relation to the
  Program Planning Cycle

12
Integrated Planning Cycle for HIV/AIDS Programs
ASSESSMENT
1
EVALUATION
STRATEGIC PLANNING
2
5
3
4
DESIGN
MONITORING
13
Integrated Planning Cycle for HIV/AIDS Programs
ASSESSMENT What is the nature of the HIV/AIDS
problem?
1
EVALUATION How do I know that the strategy is
working? How do I judge if were making a
difference?
STRATEGIC PLANNING What primary objectives should
my organization pursue to address the problem?
2
5
3
4
DESIGN What strategy, interventions and
approaches should my organization use to achieve
our objectives?
MONITORING How do I know the activities are being
implemented according to our design? How do I
fine-tune and customize to our own setting? How
do I know if outcomes are being achieved or the
program is situated to achieve outcomes?
14
Strategic Planning for ME and SI Components of
HIV/AIDS Programs Integrated Planning Cycle for
Strengthening ME and SI Systems
ASSESSMENT What is the nature of the ME/SI
challenge?
1
EVALUATION How do I know that the strategy is
working? How do I judge if the ME/SI
Intervention is making a difference?
STRATEGIC PLANNING What primary objectives should
my organization pursue to address this challenge?
2
5
3
4
DESIGN What strategy, interventions and
approaches should my organization use to achieve
these ME and SI priorities?
MONITORING How do I know the activities are being
implemented according to our design? Who do I
need to participate in this process? How do I
know if outcomes are being achieved or the
program is situated to achieve outcomes?
15
ME Planning

• Tools for ME Planning

16
Rwanda Conceptual model host government, donor
and NGO agencies and their linkages for executing
and coordinating interventions including ME and
other SI
Government of Rwanda
Other Ministries
MOH
Other MOH Divisions
Natl programs - TB/Leprosy - Malaria HIV
Ref. Lab
- Camerwa - NBTS
Central
Provincial
District Community
DISGASS
DHT
Clients, Service Providers, Community NGOs
17
(No Transcript)
18
Rwanda Framework of host government, donor and
NGO agencies and their linkages for executing and
coordinating interventions including ME and
other SI
19
Rwanda Framework of host government, donor and
NGO agencies and their linkages for executing and
coordinating interventions including ME and
other SI
20
Rwanda HIV/AIDS National Level - Coordinating
entities

• National Committee for the Fight Against AIDS
  (CNLS) Coordinates the countrys overall
  national strategic plan and multi-sectoral
  HIV/AIDS response. The CNLS structure includes an
  Executive Secretary, Social Mobilization Unit,
  Planning and Coordination Unit, ME Unit and two
  program management units to manage the Global
  Fund and World Bank MAP grants.
• Minister of State for HIV/AIDS and other Major
  Epidemics Oversees TRAC, the National TB and
  malaria programs and other MOH divisions to
  coordinate the MOH response to HIV/AIDS and other
  epidemics. Oversees prevention, care and
  treatment programs.
• Training Research AIDS Center (TRAC) Provides
  technical leadership and national coordination
  and execution of ARV treatment, facility-based
  prevention (VCT, PMTCT), HIV surveillance,
  operations research and national HIV/AIDS-related
  (health sector) ME.
• HIV Cluster GOR and donor agency HIV/AIDS
  coordinating committee. Jointly chaired by the
  Ministry of State and USAID for the purpose of
  coordinating donor support.
• PEPFAR Steering Committee Principal coordinating
  mechanism between the US mission and GOR to
  develop the five-year strategy, prepare annual
  country operation and implementation plans, and
  monitor national progress towards PEPFAR 2-7-10
  goals
• Country Coordinating Mechanism (CCM) Group of
  representatives from various multilateral,
  bilateral and NGO entities assembled for Global
  Fund activities

21
Rwanda

• Principal HIV/AIDS (externally financed) projects
• U.S. Presidents Emergency Plan For AIDS Relief
  (PEPFAR)
• The Global Fund (GF) Support prevention and
  treatment programs
• The World Banks Multi-Country AIDS Program
  (MAP) Support prevention and treatment programs
  through line Ministries including community based
  activities.

22
SYNCHRONIZING SUB-NATIONAL AND NATIONAL ME
PLANNING
23
National HIV/AIDS Programme
Sub-national projects/activities
24
Integrated Planning Cycle for HIV/AIDS Programs
ASSESSMENT What is the nature of the HIV/AIDS
problem?
1
EVALUATION How do I know that the strategy is
working? How do I judge if were making a
difference?
STRATEGIC PLANNING What primary objectives should
my organization pursue to address the problem?
2
5
3
4
DESIGN What strategy, interventions and
approaches should my organization use to achieve
our objectives?
MONITORING How do I know the activities are being
implemented according to our design? How do I
fine-tune and customize to our own setting? How
do I know if outcomes are being achieved or the
program is situated to achieve outcomes?
25
Logic Model Sub-national level
IMPACTS
ACTIVITIES
OUTPUTS
OUTCOMES
INPUTS
AGENCY
National Level ME
Sub-National ME
Sub-national ME activities contribute to
national ME frameworks. However, Sub-national
activities should be designed with their own
logic framework
26
Logic Model Reporting
OUTPUTS
IMPACTS
ACTIVITIES
OUTCOMES
INPUTS
AGENCY

• Sub-National ME
• Reporting systems
• Frequent (monthly)
• Timely
• detailed

• National Level ME
• Reporting systems
• Periodic (yearly)
• Big Picture

27
VCT Program Implementation Logic Model
IMPACTS
ACTIVITIES
OUTPUTS
OUTCOMES
INPUTS
AGENCY

Funding


HIV transmission rates decreased
clients received pre-test counseling,
clients received HIV tests, clients received
results post-test counseling
Provide pre-test counseling, HIV testing, and
post-test counseling to all clients tested

Counseling and Testing Personnel
National (Governmental) Multi-lateral, Bilateral,
NGOs
HIV incidence decreased
Risk behaviors decreased
VCT protocols, guidelines, and training
documents
Clients (HIV and -) develop adhere to
personalized HIV risk-reduction treatment
strategy
HIV test kits and materials
28
VCT Logic Model Sub-national level
IMPACTS
ACTIVITIES
OUTPUTS
OUTCOMES
INPUTS
AGENCY
Training of counselors
Funding
Number of counselors trained
NGO
HIV transmission rates decreased
clients received pre-test counseling,
clients received HIV tests, clients received
results post-test counseling
Assure adequate stock of test kits/supplies
Counseling and Testing Personnel
HIV incidence decreased
Risk behaviors decreased
Develop/implement facility reporting system
VCT protocols, guidelines, and training
documents
Reporting forms completed and submitted
Clients (HIV and -) develop adhere to
personalized HIV risk-reduction treatment
strategy
HIV test kits and materials
29
SI/ME OFFICERS VIS-À-VIS SUB-NATIONAL LEVEL,
PROJECT-SPECIFIC ME PLANS

• How involved can/should SI/ME officers be
  involved in sub-national, project-specific ME
  planning, or, how much of myself can go around?
• Discussion

30
Sub-national ME Take home message

• The scope of the program/project/activity may be
  reduced

but the ME thought process is the same
31
LOGIC MODEL FOR ME
Problem Statement Multiple parallel ME systems
(parallel to National HIS) results in costly,
inefficient, duplicative, and chaotic and
inefficient reporting.
OUTPUTS
ACTIVITIES
OUTCOMES
IMPACTS
INPUTS
Parallel ME systems better integrated national
HIS

of parallel systems is reduced

Evidence-based decision making is driving program
design, implementation, and ongoing program
improvement
Feedback loops are in place and functioning


32
Balancing M with E

• Staircase approach (generally, dont move to
  next set of questions without having answered the
  first set)
• Evaluability assessment of programs
• Beginning, more M than E with balance over time
• Questions approachhave you answered the
  questions at the lower steps of the staircase?
  If not, you probably need to answer these first
  before moving on to the next level of questions.

33
A Public Health Questions Approach to Unifying
AIDS SI/ME
Are collective efforts being implemented on a
large enough scale to impact the epidemic
(coverage impact)? Surveys Surveillance
Determining Collective Effectiveness
OUTCOMES IMPACTS MONITORING
Are interventions working/making a
difference? Outcome Evaluation Studies
Monitoring Evaluating National Programs
OUTCOMES
Are we implementing the program as planned?
Outputs Monitoring
OUTPUTS

What are we doing? Are we doing it right?
Process Monitoring Evaluation, Quality
Assessments
ACTIVITIES
What interventions and resources are needed?
Needs, Resource, Response Analysis Input
Monitoring
INPUTS
Understanding Potential Responses
What interventions can work (efficacy
effectiveness)? Are we doing the right
things? Special studies, Operations res.,
Formative res. Research synthesis
What are the contributing factors? Determinants
Research
Problem Identification
What is the problem? Situation Analysis and
Surveillance
34
EVALUABILITY ASSESSMENT

• Assessing whether programs/projects are ready to
  be evaluated.
• Stepwise series of questions (see page 74-75 of
  GAP Field Guide for further information).

35
Strategic Planning for ME Setting Realistic
Expectations
Monitoring and Evaluation Pipeline
All
Most
Some
Few
of Projects
Impact Monitoring/Evaluation
Outcome Monitoring/ Evaluation
Process Evaluation
Input/Output Monitoring
Supplemented with impact indicators
from surveillance data.
Levels of Monitoring Evaluation Effort
Adaptation of Rehle/Rugg ME Pipeline Model, FHI
2001