Monitoring and Evaluating Gender-based Violence Prevention and Mitigation Programs Final Anastasia J. Gage Melissa Dunn

1
Monitoring and Evaluating Gender-based Violence
Prevention and Mitigation ProgramsFinalAnastasi
a J. GageMelissa Dunn
June 24, 2009
MODULE II
2
Learning Objectives

• By the end of this session, participants will be
  able to
• Differentiate between monitoring and evaluation
• Write goals and smart objectives for GBV programs
• Design a logic model for a GBV program
• Identify criteria for indicator selection and
  information sources for GBV indicators
• Discuss factors to consider when choosing an
  evaluation design

3
At Least 4 Different Categories of GBV
Interventions

• Community mobilization
• Behavior change communication
• Service delivery
• Laws and public policy

4
ME Challenges of GBV Programs (1)

• Hard to separate contribution of multiple
  strategies
• Lack of investment in rigorous evaluation
• Limited evidence of effectiveness
• Non-standardized GBV definitions and indicators
• Enhancing comparability of data
• Climate of urgency around results but behavior
  change is long-term

5
ME Challenges of GBV Programs (2)

• Sensitive nature of the information and hidden
  forms of violence (femicide, trafficking, FGC)
• Changing violence levels hard to measure and
  interpret
• Ethical and methodological issues

6
Monitoring versus Evaluation

• MONITORING
• Tracking changes in program performance over time
• EVALUATION
• Assessing whether objectives have been met
• Assessing extent to which program is responsible
  for observed changes

7
Illustration of Program Monitoring
Program indicator
Program start
Program end
TIME-gt
8
Illustration of Program Impact
With program
Without program
Change in Program Outcome
Program impact
Program start
Program end
TIME-gt
9
Activity 1 Is it Monitoring or Evaluation?

• The Ministry of Womens Affairs wants to know if
  programs carried out in Province A are reducing
  the prevalence of intimate partner violence
  (IPV).
• USAID wants to know how many villages in Region B
  have been reached with anti-GBV messages by your
  program this year.
• A country director is interested in finding out
  if the care provided to rape victims in public
  clinics meets national standards of quality.

10
Purpose of Monitoring and Evaluation
The purpose of monitoring and evaluation is to
measure program effectiveness.
11
Monitoring and Evaluation Questions

• Were resources made available to the program in
  the quantities and at the times specified by the
  program plan?
• Were the program activities carried out as
  planned?
• Which program activities were more effective and
  which were less effective?
• Did the expected changes occur? How much change
  occurred?
• Can improved health outcomes be attributed to
  program efforts?
• Did the target population benefit from the
  program and at what cost?

12
ME Across Program Life Cycle
ASSESSMENT What is the nature of the (health)
problem?
1
EVALUATION How Do I know that the strategy is
working? How do I judge if the intervention is
making a difference?
STRATEGIC PLANNING What primary objectives should
my program pursue to address this problem?
2
5
4
3
3
IMPLEMENTATION/MONITORING How do I know the
activities are being implemented as designed? How
much does implementation vary from site to site?
How can be program become more efficient or
effective?
DESIGN What strategy, interventions and
approaches should my program use to achieve these
priorities?
13
Discussion Key Elements of ME Plan

• Do participants programs have ME plans?
• What are the components of the plans? What
  sections do the plans include?
• Distribute Handout 1 Template for ME plan
• What is missing from their programs ME plans?
• What additional components do their programs
  ME plans include?

14
Activity 2 Identifying the Problem

• Get into 4 groups representing the different
  categories of GBV intervention
• Community mobilization
• Behavior change
• Service delivery
• Laws and public policy
• Review Handout 2 Gender-Based Violence Case
  Study
• What is the problem from your groups
  perspective?
• What are 2-3 barriers your GBV intervention group
  might face when trying to help women like Marie?

15
Program Goals

• A goal is a broad statement of a desired,
  long-term outcome of a program. Goals represent
  general big picture statements of desired
  results.
• Examples
• To improve the lives of women who experience
  gender-based violence
• To decrease injury and mortality due to
  gender-based violence
• To eliminate gender-based violence

16
Program Objectives

• Objectives statements of desired, specific,
  realisticand measurable program results
• SMART
• Specific identifies concrete events or actions
  that will take place
• Measurable quantifies the amount of resources,
  activity, or change to be expended and achieved
• Appropriate logically relates to the overall
  problem statement and desired effects of the
  program
• Realistic Provides a realistic dimension that
  can be achieved with the available resources and
  plans for implementation
• Time-bound specifies a time within which the
  objective will be achieved

Source GAP 2003
17
How to Write Program Objectives
To action
The specify what knowledge, attitudes, skills, behaviors
Among specific population or segment
From to from baseline to desired level
or by x percent
or to specific level
By time frame
18
Activity 3 Is it a Goal or an Objective?

• To reduce GBV
• To increase the of men and women in beneficiary
  population who believe that violence is not an
  acceptable way of dealing with conflict from 40
  in 2002 to 80 by 2007
• To increase the number of domestic violence
  clients recommended for counseling who utilize
  counseling services by 30

19
Activity 4 Defining Program Goals and Objectives

• Return to small groups from previous activity
• Develop one goal and two objectives for a program
  that will address problems identified in Handout
  2 (GBV Case Study)
• After 15 minutes, a member of each group will
  share the program goal and objectives

20
Developing Program Strategies

• Role of a Conceptual Framework

21
Conceptual Frameworks

• Diagram that identifies and illustrates the
  relationships between all relevant
• systemic, organizational, individual, or other
  salient factors that may influence
• program/project operation and the successful
  achievement of program or project
• goals.
• ME Purpose
• To show where program fits into wider context
• To clarify assumptions about causal relationships
• To show how program components will operate to
  influence outcomes
• To guide identification of indicators
• To guide impact analysis (causal pathways)

22
Illustrative Conceptual Model for IPV
Individual Perpetrator
Community
Relationship
Society
Society

• Norms
• Laws
• Notions of masculinity
• Gender roles

• Being male
• Witnessing marital violence as child
• Being abused as child
• Absent or rejecting father
• Alcohol use

• Poverty
• Unemployment
• Family isolation
• Community acceptance of violence

• Marital conflict
• Male control of wealth
• Male control of decision-making in family

Source Heise 1998
23
Activity 5 Developing Program Activities

• Return to small groups from previous activity
• Develop 3 program activities for the goal and
  objectives that the group had formulated earlier
  to address the problems identified in the GBV
  case study
• In 15 minutes, a member of each group will share
  the program strategies and activities

24
Logic Models
25
What is a Logic Model?
Logic models are diagrams connecting program
inputs to processes, outputs, outcome and impact
as they relate to a specific problem or
situation. Logic models show what resources your
program will need to accomplish its goals, what
your program will do, and what it hopes to
achieve, emphasizing the links between all these
things.
26
Logic Model Components

• Input Resources used in a program (e.g., money,
  staff, curricula, and materials)
• Process Activities conducted by program to
  accomplish its objectives (e.g., outreach,
  advocacy, training, BCC, etc.)
• Output Direct results of program activities
  (e.g., people trained materials distributed
  couples counseled, etc.)
• Outcome Program results at population level
  (e.g., knowledge, attitudes, behavior, coverage)
• Impact Long-term change in morbidity,
  mortality, and fertility

27
Activity 6 Identifying Logic Model Components

• Distribute Handout 5
• Each of the scenarios in the handout corresponds
  to one of the five components in the logic model
  Input, process, output, outcome, and impact.
• As a group, decide which component the scenario
  illustrates.

28
Illustrative Logic Model for Provider Training
Program

• INPUT
• Human and financial resources to develop training
  materials implement training program

• PROCESS
• Develop GBV clinical training curriculum
• Conduct TOT workshops
• Conduct GBV training for providers

• OUTCOME
• Increased awareness of GBV as a health issue
• Increased disclosure of GBV
• Increased knowledge and utilization of GBV
  services

• IMPACT
• Improved health and safety of GBV victims

• OUTPUT
• Providers trained in GBV
• Improved provider attitudes toward GBV
• Increased provider ability to identify, counsel,
  care for, and refer GBV victims

29
Role of Logic Model

• Program planning
• Helps one think through program strategy where
  you are where you want to be
• Program management
• Helps one track and monitor operations to better
  manage results
• Communication
• Shows stakeholders at a glance what program is
  doing and what it expects to achieve

30
Role of Logic Model

• Consensus building
• Promotes common understanding about what program
  is, how it works and what it is trying to achieve
• Fundraising
• Can structure and streamline grant writing
• Monitoring and evaluation
• Provides thorough understanding of what resources
  program has to work with, what it is doing, and
  what it hopes to achieve.

31
Activity 7 Developing a Logic Model

• Return to small groups from previous activities
• Develop a logic model for two of the activities
  of your program
• After 20 minutes, a member of each group will
  share the logic model

32
Results Frameworks
33
Results Frameworks

• Diagrams that identify steps or levels of results
  and illustrate the causal relationships linking
  all levels of a programs objectives.
• Purposes
• Provides a clarified focus on the causal
  relationships that connect incremental
  achievement of results to the comprehensive
  program impact
• Clarifies project/program mechanics and factors
  relationships that suggest ways and means of
  objectively measuring the achievement of desired
  ends
• Other terms used
• Strategic frameworks

34
Illustrative Results Frameworks
SO1 Increased Utilization of Reproductive Health
Services
IR1 Strengthened sustainability of GBV programs
IR2 Expansion of high-quality health services for
GBV victims in the public and private sectors
IR1.1 Improved policy environment for GBV
prevention and response
IR2.1 Increased availability of GBV screening
and referral
IR2.2 Improved provider attitude towards GBV
IR1.2 Strengthened NGO advocacy for GBV prevention
IR2.3 Improved provider competence in GBV
screening, referral, care, counseling
35
Activity 8 Develop Monitoring and Evaluation
Questions

• Return to your previous group activities where
  you developed your program goals and objectives
  and logic model
• Come up with at least 1 monitoring question and 1
  evaluation question
• After 10 minutes, a member of the group will
  share the ME questions

36
Selecting Indicators
37
An indicator is a specific, observable and
measurable characteristic that can be used to
show changes or progress a program is making
toward achieving a specific outcome.
What is an Indicator?
38
Characteristics of Good Indicators

• Valid accurate measure of a behavior, practice
  or task
• Reliable consistently measurable in the same way
  by different observers
• Precise operationally defined in clear terms
• Measurable quantifiable using available tools
  and methods
• Timely provides a measurement at time intervals
  relevant and appropriate in terms of program
  goals and activities
• Programmatically important linked to the program
  or to achieving the program objectives that are
  needed for public health impact

39

• Indicators do not specify a particular level of
  achievement -- the words improved, increased,
  or decreased do not belong in an indicator.

40
Common Indicator Metrics

• Counts
• No. of legal and service organizations for VAW/G
  survivors
• No. women and children using VAW/G social welfare
  services
• Calculations percentages, rates, ratios
• of health units with at least one service
  provider trained to care for and refer survivors
• of women aged 15-49 who have ever experienced
  physical violence from an intimate partner
• Index, composite measures
• Gender-equitable men (GEM) scale
• Sexual relationship power scale
• Thresholds
• Presence, absence pre-determined level or
  standard

41
Common Challenges in Indicator Selection

• Choosing an indicator that program activities
  cannot affect
• Choosing an indicator that is too vague
• Indicators that do not currently exist and cannot
  realistically be collected
• Selecting an indicator that does not accurately
  represent the desired outcome
• Too many indicators

42
How Many Indicator Are Enough?

• At least one or two indicators per key activity
  or result (ideally, from different data sources)
• At least one indicator for every core activity
  (e.g., training, BCC)
• No more than 8-10 indicators per area of
  significant program focus
• Use a mix of data collection strategies/source

43
Factors to Consider When Selecting Indicators

• Logic/link to framework
• Programmatic needs/information for decision
  making
• Data availability
• Resources
• External requirements (government, donor,
  headquarters)
• Standardized indicators (if available)

44
Activity 9 Selecting Indicators

• Return to small groups from previous activities
  where you developed goals, objectives,
  activities, and ME questions
• Select 3 indicators that your group might use to
  measure progress towards your program goals and
  objectives
• Into which logic model component does each
  indicator fall?
• After 15 minutes, a member of each group will
  share the indicators

45
Activity 10 Selecting Indicators Contd

• As a class, look at the indicators that the
  groups selected in the previous activity
• Discuss the following
• To what extent are the indicators logically
  linked to the groups ME framework?
• Do the proposed indicators share the
  characteristics of good indicators?
• How will the indicators be used for program
  decision-making and for what decisions?
• Are data available to measure the indicators?
• Can the indicators be realistically collected
  given available resources?
• Are there government or donor requirements for
  measuring the proposed indicators?

46
Where to Find Standardized Indicators for GBV
Programs
Bloom, Shelah S. 2008. Violence Against Women
and Girls A Compendium of Monitoring and
Evaluation Indicators. Ms.08-30. Chapel Hill,
NC Carolina Population Center, the University of
North Carolina and Chapel Hill. http//www.cpc.unc
.edu/measure/publications UN Division for the
Advancement of Women (UNDAW), UN Economic
Commission for Europe (UNECE) and UN Statistical
Division. 2008. Indicators to measure violence
against women. Report of the Expert Group
Meeting, 8 to 10 October 2007, Geneva,
Switzerland UN Human Rights Council. 2008.
Report of the Special Rapporteur on violence
against women, its causes and consequences by
Yakin Ertürk. Indicators on violence against
women and State response. Advance edited version
(28 January 2008).
47
Illustrative Indicators Community Mobilization

• Proportion of people who would assist a woman
  being beaten by her husband or partner
• Proportion of people who say that wife beating is
  an acceptable way for husbands to discipline
  their wives
• Proportion of people who agree that rape can take
  place between a man and woman who are married
• Proportion of youth-serving organizations that
  include training for beneficiaries on sexual and
  physical VAW/G

48
Illustrative Indicators Behavior Change
Communication

• Proportion of people who have been exposed to
  VAW/G prevention messages
• Proportion of girls who say that they would be
  willing to report any experience of unwanted
  sexual activity
• Proportion of girls that feel able to say no to
  sexual activity
• Proportion of individuals who know any of the
  legal sanctions for VAW/G

49
Illustrative Indicators Service Delivery

• Proportion of health units that have documented
  and adopted a protocol for the clinical
  management of VAW/G survivors
• Proportion of health units with at least one
  service provider trained to care for and refer
  VAW/G survivors
• Proportion of women who were asked about physical
  and sexual violence during a visit to the health
  unit
• Proportion of rape survivors who received
  comprehensive care

50
Illustrative Indicators Law and Public Policy

• Proportion of law enforcement units following a
  nationally established protocol for VAW/G
  complaints
• No. of law enforcement professionals trained to
  respond to incidents of VAW/G according to an
  established protocol
• No. of VAW/G complaints reported to the police
• Proportion of VAW/G cases that were investigated
  by the police

51
Measuring Indicators
Indicator 1 of service providers trained (in
the past year or other period) to identify,
refer, and care for VAW/G survivors providers
include any clinician providing direct clinical
services to clients at public health
facilities trained refers to attending every
day of a five-day training course on VAW/G
52
Measuring Indicators
Indicator 2 of women who were asked about
physical and sexual violence during a visit to a
health unit Numerator Number of women who were
asked, during the course of their service
provision at the health unit, about any violence
that had ever occurred, either physical or
sexual, in the geographic area of study (nation,
province, state, community). Denominator
Record review Total number of womens records
that were reviewed at the health unit. Exit
interview Total number of women interviewed.
53
Activity 11 Operational Definition of Indicators

• Distribute Handout 8
• Get into 4 groups
• Write an operational definition for one of the
  following indicators
• Intimate partner physical violence
• Appropriate care for rape survivors
• Gender-norm related attitudes that reinforce
  VAW/G
• After 15 minutes, a member of each group will
  share the operational definitions

54
Measuring Composite Indicators

• Example How to measure, attitudes toward GBV ?
• What question do you ask?
• How do you know any one question is a good
  representation of general attitude?
• How do you scale each question (agree/disagree
  strongly disagree to strongly agree yes/no)?
• How do you combine answers to the questions to
  come up with one indicator?
• What value or weight do you assign to any one
  question on GBV attitude in the survey?

55
Example of Measuring Attitudes Toward Wife
Beating in the DHS

• Sometimes a man is annoyed or angered by things
  that his wife does. In your opinion, is a man
  justified in hitting or beating his wife in the
  following situations
• If she goes out without telling him?
• If she neglects the children?
• If she argues with him?
• If she refuses to have sex with him?
• If she burns the food?

56
Activity 12 Specifying Indicator Metrics

• Return to small groups from previous activities
• Look at the indicators that your group has
  selected to measure progress towards your goals
  and objectives
• Define the metrics for each indicator

57
Indicator Reference Sheets
Distribute Handout 9
58
Setting Indicator Targets Useful Information
Sources

• Past trends
• Client expectations
• Donor expectations
• Expert opinion
• Research findings
• What has been accomplished elsewhere

59
Activity 13 Setting Indicator Targets Based on
Past Trends
One of the objectives of the Ministry of Womens
Affairs in Country X is to reduce the prevalence
of intimate partner violence. The 2000 DHS
showed that 30 of ever married women aged 15-49
years in the country had ever experienced
intimate partner emotional, sexual or physical
violence. A comparable DHS conducted in
2005-2006 showed that the prevalence of intimate
partner violence (emotional, physical, or sexual)
was 25 among ever-married women aged 15-49.
Based on these trends, what would be a realistic
national target for this indicator in 2010?
60
Types of Information Sources

• Quantitative data
• Useful for tracking trends accurately and
  highlighting differences
• Qualitative data
• Useful for understanding the context in which the
  trends/differences occur and to interpret
  quantitative data accurately (e.g. focus groups
  in-depth interviews)

61
Quantitative Information Sources

• Population-based surveys
• DHS (domestic violence module) VAW survey
• Facility surveys
• Client exit interviews provider surveys clinic
  observations
• Health service statistics
• Program statistics
• Crime statistics (of poor quality in many LDCs)
• Behavioral surveillance systems

62
Triangulating Data Sources Soul City
Community
6. Cost Effectiveness Study
Individual
1. National Survey
2. Sentinel Site Studies
3. (National) Qualitative Impact Assessment
Society
4. NNVAW Partnership Study
5. Media Monitoring and Analysis
63
Activity 14 Identifying Information Sources for
Indicators

• Return to small groups from previous activities
• Look at the indicators that your group has
  selected to measure progress towards your
  program goals and objectives
• For each indicator, identify the potential source
  of data needed for its calculation
• What challenges might you face in using these
  potential information sources?
• After 10 minutes, a member of the group will
  present

64
Guiding Questions for Choosing an Evaluation
Design

• What question does your program need to answer?
• What do you want to measure (indicators)?
• How sure to you want to be? (VERY SURE GOOD )
• What is the cost of making a mistake (low,
  medium, high)?
• When do you need the results? (FAST)
• How much are you willing to pay? (CHEAP)
• Has the program already started?

65
Choosing an Evaluation Design
GOOD, FAST, CHEAP PICK 2!
HIRE SOMEONE TO HELP YOU OUT!
NO EVALUATION DESIGN IS IMMUNE TO PROBLEMS!
66
Impact Evaluation Needs Specific Design

Evaluating program impact interested in attribution/causality Experimental (with randomization) or quasi-experimental design
Program already implemented no baseline Post-test only design (then multivariate analysis)
Program not yet implemented Pretest-post-test design time series design


67
When monitoring and evaluating GBV programs, the
confidentiality, privacy, safety, and well-being
of GBV survivors must be top priority at all
times.
68
References

• Bloom, Shelah S. 2008. Violence Against Women
  and Girls A Compendium of Monitoring and
  Evaluation Indicators. Ms.08-30. Chapel Hill,
  NC Carolina Population Center, the University of
  North Carolina and Chapel Hill.
• Bott S, Guedes A, Claramunt MC, Guezmes A.
  Improving the Health Sector Response to
  Gender-Based Violence A Resource Manual for
  Health Care Professionals in Developing
  Countries. 2004. IPPF.
• IGWG. 2004. Addressing Gender-Based Violence
  from the Reproductive Health/HIV Sector A
  Literature Review. Washington, DC POPTECH.

69
References Contd.

• Soul City 4. Impact Evaluation Violence Against
  Women, Vols. I and II. 2001.
• WHO. 2007. WHO Ethical and Safety Recommendations
  for Researching, Documenting and Monitoring
  Sexual Violence in Emergencies. Geneva,
  Switzerland World Health Organization.
• http//www.who.int/hac/network/interagency/news/et
  hical_and_safety_recommendations/en/index.html