Step 1 : Vision Statement

1
(No Transcript)
2
Day 1
3
Introduction
4
Course Facilitators
5
Expectations of the Workshop
6
Agenda for Day 1
7
Workshop Process

• Start end on time
• Give everyone a chance to speak ask questions
•  Issues and Concerns
• Questions comments at any time
• Questions et commentaires en français

8
Introductory Exercise- Each person, share an
achievement youre very proud of the role you
played- Look for patterns in the stories- Be
prepared to share the patterns in plenary
9
Conceptual Overview
10
Evaluation Challenges

• Establishing cause effect in an open system
• Timing
• Encouraging iterative learning
• Clarifying values

11
Challenge 1 Establishing Cause/Effect in an
Open System

• Multiple actors factors contribute
• Causality is mutual, therefore usually
  attribution is not possible
• Unintended results often ignored

12
Shifting Influence Over Time
Influence
Time
13
Challenge 2 Timing

• Tracking incremental, cumulative change
• Not assessing results as though final

14
Are we having an impact yet ?
15
Challenge 3 Encouraging Iterative Learning

• Emphasis on development results
• Easy to measure vs. important to know
• Keeping results within their context
• Making time for reflection

16
Accountability Learning A Balancing Act
17
Accountability Learning A Balancing Act
18
Challenge 4 Clarifying Values

• Assessment is not value neutral
• Making explicit the implicit
• Building the capacity to engage

19
Challenges and Outcome Mapping

• Challenges
• Cause effect in open system
• 2. Timing
• Iterative Learning
• Clarifying Values

• OM Tools
• Vision Outcome Challenges
• Outcome challenge Progress Markers
• Org. Practices
• Monitoring Journals
• 4. Mission, Boundary Partners, Progress Markers,
  Org. Practices

20
What is Outcome Mapping?

• an integrated PME tool
• an approach that focuses on changes in the
  behaviour, relationships,or actions of partners
  (as outcomes)
• a methodology that characterizes and assesses the
  programs contributions to the achievement of
  outcomes
• an approach for designing in relation to the
  broader development context but assessing within
  your sphere of influence

21
(No Transcript)
22
At What Level?

• Course is focusing at the program level
• Definition of Program A group of related
  projects and activities with a specific set of
  resources (human, financial, capital) directed to
  the achievement of a set of goals within a
  specified period of time
• Can be used by projects, organizations, and
  communities too

23
Key Concepts of Outcome Mapping

• Sphere of influence
• Outcomes as behavioural change

24
Sphere of Influence
Planning Design
Context
Sphere of Influence
Assessing
25
The Focus of Outcome Mapping
Behavioural Changes
26
What is an  Outcome  in OM?

• Change(s) in the behaviour, relationships,
  activities, and/or actions of an individual,
  group, or organization that the program was
  helpful in bringing about.
• These changes are aimed at encouraging human and
  ecological well-being.

27
Changes Sought
Changes in State
Ecosystem
Human System
Changes in Behaviour
28
Why Behavioural Change?

• To stress that development is done by, and for,
  people
• To illustrate that although a program can
  influence the achievement of outcomes, it cannot
  control them because ultimate responsibility
  rests with the people affected

29
How Can Outcome Mapping Be Used?

• For a program to tell its performance story in
  outcome terms by
• articulating its goals and designing its
  activities
• designing a monitoring system for assessing
  internal performance and outcomes of partners
• setting a use-oriented evaluation plan

30
Why Use Outcome Mapping?

• Focussing on changes in partners behaviour,
  relationships, or actions allows a program to
• measure results within its sphere of influence
• obtain feedback about its efforts in order to
  improve its performance
• take credit for its contributions to the
  achievement of outcomes
• show progress towards outcomes

31
Stage 1 Intentional Design
32
  I have a dream!  Martin Luther King
33
(No Transcript)
34
(No Transcript)
35
Step 1 Vision Statement

• Reflects the broad human, social, environmental
  betterment in which the program is engaged and to
  which it is contributing

36
Example Vision Statement Local authorities,
communities, and international organizations in
developing countries in Africa recognize the
value of HIV/AIDS intervention as an integral
part of social economic development.
Municipal, regional, and national governments
actively support HIV/AIDS prevention activities
by formulating and implementing effective public
health policies. Using research findings, they
have developed a comprehensive public health
strategy to slow down the infection rate.
Formerly marginalized groups (e.g. women and
youth) are organized into advocacy groups that
can effectively formulate their needs to policy
makers. All groups have access to reliable and
relevant technical information about HIV/AIDS
prevention and are able to make informed choices.
In essence, there are healthier, happier, and
wealthier communities.
37
Facilitation Questions
?

• Imagine that in 3-5 years the program has been
  extremely successful. What changes will you have
  helped bring about? What are your partners doing
  differently? What have they achieved? In
  essence, what would total success look like?

38
(No Transcript)
39
VISION STATEMENT
The Mission is that bite of the Vision
Statement on which the program is going to focus
40
Step 2 Mission Statement

• Describes how the program intends to
• Operationalize its role in support of the vision
  by determining the areas in which it will work
• Support the achievement of outcomes by its
  boundary partners

41
Facilitation Questions
?

• How can the program best support or contribute
  to the achievement of the vision? What areas do
  you need to work in? What do you need to do?

42
Example Mission Statement In support of this
vision and on behalf of its donors, the program
will work in the areas of research,
dissemination, capacity building, coordination.
It will contribute to the production, synthesis,
dissemination of research data, position
papers, other information that will sensitize
local international actors to HIV/AIDS
prevention. The program will seek to expand the
range of disciplines involved in HIV/AIDS
research. It will enhance HIV/AIDS research
capacity in order to produce credible information
for local, national, international
policy-making program development. It will
promote an interest in HIV/AIDS research among
new researchers by providing research
fellowships, mentorship, training
opportunities. The program will contribute to the
development of linkages between Northern
Southern researchers encourage partnerships
between research organizations, advocates,
decision makers. It will increase its visibility
credibility among the donor community will
convince them of the utility of supporting
HIV/AIDS prevention.

43
Developing a Vision or Mission

• Warm-up general conversation
• Individual write up
• Discuss Post
• Facilitator and/or participant draft
• Present discuss
• Purge the jargon
• Verify with partners (if required) Finalize

44
Developing a Vision with the Community in Nagaland
45
Suggestions and Tips!

• If time is a limiting factor you can carry out
  an email survey in advance to create a draft
  vision mission before the workshop starts. Use
  the draft version as a starting point for
  discussion in order to save time.

46
Exercise 1 Correct Errors in Vision
Mission Statements
47
(No Transcript)
48
Step 3 Boundary Partners
Those individuals, groups, organizations with
whom the program interacts directly to effect
change with whom the program can anticipate
some opportunities for influence.
49
Programs Sphere of Influence
The Real World
Program
Programs Boundary Partners
50
Boundary Partners (have boundary partners)
Programs Boundary Partners
Boundary Partners Boundary Partners
Program
51
Typology of IDRC Boundary Partners
Applied Research Community e.g. researchers,
institutions, NGOs, networks
IDRC Program
System Gatekeepers / Devt Agents e.g. business,
policy makers, NGOs, govt
Beneficiaries e.g. communities, farmers, women,
youth
52
Facilitation Questions
?

•  In which individuals, groups, or organizations
  is your program trying to encourage change so
  that they can contribute to the vision?  With
  whom will you work directly? 

53
Suggestions and Tips!

• A program can choose its boundary partners
  because it wants to influence them or because
  they will influence others
• Example a rural development NGO chooses to work
  directly with farmer organizations who in turn,
  lobby the Minister of Agriculture to effect
  change.

54
Exercise 2Identify the Programs Boundary
Partners
55
Working on Your Own Case
56
Guiding Questions

• In your context
• How would the ideas of Outcome Mapping
  resonate?
• Is a focus on behavioural change appropriate?
• How would you foster the imagination of your
  group to come up with a vision?
• Are you clear about your contributions to the
  vision? (i.e., your mission)
• Who are your boundary partners?
• How could you include them in the process?

57
Conclusion Evaluation of Day 1Whats Working?
Whats Not?
58
Day 2
59
Review of Day 1 Feedback from Evaluation
60
Agenda for Day 2
61
(No Transcript)
62
Step 4 Outcome Challenge

• Description of
• The changed behaviours (relationships,
  activities, and/or actions) of a boundary partner
• How they could contribute ideally to the vision

63
Example Outcome Challenge Statement The program
intends to see researchers in developing
countries that have gained the expertise required
to carry out HIV/AIDS research in a
multidisciplinary fashion using quantitative,
qualitative, and/or participatory methodologies.
They are conducting research on the legislative,
economic, social, and health aspects of HIV/AIDS
with special emphasis on vulnerable groups
including youth and women. They are providing
credible evidence that allows research findings
to be translated into effective HIV/AIDS policies
and programs. They are skillfully disseminating
research results in the appropriate format for
the audience (e.g., to the scientific community,
the general public, NGOs, government officials).
They are successfully obtaining funding from
multiple sources to conduct HIV/AIDS research.
More experienced researchers are mentoring those
new to the field and HIV/AIDS researchers
globally are engaged in productive communications
using various means especially electronic
networks.
64
 Those Who Dream Make a Difference!  Ms.
Kalpana Pant, Chaitanya
65
Phrasing of an Outcome Challenge

• The program intends to see boundary partner
  who description of behaviours in the active
  present tense

66
Facilitation Questions
?

• Ideally, in order to contribute to the vision,
  how would the boundary partner be behaving?
• What would your boundary partner be doing to
  contribute maximally to the vision?

67
Suggestions and Tips!

• Similarity of behaviours sought among boundary
  partners (policy makers and community members) in
  international model forests was unexpected by
  program staff

68
Outcome Challenge Checklist

• Outcome Challenge Statements
• Describe behavioural changes
• About a single boundary partner
• Set out the ideal behavioural changes
• Describe the boundary partners contributions to
  the vision

69
Exercise 3 Identify Errors in Outcome
Challenge Statements
70
Program Framework
Vision
Mission
Boundary Partner
Outcome Challenge
Boundary Partner
Outcome Challenge
Boundary Partner
Outcome Challenge
Boundary Partner
Outcome Challenge
71
Overview Exercise Put Together a Program
Framework
72
(No Transcript)
73
Step 5 Progress Markers

• A graduated set of statements describing a
  progression of changed behaviours in the boundary
  partner
• Changes in actions, activities, relationships
  leading up to the ideal outcome challenge
  statement

74
Levels of Progress Markers

• The program sets out what it would
• Expect to see its boundary partners doing?
• Like to see its boundary partners doing?
• Love to see its boundary partners doing?

75
How Many Progress Markers?

• Limit to a manageable amount of data to be
  collected
• 4 expect to see
• 8 like to see
• 3 love to see

76
Why Graduated Progress Markers?

• Articulate the complexity of the change process
• Permit on-going assessment of partners progress
  (including unintended results)
• Encourages the program to think about how it can
  intentionally contribute to the most profound
  transformation possible
• Make mid-course corrections improvement easier

77
Example Progress Markers

• The program Expects to See local communities
• Participating in regular model forest (MF)
  partnership meetings
• 2. Establishing a structure for cooperation in
  the partnership
• 3. Acquiring new skills for involvement in the
  MF
• 4. Contributing resources necessary to get the
  MF operational (minimum)

78
Example Progress Markers

• The program would Like to See local communities
• Articulating a vision for the MF that is locally
  relevant
• Promoting their involvement with the MF
  nationally
• Expanding the partnership
• Calling upon external experts when necessary
• Requesting new opportunities for training
• Disseminating concrete examples of benefits
  arising from MF activities
• Seeking out new partners for the MF
• Obtaining funding from different national sources

79
Example Progress Markers

• The program would Love to See
• local communities
• 13. Helping other MFs establish themselves
• 14. Sharing lessons-learned internationally
• 15. Influencing national policy debates
  formulation on resource use and
  management

80
Facilitation Questions
?

•  How can the program know the boundary partner
  is moving toward the outcome? What would they be
  doing? 
•  What milestones would be reached as the
  boundary partner moves towards their intended
  role in contributing to the vision? 

81
Suggestions and Tips!

• If a program has multiple boundary partners but
  the primary result to be achieved is changing
  their relationships with one another then it is
  possible to create a single outcome challenge
  set of progress markers for the whole group

82
Progress Marker Checklist

• Each Progress Marker
• Describes a changed behaviour by the boundary
  partner
• Can be monitored observed
• As a set, Progress Markers
• Are graduated from easier to more difficult to
  achieve changes in behaviour
• Describe the change process of a single boundary
  partner

83
Exercise 4Identify Correct Progress Markers
Categorize as Expect, Like, or Love to See
84
(No Transcript)
85
Step 6 Strategy Map

• Outlines the programs approach in working with
  the boundary partner
• Indicates the relative influence the program
  is likely to have on boundary partner
• Helps pinpoint strategic gaps in the approach or
  if the program is overextended
• Suggests the type of evaluation method
  appropriate to track and assess the performance
  of the program

86
6 Types of Strategies
Strategy
Supportive
Persuasive
Causal

• I-1
• Direct Output

• I-2
• Arouse New Skills/ Thinking

• I-3
• Supporter who guides change over time

Aimed at the Boundary Parnter

• E-1
• Alter physical or regulatory environment

• E-2
• Modify the information system

• E-3
• Create / Strengthen a Peer Network

Aimed at the Boundary Partners Environment
87
6 Types of Strategies
88
Facilitation Questions
?
89
Suggestions and Tips!

• Facilitator asks the group questions about their
  strategies
• While they are discussing them (s)he quietly
  plots them on a strategy map
• After being shown the strategy map, the group is
  asked to analyze it for gaps, overextension, and
  the influence that can be expected

90
Exercise 5 Plot Activities on a Strategy Map
91
(No Transcript)
92
 The oft repeated and echoing question in
Outcome Mapping  what or who needs to change? 
raised us from being providers of development,
achieving outputs, to actually believing we were
agents of change.  Raj Verma, NEPED
93
Step 7 Organizational Practices

• The things that you do as an organization to
• foster creativity innovation
• seek the best ways to assist your partners
• maintain your niche

94
Why Organizational Practices?

• Important to how the program is going to function
  to effectively fulfill its mission
• Supporting change in its boundary partners
  requires that the program be able to change and
  adapt as well

95
8 Organizational Practices

• Prospecting for new ideas, opportunities,
  resources
• Seeking feedback from key informants
• Obtaining the support of your next highest power
• Assessing (re)designing products, services,
  systems, and procedures

96
8 Organizational Practices

• Checking up on those already served to add value
• Sharing your best wisdom with the world
• Experimenting to remain innovative
• Engaging in organizational reflection

97
Facilitation Question
?

• What do you have to be doing in order to remain
  effective?

98
Suggestions and Tips!

• This can be a difficult set of concepts to get
  across frame it in terms of changes in behaviour
  internal to the organization
• Have the group think of the program as an
  organization in its own right

99
Exercise 6Identify the Organizational
Practices
100
Exercise 6aDraw the 7 steps of Outcome Mapping
101
Overview of Stage 1
Organizational Practices
102
Working on Your Own Case
103
Guiding Questions

• In your own context
• How would the idea of planning beyond what can be
  achieved be accepted? (i.e. outcome challenge and
  progress markers)
• What kinds of strategies do you use causal,
  persuasive, supportive?
• How will you get your group to talk about your
  own organizational practices?

104
Evaluation of Day 2Feedback from  critical
friends 
105
Day 3
106
Review of Day 2 Feedback from Evaluation
107
Agenda for Day 3

108
Stage 2 Outcome Performance Monitoring
109
Outcome Mapping Monitoring

• Systematic collection of data
• A regular learning improvement cycle
• Encourages the program to challenge itself
• Collects organizes, does not analyze data

110
(No Transcript)
111

•  Being busy creates a mindset that is not
  conducive to innovation and creativity. Without
  interaction, there is no innovation. Time to
  discuss, reflect, and generate new ideas is the
  ransom that outcome mapping demands for
  innovation. 
• Thierry Barreto-Fernandez
• West Africa Rural Foundation, Senegal

112
Step 8 Monitoring Priorities

• Tempting to gather information on a whole range
  of issues but this is not practical
• Base priorities on intended use of intended
  users
• Use(s) for findings and from process
• Improve performance through learning
• Help meet accountability/reporting requirements
• Supply information for planned evaluation
• Inform publicity documents, communication
  activities, or case-study materials

113
Monitoring Plan
114
Boundary Partner
Program
115
Boundary Partner
Program
Outcome Challenge Progress Markers
116
Boundary Partner
Program
Strategies Activities
117
Boundary Partner
Program
Organizational Practices
118
Boundary Partner
Program
Outcome Journal
Strategy Journal
Performance Journal
119
Organizing Monitoring Meetings

• Regular face-to-face meetings to record observed
  changes
• Filling out electronic data sheets on an on-going
  basis, then aggregating changes for entire
  program
• Interviews and/or focus groups with boundary
  partners

120
(No Transcript)
121
Step 9 Outcome Journal

• Uses progress markers to chart boundary partners
  progress
• Sets starting point or benchmark
• Captures details on changes in the boundary
  partner, contributing factors actors, and
  sources of evidence

122
Outcome Journal
Work Dating From/To Contributors to Monitoring
Update
Outcome Challenge
Low Medium High
Expect to see
Who?
LMH
1
2
3
123
Set Values for Low, Medium High

• Can capture either quantity or quality of change
• Quantity (number of boundary partners)
• Quality (depth of change)

124
Baseline

• multiple individuals, groups or organizations in
  each boundary partner set
• identify each boundary partners starting point
  in terms of progress markers
• boundary partners may be at different stages of
  development
• individual change process can be captured over
  time

125
(No Transcript)
126
Step 10 Strategy Journal

• Captures data on the strategies being employed to
  encourage change in the boundary partner
• Helps the program gauge its contributions to the
  achievement of outcomes and modify its actions
  accordingly
• Feeds into work planning (modifications) or new
  activities

127
Strategy Journal
Working Dating From/To Contributors to
Monitoring Update
Strategy to be Monitored
Effectiveness? (How did it help the boundary
partner?)
Outputs
128
Step 11 Performance Journal

• Records data on how the program is operating as
  an organization to fulfill its mission
• Includes information on the programs practices
• Feeds into work planning

129
Performance Journal
Working Dating From/To Contributors to
Monitoring Update
Practice 1 Prospecting for New Ideas,
Opportunities, Resources
Example or Indicators Sources of
Evidence Lessons
Practice 2 Seeking Feedback from Key Informants
Example or Indicators Sources of
Evidence Lessons
130
Exercise 7 Identify your monitoring priorities
(e.g. work on your real case)
131
Stage 3 Evaluation Planning
132
(No Transcript)
133
 Evaluative thinking is a learned behaviour that
you have to practice and improve at on an
on-going basis! Michael Quinn Patton
134
Step 12 Evaluation Plan

• A short description of the main planning elements
  of an evaluation study
• For Whom? What? Where? How? By Whom? When? How
  Much?

135
Elements of an Evaluation Plan
136
Setting Evaluation Priorities

• Criteria with which to select include
• Learning needs
• Reporting requirements
• Partners information needs

137
Guiding Principles for Evaluation

• Balancing learning accountability
• Action oriented
• Each study is unique
• Process not only product
• Build capacity at all levels through
  participation

138
 Not everything that counts can be counted. And
not everything that can be counted,
counts.  Albert Einstein
139
An Evaluation of IMFNS

• Assessment of 1st phase
• Wanted to measure human, community, partnership
  elements of model forests
• Client acceptance of methodology essential
• OM provided a framework for the study

140
Evaluation Questions

• Who has IMFNS reached?
• How have the behaviours of IMFNSboundary
  partners changed?
• What has been the nature of IMFNS interactions
  with its partners?

141
Process of the IMFNS Evaluation

• Recreated planning framework in 1 day with IMFNS
  staff
• Developed progress markers for communities
  goverment officials across the 5 sites
• Conducted a document review key informant
  interviews
• Comparison of boundary partners across MF sites
  for analysis

142
Comparing Boundary Partners
143
Lessons from the IMFNS Study

• Difficult to link what the Secretariat did with
  outcomes achieved without using cause-effect
• Boundary partners should have been involved in
  the development of the framework analysis of
  data
• Should have included more analysis of
  organizational practices
• Did not study the relevance of boundary partners
  within their environment

144
A Self-Assessment of NEPED

• Preparation for external evaluation
• Process
• Fred knew people and context prior to workshop
• Retrospectively built the framework (took 1.5
  days including HML values)
•  Ah-ha  moment happened at end for some
  participants
• Organizing data collection (preliminary
  identification, collection and verification,
  analysis and writing report took another 1.5 days)

145
Lessons from the NEPED Study

• Group dynamics (worked together well,willing to
  trust Fred, cooperative)
• Data (they had the data but it wasnt organized)
• Preparation for external review (intended use)
• Relevance of work with state government (beyond
  trees, number of women involved, number of
  villages worked with)
• Could have had greater external participation
  (Govt of Nagaland, etc.)

146
Conclusion
147
(No Transcript)
148
Conclusion

• A system to think holistically strategically
  about how to contribute maximally to development
• A method to articulate contributions to the
  development process through boundary partners
• Outcomes are changes in the behaviour, actions,
  and/or relationships of boundary partners
• An approach that encourages on-going learning,
  challenges, improvement through ME activities

149
Back to Work How are you going to put your
training in Outcome Mapping to use?
150
Were Your Expectations Met?
151
Evaluation of the WorkshopIn terms of the
content and process, what should we keep, add,
change?
152
Thank You!http//www.idrc.ca/evaluation