EMPOWERMENT EVALUATION: Khulisa

1
EMPOWERMENT EVALUATIONKhulisas Experience in
Using Project Staff as Data Collectors

• Peter Njaramba, Mary Pat Selvaggio and Josie
  Mangxaba
• Khulisa Management Services
• Presented at SAMEA Conference
• March 2007

2
What is Empowerment Evaluation?

• Developed by David Fetterman at Stanford
  University in 1994
• Empowerment Evaluation is the use of evaluation
  concepts, techniques, and findings to foster
  improvement and self-determination (Fetterman et
  al. 1996)

3
What is Empowerment Evaluation?(2)
Based on 10 Principles
Evaluation Process Principles
Evaluation Outcomes Principles

• Community ownership
• Inclusion
• Democratic participation
• Community knowledge
• Accountability
• Evidence-based strategies

• Improvement
• Organizational learning
• Social justice
• Capacity-building

4
Traditional vs. Empowerment Evaluation

• In the evolution toward empowerment evaluation,
  there has been increasing use of participatory
  and collaborative approaches to evaluation

Evaluation Generation First Second Third Fourth
Evaluation Activity Measurement Description Judgment Responsive Constructivist
Role of Evaluator Technical Describer Judge Collaborator Change agent
5
Traditional vs. Empowerment Evaluation (2)
Traditional Evaluation Empowerment Evaluation
External Internal
Outsiders are evaluators Outsiders are facilitators
Stakeholders often don't participate Broad range of stakeholders participate
Dependency Self-determination, capacity-bldg
Donor focus and ownership of evaluation Participant focus and ownership of evaluation
Focus on accountability Focus on learning
Predetermined design Flexible design
Formal methods Rapid appraisal methods
6
Traditional vs. Empowerment Evaluation (3)

• Distinguishing feature from other participatory
  evaluation methods
• .Its acknowledgement and deep respect for
  people's capacity to create knowledge about, and
  solutions to, their own experiences (AEA, 2005).

7
Traditional vs. Empowerment Evaluation (4)

• The role of evaluation expert in EE
• Facilitator and teacher
• Program supporter rather than being impartial
• Change agent
• Facilitate the development of a rigorous and
  organized approach to evaluation
• Help establish baseline data, monitor change over
  time
• Ensure everyone is heard

8
Khulisa Evaluations

• Khulisa establishes a participative working
  relationship with clients
• Participation takes place throughout all phases
  of the evaluation
• Degree and breadth of participation differs by
• Nature of the program
• Contractual agreement with funders
• Requests by clients

9
Khulisas use of EE Principles

• One of the principles of Empowerment evaluations
  is to facilitate organizations use of data to
  learn and their ability to sustain their
  evaluation efforts (capacity-building principle).
• Khulisa has incorporated this EE principle into
  several programme evaluations by encouraging
  programme staff to
• Be involved in design of the data collection
  tools, and
• To serve as data collectors
• ..In order to build greater understanding of
  evaluation among programme staff and build buy-in
  to the findings.

10
Projects that used Programme Staff as Data
Collectors

• HIV and AIDS Audit Interventions in South
  African Higher Education Institutions
• District Development and Support Programme for
  Schools in South Africa
• Evaluation of the Life Skills-HIV/AIDS Programme
  in Primary and Secondary Schools of Northwest
  Province
• AIDS Prevention, Positive Living and Empowerment
  (APPLE) Project in Mozambique and Malawi

11
Did the use of Programme Staff increase knowledge
and understanding of Evaluation?

• We recently contacted 20 individuals who were
  involved in data collection, and we asked them
  about their experience in collecting the data.
• A few of the respondents indicated that they
  already had good ME understanding at the time of
  the evaluation.
• No. of who reported that participating in the
  evaluation
• Facilitated learning 14 of 20
• Taught them general evaluation issues and
  techniques 11 of 20

12
Did the use of Programme Staff increase knowledge
and understanding of Evaluation? (2)

• I found the training beneficial. I also was
  able to know actual issues on the ground, not
  what is reported as differences in program
  implementation from school to school.
• I learned skills on management of data
  collection
• I learnt about questionnaire design. I
  learnt what to include in tools
• I developed skills for data collection for
  monitoring
• I found the workshop we had with Khulisa to
  look at Terms of Reference and to get together
  basic questions useful-data collection process
  and trying to get people to return questionnaires
  was a lot of hard work and frustrating rather
  than a learning process

13
Were programme staff able to apply what they
learned about evaluation to other work
situations?

• Number who reported having applied what they
  learned about evaluation to other work situations
  10 out of 20
• I used the experience in data collection and
  analysis in my Masters Research. In another
  program, the experience increased my awareness
  and ability to determine program problems and
  solutions
• I made changes to the monitoring tool used in
  managing those who are involved in OVC, and also
  to include community in programs not just rely on
  Government
• Based on this experience, I realized the
  internal audits that are done in my workplace are
  not formulated correctly. We came up with new
  instruments

14
Did the use of programme staff as data collectors
increase their buy-in to the evaluation findings?
No. who report that the experience increased their No.
Ownership of the overall evaluation activity and the results (N20) 8
Capacity to use evaluation data (N20) 11
Capacity to interpret evidence (N20) 10
Capacity to draw conclusions (N20) 9
Capacity to make judgments (N20) 9
Capacity to disseminate findings (N20) 9
Capacity to consider inconsistencies and contradictions (N20) 7
15
Other Feedback from Project Staff on their
participation
No. reporting ability to No.
Identify programme strengths (N20) 9
Identify programme problems (N20) 9
Identify programme solutions (N20) 10
Implement programme solutions (N20) 9
16
Conclusions

• Many (but not all) respondents report having
  benefited from participating in data collection.
  
• Benefits were felt more strongly by those who had
  little prior ME experience before participating
  in Khulisas evaluation
• Perceived benefits include understanding of
  general issues and ultimate ownership of
  collected data.

17
Empowerment Evaluation Resources

• Robin Lin Miller and Rebecca Campbell Taking
  Stock of Empowerment Evaluation An Empirical
  Review, American Journal of Evaluation 2006 27
  296
• USAID Center for Development Information and
  Evaluation Conducting A Participatory
  Evaluation, 1996
• David Fetterman Empowerment evaluation in
  Brazil building capacity and facilitating
  self-determination, Dec 2004
• David Fetterman Empowerment Evaluation Building
  a Learning Organization, March 3, 2006
• Wayne Miller and June Lennie Challenges issues
  in applying empowerment evaluation principles in
  practice Case study of the evaluation of a
  national school breakfast program, 2006
• Empowerment Evaluation website
  www.stanford.edu/davidf/empowermentevaluation.htm
  l

18
The End
Thank You