Chapter 4: Understanding the Evaluation Context and Program Theory of Change ?? 4: ?????????????

1
Chapter 4 Understanding the Evaluation Context
and Program Theory of Change?? 4 ?????????????
Organizing Evaluation ????

• ? Front-end Analysis ????
• Relationship between Stages and Questions
• ????????
• Balancing Costs and Benefits ???????
• Involving Stakeholders ??????????
• Pitfalls in Front End Planning
• ????????
• Policy Context and Framework
• ???????
• Theoretical and Empirical Knowledge
• ?????????
• Program Logic, Program Theory, and Logical
  
• Frameworks ??????????????

2
Introduction ??

• Front-end Analysis
• ????
• Identifying the Main Client and Key Stakeholders
• ??????????????
• Understanding the Context
• ????
• Constructing, Working with, and Assessing a
  Theory of Change
• ????????????

3
Front-End Analysis ????

• Client and stakeholders
• ????????
• Timing and time management
• ???????
• Nature of resources ????
• Relevance of social science theory
• ???????????
• Findings of similar evaluations
• ???????
• Role of theory of change ???????
• Existing data ????

4
Balancing Costs and Benefits ????????Benefits ??

• strong knowledge ???????
• Intended use by main client ?????????
• Reauthorization or replenishment ???????
• Potential for learning what works ???????????
• context specification of what will work
  ???????????????
• Building local capacity ??????
• answers to questions of stakeholder concern
  ????????????

5
Balancing Costs and Benefits ????????Costs ??

• Evaluations are limited by time and money, some
  more than others ?????????????????
• Examples of costs ????
• cost of evaluation in relation to the cost of the
  program ????????????
• costs in terms of the social burden to program
  officials, evaluands, respondents
  ?????????????????????
• reputation costs to the evaluator and the
  evaluation community ??????????????

6
Dangers or Pitfalls ?????

• The belief that everything should be covered up
  front ???????
• The to-do-ism fixation ???????
• McDonaldisation of society ??????
• Truisms pop up while doing the front-end planning
  ?????????????
• Front-end planning does not prevent people to
  join in group think ????????????????
• Power matters, never forget to look at the power
  position of the participants ?????????????????????

7
Main Client ????

• One main client ??????
• Key stakeholder sponsoring or requesting the
  evaluation ??????????????????
• May be??
• authorizing or funding the program????????
• authorizing or funding the evaluation???????
• accountable for the public for the
  intervention????????
• to whom the evaluators are accountable ???????

8
Stakeholders ?????

• People or representatives of organizations that
  have a stake in the intervention
• ???????????????????
• Those affected by an intervention, in its
  lifetime, or in subsequent years
• ???????????????????????
• Important to include those who would typically
  not be asked to participate
• ???????????????????,??????

9
Examples of Stakeholders ???????

• Participants ???
• Direct beneficiaries ?????
• Indirect beneficiaries ?????
• Others impacted (directly or indirectly)
  ??(?????)??????
• Donors ???
• Government officials ????
• Program directors ?????
• Policy-makers ???
• Community and interest groups or associations
  ??????????

10
Stakeholder Roles ????????

• Each evaluation is different, stakeholders have
  different roles
• ????????,??????????????
• Use a checklist to determine the role(s) of each
  stakeholder (example on next slide)
• ?????????????????(?????)

11
Checklist of Stakeholder Roles ?????????
Individuals, groups, or agencies ???????? To make policy ???? To make operational decisions ?????? To provide input to evaluation ??????? To react ???? For interest only ??????
Developer of program ?????
Funder of program ?????
Boards/agencies ???/??
etc. ??
12
Identifying and Involving Stakeholders
?????????????

• Identify the stakeholders
• ???????
• Have periodic stakeholder meetings
• ???????????
• Involve early on so all have understanding of the
  intervention
• ????,???????????
• Describe the extent to which stakeholders are
  actively involved
• ??????????????

13
Stakeholder Analysis ???????

• A technique to identify and assess the importance
  of key people, groups of people or institutions
  ????????????????????????????
• Reasons to do ????
• identify people, groups, and institutions
  
  ????????????
• anticipate the kind of influence positive or
  negative these groups will have on your
  initiative
  
  ????????????????????????
• develop strategies to get the most effective
  support possible
  ????,???????????

14
Stakeholder Analysis How To ?????????
Stakeholder ????? Stakeholder Interest(s) in the Project ????????????? Assessment of Impact ???? a) evaluation on stakeholder b) evaluation on stakeholder Potential Strategies for Obtaining Support or Reducing Obstacles ??????????????



15
Program Stages and the Broad Evaluation Question
????????????
Stage of program development ?????? Evaluation question to be asked ???????
Assessment of social problem and needs ??????????? To what extent are community needs and standards met? ??????????????????????
Determination of goals ????? What must be done to meet those needs and standards? ???????????????????
Design of program alternatives ????????? What services could be used to produce the desired changes? ????????????????
Selection of alternative ??????? Which of the possible program approaches is best? ?????????????
Program implementation ???? How should the program be put into operation? ??????????
Program operation ???? Is the program operating as planned? ?????????????
Program outcomes/effects/impact ????/??/?? Is the program having the desired effects? ?????????????
Program efficiency ????? Are the program effects attainted at a reasonable cost? ??????????????????
16
Existing Theory and Knowledge???????

• The underlying theory of change
• ???????
• Using existing evidence through research
  synthesis
• ????????????
• Interpreting a complex program as intervention
  chains
• ???????????????
• one set of stakeholders provide resources to
  other stakeholders
• ???????????????????
• belief that behavior change will follow
• ???????????

17
Sources of Theoretical and Empirical Knowledge
??????????

• Journals synthesizing the accumulation of
  explanatory knowledge
• ???????????
• evaluations and other social science research,
  including psychological and economic studies
• ?????????,???????????
• problem-based research
• ???????
• repositories of randomized experiments
• ???????
• Knowledge Fund ????

18
Function of the Theory of Change???????

• Depict a sequence of the??????
• inputs the program will use ????????
• activities the inputs will support ????????
• outputs to which the program is budgeting
  ??????????
• results anticipated (outcomes and impacts)
  ????(?????)
• Depict other events or conditions ?????????
• Identify assumptions for cause and effect
• ??????
• Identify critical assumptions that need to be
  examined by the evaluation
• ?????????????

19
How Theories Work ??????????

• Note the research that explains or predicts an
  event ??????????????
• example students do better when parents are
  involved in their studies ????????????????,??????
  ??
• The program theory links what the research is
  showing to a theory of the effects of the
  project/program
• ?????????????/???????????

20
Potential Influences on Program
Results??????????
Political Environment (Governance, etc.) ????(???)
21
Why Use a Theory of Change?????????

• Helps identify elements of programs critical to
  success
• ?????????????????
• Helps build a common understanding of the program
  and expectations among stakeholders
• ?????????????????????
• Identifies and measures progress on intervening
  variables
• ????????????
• Identifies assumptions that may become the basis
  of evaluation questions
• ????????????????

22
Review or Create ?????

• If the program/project already has a developed
  theory of change, carefully review it
• ????/??????????????,??????????
• may need to refine or rework it
  ???????????
• If there is no theory of change, need to create
  one
• ?????????,??????????

23
Constructing a Theory of Change??????

• Is there research underlying the program?
• ??????????
• What is the logic of the program?
• ?????????
• What are the key assumptions being made?
• ??????????

24
Process for Constructing a Theory of
Change?????????
Logic ??
Key assumptions ????
Research ??
25
Is There Research Underlying the
Program???????????

• Evaluative studies within organization
• ?????????
• OECD DAC repository of Publications and Documents
  or Information by Country
• ???????????????????????????
• Evaluation studies done by development
  organizations
• ????????????
• Evaluation studies in related fields
• ??????????
• Research of theories of development
• ???????

26
What is the Logic of the Program??????????

• Identify probable occurrences surrounding the
  intervention
• ?????????????
• long-term goals ????
• specify the details about??????????
• target audience ????
• amount of change to signal success ????????
• Timeframe ???
• specify the chain of events ?????????

27
Questions to Check Logic?????????

• Is this plausible? Is it likely to lead to the
  long-term goal?
• ????????????????
• Is this feasible? Are the capabilities and
  resources to implement the strategies possible?
• ???????????????????
• Is it testable? Are the measurements of how
  success will be determined specific?
• ?????????????????????

28
Mapping or Drawing the Logic????

• Can be done in many different ways
• ??????????
• Most are not linear, but might be circular, or a
  storyboard
• ????????,?????????????
• Show complex relationships
• ??????
• Usually have boxes and/or arrows that link back
  to earlier or ahead to later parts of the theory
• ?????/???????????????

29
What are the Key Assumptions being Made??????????

• Assertions about the connections between
  long-term, immediate, and early outcomes
• ???????????????????
• Substantiation for the claim that all of the
  important preconditions for success have been
  identified
• ????????????????????
• Justifications supporting the links between
  program activities and outcomes they are expected
  to produce
• ???????????????????

30
Theory of Change Template??????
Strategies??
Assumptions??
5
6
Desired Results (outputs, outcomes, and
impact)????(??,??,??)
Influential Factors ???????
Problem or Issue??
1
4
3
Community Needs/Assets
????/??
2
31
Simple Theory of Change Model ???????
By training ????
32
Theory of Change with Assumptions????????
Report writing skills adequate to communicate
with government
????????,???????
Needs of trainees met ??????
Training ??
Better Decisions ?????
High Quality Evaluations ??????
Useful Information ??????
Resources given to follow what they learned
????,????
Enough time to learn ???????
33
Example of Program Theory Model ????????
Visits by teachers to students homes
?????????
Identification of special problems that retard
students achievement (health, emotional, etc.)
????????(??????)????
Parents knowledge of schools expectations for
students ????????????
Students receipt of special help ????????
Improvement of condition (health, emotional)
??(?????)???
34
Example Basic Model Theory of Change (Research
Grant Proposal)????????????????
35
Flow Chart of Classic Logic Model?????????
36
Logic Model Emphasizing Assumptions?????????
??
??
??
??
??
??????
????????
????
37
Results Chain???
Area of Control Internal to the Organization
????????
Outputs Reach Direct Beneficiaries ?????????
Area of Influence External to the Organization
????????
Inputs (Resources) ??(??)
Activities??
Outputs ??
Short-Term Results(Direct) ????(??)
Intermediate Results (Indirect) ????(??)
Long Term Result ????
External Factors ????
38
(No Transcript)
39
Women have limited access to economic
opportunities due to lack of access to credit and
other productive resources and to social control

??????????????????,?????????,??????????
Project offers credit, technical assistance and
group information
??????????????????
Women create business ??????
Generate profits ??
Profits re-invested
?????
Short-term improvement in household welfare
?????????
Permanent improvement in household welfare
?????????
Business sustained ??????
Impact nutrition, health, and clothing
????????????
Improved housing ????????
Economic improvements ??????
Improved education for girls
???????????
40
Example of Program Outcome Model ????????
41
Logical Framework (Logframe) ????

• Links the activities, results, purpose, and
  objectives of an intervention
  
  ???????????????????????
• Is a specific logic model using a 4 x 4 matrix
  ?????4 x 4 ?????????
• For each component, the evaluator identifies the
  ???????,????????
• indicators that are needed ?????
• sources ??
• assumptions ??

42
Logframe Model ??????
Narrative Summary ????? Performance Indicators ???? ME/Supervision/ Verification ?????/??/?? Key Assumptions ????
Program Goal ????
Project Development Objective ??????
Outputs ??
Components ??
43
GOAL?? (general objective)(????) Improve the economic and social welfare of women and their families INDICATORS ?? Improvements in family income in x of participating families Improvements in measures of health status, nutritional status, and educational participation VERIFICATION ?? Household surveys of the economic, social, and health condition of all family members  
PURPOSE?? (specific objective)(????) Provide women with opportunities to earn and learn while their children are cared for in home day care centers INDICATORS?? ? day care homes functioning, providing accessible, affordable care of adequate quality during working hours and thus allowing shifts in women's employment and education activities VERIFICATION?? From surveys changes in women's employment and education and their evaluations of the care provided Evaluations of quality of care provided based on observation ASSUMPTIONS ?? Other family members maintain or improve their employment and earnings Economic conditions remain stable or improve
OUTPUTS ?? Trained caregivers, supervisors, and directors Day care homes upgraded and operating Materials developed Administrative system in place MIS in place INDICATORS?? ? caregivers trained ? homes upgraded and operating ? materials created and distributed ? a functioning MIS VERIFICATION?? Data from MIS on trainees, homes, and materials Evaluations of trainees after initial training and during course of continuous training ASSUMPTIONS ?? Family conditions allow home day care mothers to carry through on their agreements to provide care
ACTIVITIES ?? Select caregivers and supervisors and provide initial training Upgrade homes Develop materials Develop administrative system Deliver home day care Provide continuous training and supervision Develop monitoring and evaluation system RESOURCES ?? Budget Technology Human resources VERIFICATION?? Plan of action, budgets, and accounting records Studies showing that the chosen model and curriculum work Evaluations to see that the activities were not only carried out but done well Survey o
IPDET
44
Assessing a Theory of Change Viewpoint or
Framework????????????

• In relation to social needs ???????
• Of logic and plausibility ???????
• Through comparison with research and practice
• ???????????
• By confronting a program theory with one or more
  relevant scientific theories
• ?????????????????
• Via preliminary observation
• ??????

45
Theories of Change Models should Answer these
Questions ??????????????

• Is the model an accurate depiction of the
  program?
• ?????????????
• Are all elements well defined?
• ?????????????
• Are there any gaps in the logical chain of
  events?
• ???????????????
• Is it realistic to assume that the program will
  result in the attainment of stated goals in a
  meaningful manner?
• ????????????????????,??????