Assessing Program Impact

1
Assessing Program Impact

• Chapter 8

2
Impact assessmentsanswer

• Does a program really work?
• Does a program produce desired effects over and
  above what would have occurred without the
  intervention?
• Are there important unintended effects?

3
Impact Assessmentscan be relevant at many points

• Policy formation
• Evaluating a pilot program
• Ongoing programs may need to modify or refine
• high costs of programs must continually
  evaluate their efficacy
• demonstrations of effectiveness are required to
  renew grants
• justify uses of public funds

4
Prerequisite conditions

• impact assessments build on earlier forms of
  evaluation
• the programs objectives must be well articulated
  and have plausible, measurable outcomes
• intervention is well implemented and has been in
  place for sufficient time

5
Equivalence

• identical composition - same mixes in terms of
  their program-related and outcome-related
  characteristics
• identical predispositions - equally disposed
  toward the project and equally likely to attain
  any given outcome status
• identical experiences over the time of
  observation

6
If two groups are equivalent, both are subject
to the same degree of change induced by factors
outside of the program. Any difference in
outcome between them, therefore, should represent
the effect of the program.
7
Two classes of Approaches Experimental vs
Quasi-Experimental Research Designs

• Randomized field experiment
• the gold standard research design
• Participants are randomly sorted into at least
  two groups a control group and an intervention
  group
• Quasi-experiments
• Nonrandomized in which participants (given
  intervention) are compared with nonparticipants
  (the controls)

8
Randomized Field Experiments

• Principal advantage
• it isolates the effect of the intervention being
  evaluated by ensuring that intervention and
  control groups are statistically equivalent
  except for the intervention received
• Best way to achieve equivalence!

9
Randomization is just not a shot in the dark

• Allocating targets to intervention and control
  groups requires considerable care
• The evaluator must use an explicit chance-based
  procedure
• Be careful when using lists
• use statistical significance testing to judge
  whether a specific difference is likely to have
  occurred simply by chance

10
Data Collection Strategies

• Two strategies can improve the estimates of
  program effects
• Make multiple measurements of the outcome
  variable
• Collect data periodically during the course of an
  intervention

11
Units of Analysis

• the units on which the outcome measures are taken
• should be based on the nature of the intervention
  and the target units
• Be careful!
• Observational vs. experimental units
• Assumption of independence

12
Limitations of Randomized Experiments

• These research designs are not applicable to all
  program situations
• The program usually has to be stable and
  operationally mature for authentic results
• If the program changes during the course of the
  experiment, it is difficult to differentiate
  between what program version produced what
  effects.
• Difference between Experimental and Actual
  program delivery

13
Ethical Considerations

• ethical qualms about randomization deprives
  control groups of positive benefits
• If program resources are scarce
• do you allot services by chance?
• or insist that the most needy targets receive
  priority?
• The next chapter that will look at alternative
  designs.

14
Perfect vs Good Enough

• time and resource constraints
• intended use of the results
• feasibility of design
• ethical considerations
• credibility
• differences between experimental and actual
  intervention delivery
• high turnover in policy considerations
• integrity of a randomized experiment is easily
  threatened

15
Impact Assessments Overview

• All impact assessments are comparative with a
  group receiving alternative services or no
  treatment.
• Ideally, the conditions being compared should be
  identical in all respects except for the
  intervention
• All assessment involves establishing control
  conditions
• The most valid results generally require more
  skills, more time to complete and more cost.

16
Beware! Rigorous Impact Assessments involve
technical and managerial challenges and
significant resources -sometimes has political
pressures

• Is it justified by the circumstances?