Health Program Evaluation: Impact and Outcome Evaluation Designs

1
Health Program EvaluationImpact and Outcome
Evaluation Designs

• CHSC 433
• Module 5/Chapter10
• L. Michele Issel, PhD
• UIC School of Public Health

2
Objectives

1. Develop a sampling strategy
2. Develop a design for an effect evaluation
3. Appreciate the trade-offs between rigor and costs

3
Avoid the 3rd Failure

• Next slide adds a failure to the two discussed
  previously.
• Design the effect evaluation to avoid evaluation
  failure the failure to detect program effects
  due to poor design, measurement, or sample of the
  evaluation.

4
(No Transcript)
5
Rigor Considerations inEffect Evaluation

• Sampling strategy
• Validity and Reliability
• Related to design
• Related to method
• Related to measures
• Power to find effects

6
Sampling

• Plan for selecting participants in the evaluation
• Plan for selecting/choosing a control or
  comparison group
• Plan for reaching and accessing the evaluation
  participants or groups
• Plan assures generalizability

7
Sample Size Influence

• Cost of the evaluation
• Time and effort required
• Power to detect a difference made by the program

8
Power

• Statistical power begins with sample size
• Power is the degree of certainty that will be
  able to find a measurable and significant effect
  from the intervention
• Is a trade-off between

9
Not so Random

• Randomization
• Randomly assign to receive the program or not
• Random selection
• Randomly select from among program participants
  and non-participating target audience for
  inclusion in the evaluation

10
Effect Evaluation Designs

• Refer to research or epidemiology textbook
  chapters on
• experimental
• quasi-experimental
• non-experimental
• designs

11
Design Considerations

• Ethical issues with random selection or random
  assignment
• Design to assess impact objectives
• Timing of data collection relative to receiving
  the intervention
• Who are the controls???

12
(No Transcript)
13
Design Cost vs Rigor
14
Measure Program Impact Across the Pyramid