Practical Program EvaluationFollow Up: Working a Case Example

1
Practical Program EvaluationFollow Up Working a
Case Example

• By
• Thomas J. Chapel, Office of Workforce and Career
  Development
• Betty Apt, Division of STD Prevention
• Centers for Disease Control and Prevention

2
Today

• Review main points from overview by applying to a
  case example
• Choosing a component to evaluate
• Identifying stakeholders that matter most
• Developing simple logic models
• Choosing a strong evaluation focus
• Discuss November workshop

3
Why Bother? Because We Want Programs to Improve
What do we do?

• Continuous Quality Improvement (CQI) cycle.
• PlanningWhat actions will best reach our goals
  and objectives.
• Performance measurement How are we doing?
• EvaluationWhy are we doing well or poorly?

Why are we doing well or poorly?
How do we do it?
How are we doing?
4
What can Program Evaluation do?

• Increase Program Knowledge
• Direct Program Improvement
• Maximize Resources
• Provide Accountability

5
Avoid Wasted Effort!
6
Two Pieces of Work

• Choose a program component to evaluate
• Apply evaluation steps and standards

7
Choosing a Program Component to Evaluate

• Evaluation can focus on any/all parts of a large
  program like the immunization program

8
Some Potential Criteria..

• Where am I spending the most?
• Where am I concerned the most?
• Where are my big opportunities/new areas?
• Where are my big successes?
• Other
• Other

9
Some Potential Components to Evaluate

• Immunization information systems
• Adolescent immunization
• Perinatal hepatitis B prevention
• AFIX
• VFC

10
Todays Case Example AFIXReasons for Selecting
Component

• Substantial program effort
• Research shows it works
• But implemented in various ways
• Want to ensure our approach is working

11
NextApply Evaluation Steps and Standards
11
12
Evaluation StepStakeholders

• Identify and engage stakeholders
• Describe the program component
• Focus the evaluation design
• Gather credible evidence
• Justify conclusions
• Ensure use and share lessons learned

13
Which Stakeholders Matter Most?

• Who is
• Affected by the program?
• Involved in program operations?
• Intended users of evaluation findings?

Of these, who do we most need to Enhance
credibility? Implement program changes? Advocate
for changes? Fund, authorize, expand program?
14
Some Key Stakeholders
15
Payoffs for Involving Stakeholders

• Strong input on purpose and design of the
  evaluation
• Reality check on utility and feasibility of
  evaluation plan and methods.
• Increase credibility of evaluation findings.
• Increase likelihood that evaluation results will
  be acted upon.

16
Involve Stakeholders from the Start!

• Potential stakeholder roles at all steps
• Input on evaluation design
• Input on data collection methods
• Assist with data collection/analysis
• Outreach to the target population for the
  evaluation
• Interpretation and use of results

17
Case Example AFIX Stakeholders

• Program staff
• Clinical staff (clinic managers, nurses,
  physicians)

18
Evaluation StepDescription

• Identify and engage stakeholders
• Describe the program component
• Focus the evaluation design
• Gather credible evidence
• Justify conclusions
• Ensure use and share lessons learned

19
Seeking Consensus On

• The activities and outcomes depicted
• The roadmap
• Which outcomes program success
• This will facilitate future discussions of
• How much progress on outcomes program success
• Choices of data collection/analysis methods

20
Describing the ProgramComplete Logic Model
Intermediate Effects/ Outcomes
Short-term Effects/ Outcomes
Long-term Effects/ Outcomes
Inputs
Outputs
Activities
Context Assumptions Stage of Development
21

• What the program and its staff actually do

Intermediate Effects/ Outcomes
Short-term Effects/ Outcomes
Long-term Effects/ Outcomes
Inputs
Outputs
Activities
Context Assumptions Stage of Development
22

• Results of activities Who/what will change?

Intermediate Effects/ Outcomes
Short-term Effects/ Outcomes
Long-term Effects/ Outcomes
Inputs
Outputs
Activities
Context Assumptions Stage of Development
23
Simple Logic ModelAFIX
ACTIVITIES OUTCOMES
Early Later Short-Term
Long-Term
If Then If Then If
Then

24
Case Example AFIX Logic Model
25

• Resource platform for the program

Intermediate Effects/ Outcomes
Short-term Effects/ Outcomes
Long-term Effects/ Outcomes
Inputs
Outputs
Activities

• Tangible products of activities

Context Assumptions Stage of Development
26
AFIX Full Logic Model
27

• Moderators Contextual factors that will
  facilitate or hinder getting our outcomes

Intermediate Effects/ Outcomes
Short-term Effects/ Outcomes
Long-term Effects/ Outcomes
Inputs
Outputs
Activities
Context Assumptions Stage of Development
28
Evaluation StepFocusing the Evaluation

• Identify and engage stakeholders
• Describe the program component
• Focus the evaluation design
• Tailor evaluation to program and stakeholders
  needs
• Determine resources and personnel available
• Develop and prioritize evaluation questions
• Gather credible evidence
• Justify conclusions
• Ensure use and share lessons learned

29
Evaluation Standards Help Choose Focus

• Utility Which stakeholders/others want the
  evaluation findings? How/for what purpose will
  they use the findings?
• Feasibility How reasonable are these questions
  given How long program has been in existence
  program intensity data collection
  logistics/constraints.

30
Generating and Prioritizing Evaluation Questions

• Utility Questions which
• Are important to stakeholders
• Are consistent with program goals and objectives
• Will provide information that can be acted upon
• Feasibility Questions which
• Can be answered with available resources and
  within available timeframe
• Are consistent with program maturity and
  intensity
• Will be supported by program decision-makers.

31
Setting Focus Process and Outcome Evaluation

• Process Evaluation
• Determines if program activities are implemented
  as intended.
• Tracks who, what, when, and where program
  information.
• Provides feedback loop for program improvement.
• Conducted throughout project life.
• Outcome Evaluation
• Measures program effects changes in target
  populations knowledge, attitudes, self-efficacy,
  skills, intentions, behaviors or organizational
  changes (e.g., policy adoption).
• Linked to process evaluation.
• Provides feedback loop for program improvement

32
Case Example Setting AFIX Focus

• Utility
• Purpose/User Program staff
• Use Determine if
• AFIX feedback is actually implemented by
  providers.
• If not, determine barriers and challenges to use
  of feedback
• And, determine how to improve activities and
  followup to surmount identified barriers and
  challenges
• Feasibility
• Stage/Intensity Well-established and
  multi-faceted program. Should be making these
  impacts if working
• Resources Assumption Evaluation questions of
  interest have some ready data sources that would
  be credible for those interested in findings

33
AFIX Logic ModelFocus
34
AFIXSample Evaluation Questions

• Proposed focus both process and outcomes
• Outcome
• Are AFIX recommendations implemented by practices
  within 6 months of visit? (short term outcome)
• Process
• Factors that may be related to implementation/non-
  implementation of AFIX recommendations
• Are AFIX staff appropriately trained?
• Are practice contacts the right people?
• What is the quality of the feedback? Are
  practices satisfied with feedback received?

35
Where Next
Indicators
Data Collection Sources and Methods
Engage Stakeholders Throughout!
Data Collection Procedures
Data Analysis
Data Interpretation
Shared Results and Lessons Learned
Results used to improve program
36
But

• These steps more likely to be done well because
  we did a good job on earlier steps!!!

37
Practical Program Evaluation

• Coming! This Fall

38
Fall Training Session

• Reaffirm/reinforce todays points
• Work through remaining steps
• Define indicators
• Identify data collection sources and methods
• Consider data analysis and interpretation
• Maximizing use

39
Pre-Work for November

• Consider criteria for choosing a program
  component
• Give some preliminary thought to
• Potential program components
• Key stakeholders
• Purpose/use of evaluation

40
Practical Program Evaluation

• Life Post-Session

41
Helpful Publications _at_ www.cdc.gov/eval
41
42
Helpful Resources

• NEW! Intro to Program Evaluation for PH
  ProgramsA Self-Study Guide http//www.cdc.gov/ev
  al/whatsnew.htm
• Logic Model Sites
• Innovation Network
• http//www.innonet.org/
• Harvard Family Research Project
  http//www.gse.harvard.edu/hfrp/
• University of Wisconsin-Extension
  http//www.uwex.edu/ces/lmcourse/
• CDC/DASHhttp//www.cdc.gov/healthyyouth/evaluatio
  n/resources.htm4
• CDC/STD http//www.cdc.gov/std/program/progeval/T
  OC-PGprogeval.htm
• Texts
• Kellogg Foundation Logic Model Development Guide
  www.wkkf.org
• W.K. Kellogg Foundation Evaluation Resources
  http//www.wkkf.org/programming/overview.aspx?CID
  281
• Rogers et al. Program Theory in Evaluation. New
  Directions Series Jossey-Bass, Fall 2000
• Chen, H. Theory-Driven Evaluations. Sage. 1990

43
Community Tool Boxhttp//ctb.ku.edu
43
44
This document can be found on the CDC website at

• http//www.cdc.gov/vaccines/programs/progeval/down
  loads/Eval_Course_2nd.ppt