Immunization Program Evaluation: Writing an Evaluation Plan

1
Immunization Program Evaluation Writing an
Evaluation Plan

• Pascale Wortley, MD, MPH
• Suchita Lorick, DO, MPH
• Immunization Services Division, NCIRD
• June 3, 2008

2
Webinar Overview

• Evaluation Plan Background
• Evaluation Plan Template Overview
• Sample Evaluation Plan AFIX
• Questions and Answers

3
Evaluation Plan Background

• Plan due to CDC by August 30, 2008
• Submitted via PAPA/CDC Secure Data Network
  (https//sdn.cdc.gov)
• Expected to go live end of June
• Grantees will receive email about applying for
  digital certificate in the next 2 weeks
• CDC will provide feedback on the submitted
  evaluation plans by October 31, 2008
• Immunization Program Evaluation (IPE) Resources
  at http//www.cdc.gov/vaccines/programs/progeval/
• Questions/Comments/Suggestions can be submitted
  via IPE_at_cdc.gov or your CDC project officer

4
Evaluation Plan Template -Overview

• Program Component/Activity
• Activities
• Evaluation Goals
• Stakeholders (Interests/Roles/Responsibilities)
• Include evaluation team at the program
• Describing program component/activity
• Focusing evaluation
• Data collection and analysis
• Ensuring use and sharing lessons learned

5
AFIX Sample PlanPart 1
6
AFIX Sample PlanProgram Component / Activity

• AFIX
• The feedback sessions of AFIX visits

7
AFIX Sample PlanEvaluation Goal

• To improve the quality and effectiveness of AFIX
  feedback sessions
• Should reflect where you want to be

8
Stakeholders

• Gain broader perspective, avoid blind spots, try
  to ensure utilization of results
• Key stakeholders
• Those served or affected by activity
• Those involved in program operations
• Those in a position to make decisions about the
  activity
• For a manageable process, the list of
  stakeholders must be narrowed to primary intended
  users

9
AFIX Sample PlanTables 1 and 2 Stakeholders -
Interests, Roles, and Responsibilities
Note Table numbers are based on numbering of
tables in the Guide to Immunization Program
Evaluation
The next four slides will highlight the columns
of Tables 1 and 2 Group Name OR Persons Name
and job and title Program Manager Interests in
or perspectives about the evaluation Ensure
adequate stakeholder input Complete evaluation
plan on time Ensure results of evaluation are
useful and utilized to improve program
performance Roles in the Evaluation Evaluation
team leader Responsibilities in the evaluation
Oversee data collection and analysis Write
evaluation plan Utilize evaluation findings How
and when to engage Involved in all activities
throughout the evaluation
10
Group Name OR Persons Name and job and title
AFIX coordinator Interests in or perspectives
about the evaluation Evaluation and its findings
could impact routine Activities Interested in
providing useful feedback for providers Roles in
the Evaluation Stakeholder liaison data
collection and analysis Responsibilities in the
evaluation Coordinate stakeholder meetings
provide input on evaluation Design, methods, and
interpretation and use of findings share
findings with stakeholders utilize Evaluation
findings How and when to engage Meet regularly
with program manager to develop methods attend
all Stakeholder meetings
11
Group Name OR Persons Name and job and title
AFIX field staff Interests in or perspectives
about the evaluation Evaluation and its findings
will directly impact Day to day activities
interested in providing useful feedback for
providers Roles in the Evaluation provider
liaisons data collection Responsibilities in
the evaluation Provide input on data collection
methods data source and data Collectors share
findings with providers utilize evaluation
findings. How and when to engage Meet regularly
with AFIX coordinator attend initial stakeholder
Meeting and subsequent ones as needed
12
Group Name OR Persons Name and job and title
Provider office staff (clinical and
Administrative) Interests in or perspectives
about the evaluation interested in receiving
constructive feedback On ways to improve
immunization delivery and providers interested
in minimizing AFIX Related burden on staff
time Roles in the Evaluation represent
providers perspectives on AFIX
feedback Responsibilities in the evaluation
provide feedback during evaluation planning data
source Utilize evaluation findings How and
when to engage invite to initial stakeholder
meeting phone/email communication During data
collection phase as needed invite to other
stakeholder meetings to troubleshoot, Share
findings, etc.
13

• Background
• Program has 5 AFIX staff
• Staff get some instructions on AFIX feedback
  sessions at the time they are hired
• None have received standardized training on AFIX
  Feedback standards
• Currently, program does not track how many
  feedback sessions include use of written quality
  improvement (QI) plans (as described in the AFIX
  standards, Feedback, Level I pg 10)
• Additionally, we do not know if QI plans contain
  the minimum requirements
• Although AFIX staff work with providers to agree
  upon opportunities for improvement, we do not
  track in a standardized manner whether providers
  are implementing the agreed upon changes.

14

• Context
• Feedback sessions are the only opportunity to
  interact directly with provider staff (especially
  decision-makers)
• Feedback visits are often combined with VFC
  visits and therefore, some of the items discussed
  for the AFIX portion might not get adequate time
  and attention
• Want to use the visit to help providers improve
  immunization delivery and increase their
  immunization rates

15

• Stage of Development
• We have been implementing AFIX visits (including
  Feedback sessions) for the past 8 years. The
  process is very well-established.
• Target Population
• Clinical and administrative staff in private and
  public immunization provider offices
• AFIX staff

16

• Objectives (should reflect where you want to be)
• By March 2009, 100 of AFIX field staff in the
  state will receive training on how to conduct
  feedback sessions that meet all AFIX Level I
  standards.
• By June 2009, all AFIX field staff will report an
  increase in knowledge about the components of a
  feedback session including which key items should
  be included in a quality improvement (QI) plan.

17

• Objectives continued
• Between July and December 2009, during AFIX
  Feedback sessions, all AFIX field staff will
  develop QI plans that contain all key items.
• Starting in January 2010, all AFIX field staff
  will follow up with providers at least on a
  biannual basis to document their progress in
  implementing the improvement opportunities that
  were agreed upon at their last feedback session.

18

• Objectives continued
• Starting in January 2010, all providers will
  implement by the proposed date at least 50 of
  the improvement opportunities that were agreed
  upon at their last feedback session.

19
Describing the ProgramComplete Logic Model

• A flow chart with boxes and arrows leading to the
  next item. The flowchart includes the following
  boxes horizontally
• Inputs
• Activities
• Outputs
• Short Term Effects/Outcomes
• Intermediate Effects/Outcomes
• Long term effects/outcomes

20
AFIX Sample Plan Table 3 Program
Component/Activity Description
Note this table is laid out in logic model
format with the following headings Inputs,
activities (initial and subsequent, outputs,
outcomes (short/midterm and Long
term) Inputs -funding -staff -time -CoCasa Inii
tal Activities -develop AFIX protocol -train
AFIX staff -set up provider visits
21
Subsequent activities -review provider records
and assess coverage level -provide feedback and
recommend strategies for improvement -acknowledge
and reward improved performance -document outcome
of visit in database (CoCasa) -promote
information exchange Outputs -protocols -trained
staff -providers visited -coverage
assessment -feedback provided -rewards
provided -data in database -information exchanged
22
Short/Midterm Outcomes -increased provider
knowledge -increased provider motivation -change
in office practices and policies -change in
physician behavior Long term Outcomes -decrease
in missed opportunities -decrease in invalid
doses -increase in timely vaccine receipt for
eligible population -increased immunization
coverage -decreased vaccine preventable diseases
23
AFIX Sample Plan Optional Logic Model for AFIX
Inputs
Funding Staff Time CoCASA
24
Activities
Develop AFIX protocol Train AFIX staff Set up
provider visits
Review provider records assess coverage
level Provide feedback recommend strategies
for improvement Acknowledge reward improved
performance
Note The activities train AFIX staff and
provide feedback and recommend strategies for
improvement are circled in red because the
sample evaluation focuses on these areas
Document outcome of visit in database
(CoCASA) Promote information exchange
25
Outputs
Protocols Trained staff Providers visited
Coverage assessment Recommendations Rewards
provided
Data in database Information exchanged
26
Outcomes
Increased provider knowledge Increased
provider motivation
Note The outcome change in office Practices
and policies is circled in red because the
sample evaluation focuses on this area
Change in office practices and policies Change
in physician behavior
missed opportunities invalid doses
timely vaccine receipt
Vaccinations VPD
27
AFIX Sample PlanPart 2
28
Focusing the Evaluation

• Formulate and prioritize questions
• Develop indicators and targets for each question
• Identify data sources to help you answer these
  questions
• Outline the who, when and how of data collection
  and analysis

29
Prioritizing evaluation questions

• Is the question
• Important to your program staff and stakeholders?
• Does it reflect key elements of your logic model?
• Will it provide information you can act upon to
  make program improvements?
• Can it be answered using available program
  resources?
• Are there any available data sources?

30
AFIX Sample PlanPriority Evaluation Questions

• Process Questions
• How many providers receive a quality improvement
  (QI) plan containing the 4 key items (Level 1,
  AFIX Feedback Standards)?
• Do AFIX staff know how to write a QI plan
  (including 4 key items)?
• Outcome Questions
• How many providers implement the interventions
  (AKA recommendations) that were agreed upon
  during their AFIX Feedback visit?

4 key items intervention, action steps,
responsible party, date of implementation
31
AFIX Sample PlanTable 8 Data Collection and
Analysis
Evaluation Question 1 How many providers
receive a quality improvement (QI) plan
containing the 4 key items (Level 1, AFIX
Feedback Standards)?

• Indicators
• Percent of providers that received a QI plan
  following an AFIX visit
• Percent of QI plans that include all four key
  items
• Targets
• 80 of providers received a QI plan following an
  AFIX visit
• 100 of QI plans contain all four key items
• Data Sources
• AFIX field staff site visit notes
• QI plans

32
AFIX Sample PlanTable 8 Data Collection and
Analysis continued

• Data Collection
• Method Review site visit notes from 2007-2008
  if not available or incomplete, survey AFIX field
  staff
• Timeline December 2008-January 2009
• Person Responsible AFIX coordinator and AFIX
  field staff
• 2. Method Review a sample of QI plans from
  2007-2008 developed by each AFIX field staff
• Timeline December 2008-January 2009
• Person Responsible AFIX coordinator

33
AFIX Sample PlanTable 8 Data Collection and
Analysis continued

• Evaluation Question 2 Do AFIX staff know how to
  write a QI plan
• (including 4 key items)?
• Indicators
• Percent of AFIX field staff trained on how to
  write a QI plan (including all four key
• Items
• 2. Percent of AFIX field staff who feel confident
  in their ability to write a QI plan
• Targets
• 100 of AFIX field staff received training on QI
  plans before conducting AFIX visits
• 100 of AFIX field staff feel confident in their
  ability to write a QI plan
• Data Sources
• Staff training records
• AFIX field staff

34
AFIX Sample PlanTable 8 Data Collection and
Analysis continued

• Data Collection
• Method Review staff training records
• Timeline December 2008
• Person Responsible AFIX coordinator
• 2. Method Survey AFIX field staff about barriers
  and facilitators of QI plan
• Development
• Timeline December 2008-January 2009
• Person Responsible AFIX coordinator

35
AFIX Sample PlanTable 8 Data Collection and
Analysis continued

• Analysis
• Method Frequencies
• Timeline December 2008
• Person Responsible AFIX coordinator
• 2. Method Frequencies qualitative content
• Timeline January 2009
• Person Responsible AFIX coordinator

36
AFIX Sample PlanTable 8 Data Collection and
Analysis continued

• Evaluation Question 3 How many providers
  implement the interventions
• (AKA recommendations) that were agreed upon
  during their AFIX Feedback visit?
• Indicators
• Percent of providers that implemented one or more
  recommendations (by the
• Proposed date)
• 2. Percent of feedback visits that were conducted
  with provider staff who have the
• Authority to implement changes
• Targets
• Providers implemented at least 50 of
  recommendations (by the proposed date)
• 100 of AFIX feedback sessions are conducted with
  provider staff who have the
• Authority to implement changes
• Data Sources
• Providers AFIX field staff site visit notes and
  QI plans from 2007-2008
• Site visit notes from 2007-2008

37
AFIX Sample PlanTable 8 Data Collection and
Analysis continued

• Data Collection
• Method Survey (a sample of) providers who
  received visits during 2007-2008 to
• Identify which recommendations were implemented
  survey (a sample of)
• Providers and all AFIX field staff about barriers
  and facilitators of implementation
• of recommendations Review site visit notes and
  QI plans from 2007-2008
• Timeline December 2008-February 2009
• Person Responsible AFIX coordinator and AFIX
  field staff
• 2. Method Review site visit notes and QI plans
  from 2007-2008
• Timeline December 2008 January 2009
• Person Responsible AFIX field staff
• Analysis
• Method Frequencies qualitative content
• Timeline February 2009
• Person Responsible AFIX coordinator and AFIX
  field staff
• 2. Method Frequencies
• Timeline January 2009

38
AFIX Sample Plan Table 9 Disseminating Findings
(Ensuring Use)
Target Person or Groups Name Program
Manager Targets Evaluation Use Determine the
current status of AFIX feedback sessions
Utilize findings to improve training provided to
field staff and maximize feedback Sessions to
ensure more providers are implementing agreed
upon recommendations Dissemination Items and
Methods provide a complete report of findings
(electronic and hardcopy) Timeline for
Dissemination at the completion of the
evaluation (March 2009) Person Responsible AFIX
coordinator Target Person or Groups Name AFIX
field staff Targets Evaluation Use Identify
areas of improvement for AFIX feedback
sessions Adjust methodology for recommendations
made during feedback sessions Dissemination Items
and Methods present a summary of evaluation
findings During monthly staff meetings Timeline
for Dissemination at the completion of the
evaluation (March 2009) Person Responsible AFIX
coordinator
39
AFIX Sample Plan Table 9 Disseminating Findings
(Ensuring Use) continued
Target Person or Groups Name Immunization
providers (All AFIX providers Including data
sources) Targets Evaluation Use Improve
implementation of agree upon interventions Use
as a benchmark to see how they are performing
compared to other providers with regard to
implementing recommendations Dissemination Items
and Methods provide a summary of relevant
findings (electronic and hardcopy) Timeline for
Dissemination at the completion of the
evaluation (March 2009) Person Responsible AFIX
field staff Target Person or Groups Name NIC
2009 conference Targets Evaluation Use learn
about methodology for evaluating AFIX feedback
visits Dissemination Items and Methods
presentation Timeline for Dissemination February
2009 Person Responsible program manager
40
Summary

• Evaluation plan will be submitted via PAPA/CDC
  Secure Data Network
• Due date is August 30, 2008
• Evaluation plan template (June 2008 version)
  includes all the items that you will need to
  submit to us

41
Resources

• Immunization evaluation resources are available
  at the Immunization Program Evaluation (IPE)
  website at www.cdc.gov/vaccines/programs/progeval
• Questions / Comments / Suggestions
• You can Contact us either via the IPE email at
  IPE_at_cdc.gov or via your CDC project officer
• THANK YOU!

42
Extra Slides
43
Indicators

• A measurable piece of information that helps you
  answer your evaluation question
• Relates to the objectives of your evaluation
• Can be related to processes or outcomes
• Also referred to as a performance measure in the
  IPOM
• Can use existing ones or develop ones tailored to
  a particular question

44
Targets

• Reasonable expectations about what success
  means
• Should create one for each indicator
• Based on the current status of an activity
• Consider program requirements

45
What are Data Sources?

• Where or from whom you will get data for each of
  your indicators to answer your evaluation
  questions.

Data Source Documents Examples medical records,
meeting minutes, surveillance reports, interview
records Data Source Individuals Examples
staff, providers, partnership members Data
Source Observations Examples data obtained
from observations of staff, environment
(reception area), office flow, activities, etc.
46
Advantages/Disadvantages of Data Sources
Data Source Documents Advantages data available
and accessible Disadvantages value of data
depends on how accurately it was recorded may
lack data needed for the evaluation Data
Source Individuals Advantages can be collected
directly from target population Disadvantages
may be unreliable due to social desirability
and/or recall difficulty Data Source
Observations Advantages can supplement
self-report provide information on behavior,
skills, environment Disadvantages value of data
depends on training of observer and specificity
of instrument
47
Data Collection Methods
Method Survey Advantages anonymity possible
can administer to groups efficient and cost
effective Disadvantages forced choices limit
response wording may bias response
impersonal Method Individual interviews Advantag
es can build rapport can probe for more
information can get breadth/depth of
information Disadvantages time consuming,
expensive, interview style may bias Method
focus groups Advantages can get breadth and
depth of information in short time frame can
convey key information regarding the
program Disadvantages need trained facilitator
time consuming to analyze responses Method
Observation Advantages Can assess fidelity as
activities occur Disadvantages interpretation of
behavior difficult expensive and time
consuming Method Document review Advantages
information already exists does not disrupt
program Disadvantages depends on quality of
information time consuming
48
This document

• This document can be found on the CDC website
  at
• http//www.cdc.gov/vaccines/programs/progeval/down
  loads/Eval_Course_4th-508.ppt