The Path to Effective

1

HEALTH COMMUNICATIONS PANEL OF NATIONAL EXPERTS
The Path to Effective Tobacco
Counter-Marketing C Rebecca Murphy-Hoefer,
PhD, MPH May 28, 2008 Maple Hill Farm Hallowell,
Maine
2
Objectives

• Present CDC Framework for Program Evaluation as a
  framework or Roadmap
• Highlight Program Planning, Managing and
  Implementation, and Evaluation
• Explore Program Planning, Best Practces and
  Lessons Learned

3
Why Invest in Media Campaign Evaluation?

• Demonstrate accountability to stakeholders
• Media Campaigns most visible and
• Measure program achievement
• Are KABB changing?
• Manage program resources
• Managing calls to Quitline
• Document and improve program operations
• Feedback on ads, placement, funding
• The evaluation of the counter-marketing efforts
  should be coordinated with the evaluation of the
  entire tobacco control program

4
CDCs Framework for Program Evaluation
5
Framework for Program Evaluation in Public Health
6
CDCs Designing and Implementing an Effective
Tobacco Counter-Marketing Campaign
Program Planning
Advertising
Target Audience Insights
Specific Populations
PublicRelations
Counter- Marketing Campaign
Managing Implementing
MediaLiteracy
Media Advocacy
Evaluation
GrassrootsMarketing
7
Framework for Program Evaluation in Public Health
8
Outcome Evaluation

• Results of campaign
• Changes in attitudes or beliefs about tobacco
• Changes in intentions
• Changes in behaviors
• Changes in the environment
• Changes in disease trend
• Activity

9
CDC Framework for Program Evaluation
10
Commonly Used Indicators

• Participation rates
• Communities reached
• Attitudes
• Individual behavior (prevalence)
• Community norms
• Policies
• Health status
• http//www.cdc.gov/tobacco/Indicators/KeyIndicator
  s.htm
• Primary and secondary data

11
Framework for Program Evaluation in Public Health
12
Justify Conclusions

• Analyze and Interpret
• Types- Descriptive, comparative statistics,
  segments, time, region, level of awareness of
  campaign, internal, external
• Attribution- Campaign accounts for difference
• Changes overtime, exposure effects , regional
  effects
• Multivariate Analyses Effects of multiple IVs
  (timing of ads, change in awareness, attitudes)
  on DVs (change in behavior), control for age,
  sex, race, ethnicity
• Measure attitudes, beliefs and behaviors expected
  to change those not expected to change
• Longitudinal study-Changes over time
• Field study-Compare communities (etc.)
• Web-based

13
Framework for Program Evaluation in Public Health
14
Ensure Use

• Support annual and long-range planning
• Focus attention on issues important to program
• Promote your program
• Identify partners for future collaborations
• Enhance the public image of your program
• Retain or increase funding
• Provide direction for program staff

15
Checklist for an Effective Report

• Include an executive summary
• Describe the stakeholders and involvement
• Describe features of the program, include the
  logic model
• Outline key evaluation questions
• Include a description of the methods,
  methodological strengths and weaknesses
• Present results and conclusions into context
  (what is reasonable at this point and how the
  results should be interpreted)

• Translate findings into recommendations
• Minimize technical jargon
• Provide detailed information in appendices
• Use examples, illustrations, graphics, and
  stories
• Involve stakeholders in preparation of the report
• Consider how the findings might affect others
• Develop additional communication products suited
  to a variety of audiences, for sharing the results

16

Evaluation of Media Campaigns

• Resources
• New CDCs Tobacco Counter-Marketing Outcome
  Evaluation Manual
• Designing and Implementing an Effective Tobacco
• Counter-Marketing Campaign
• Telephone Quitlines A Resource for Development,
  Implementation, and Evaluation
• Conclusion
• Rebecca Murphy-Hoefer, PhD, MPH
  (rmurph27_at_kennesaw.edu)
• Assistant Professor, Department of Communication
• Kennesaw State University

17

HEALTH COMMUNICATIONS PANEL OF NATIONAL EXPERTS
The Path to Effective Tobacco
Counter-Marketing C Rebecca Murphy-Hoefer,
PhD, MPH May 28, 2008 Maple Hill Farm Hallowell,
Maine
18
CDCs Framework for Program Evaluation
19
Involve StakeholdersWhy?

• Ensure that media evaluation designed to answer
  questions important to stakeholders
• Increase likelihood of continued support
• Build wider competency in evaluation
• Increase possibility evaluation findings used
• Clear understanding of limitations
• Clarify goals, objectives and limitations
• with stakeholders
• Making broad conclusions are difficult

20
Involve StakeholdersWhy?

• Reduces suspicion and fear
• Encourages differing perspectives and many voices
• Increases awareness and commitment
• Increases the possibility of reaching objectives
• Feel ownership, defend the program
• Avoid ignored, criticized, or resisted.
• Will support...

21
Potential Stakeholders

• The health department and commissioner
• Law enforcement
• Schools and educational groups
• The medical community
• Community-based organizations, advocacy groups
• Elected officials

• Program staff, Campaigns operation
• Local and regional coalitions
• Community leaders, members, and grantees
• Local, state, and national partners
• Program funders
• Target audience

22
ChecklistEngage Stakeholders

• Target key stakeholders for participation
• Identify key areas for stakeholder input
• Balance inclusivity, individuals, and orgs
• Create a plan for strategic involvement
• Bring stakeholders together regularly
• Understand and reflect stakeholder values,
  increase credibility
• Establish a method of communication

23
Establish a Media Evaluation Team

• Budgeting
• Developing and communicating program objectives
  and the logic model
• Managing evaluating contracts
• Coordinating evaluation activities between
  program staff and the evaluation team
• Incorporating evaluation findings into program
  planning and revision
• Technical expertise to design and implement
  specific tasks
• Health departments with personnel with technical
  expertise can be a part of the evaluation team,
  but should involve outside evaluation experts
  (neutral and objective)

24
Framework for Program Evaluation in Public Health
25
CDCs Designing and Implementing an Effective
Tobacco Counter-Marketing Campaign
Program Planning
Advertising
Target Audience Insights
Specific Populations
PublicRelations
Counter- Marketing Campaign
Managing Implementing
MediaLiteracy
Media Advocacy
Evaluation
GrassrootsMarketing
26
Describe the Program

• Stage of development
• Context
• Need
• Resources
• Goals (CDC)
• SMART objectives
• Activities
• Expected effects
• Logic model

27
Goals Objectives

• Goals
• Prevent the initiation of tobacco use among young
  people
• Promote quitting among young people and adults
• Eliminate exposure to secondhand smoke
• Identify and eliminate the disparities related to
  tobacco use and its effects among different
  population groups
• Objectives
• Select a limited set of objectives to focus on
  the most important results feasible to obtain

28
Strong Program Objectives Are
Specific event or action Measurable amount of
change Achievable realistic and
achievable Relevant relate to the program
goal Time-bound when objective will be achieved
29
Strong Program Objectives Are
Specific Measurable Achievable Relevant Time-bound
Increase the percentage of youth aged 12-17 who
demonstrate confirmed aided awareness of the
media campaign slogan from 0 in January 2009 to
65 in July 2009.
30
(No Transcript)
31
Waterford, ME, Intersection of Route 5 Route 35
32
Generic Program Logic Model
INPUTS
ACTIVITIES
OUTCOMES
OUTPUTS
Benefits for participants during and after
program activitiesshort term, intermediate,
long term
Program Resources
What we dowith program resources to fulfill
mission
Direct productsof activities
33
INPUTS
ACTIVITIES
OUTCOMES
OUTPUTS
Budget, staff, stakeholders, policy, state health
department, ad agency,
Placement ofpaid media messages
Reach and frequency of TV, radio, and billboard
ads
Awareness and level of receptivity to ads
Changes in normative beliefs
Program Goal Prevent the initiationof tobacco
use amongyouth aged 12-17
Reduce smoking initiationamong teens
34
Inputs
Activities
Short-term Outcome
Long-term Outcome
Outputs
IntermediateOutcome
Goal
Output
Activity
Reach and frequency of TV, radio, and billboard
ads.
Placement of paid media messages.
Objective By October 2009, a pretested health
consequences ad will have been placed in the
largest state media market to reach 12-17 year
olds. Indicator Reach and frequency of TV,
radio, and billboard ads. Data Source- Media
Reports, GRPs, etc.
35
Inputs
Activities
Short-term Outcome
Long-term Outcome
Outputs
IntermediateOutcome
Goal
Short-term Outcome
Output
Increase the percentage of youth who are aware of
the campaign slogan.
Reach and frequency on TV, radio, and billboard
ads.
Performance Objective Increase the percentage of
youth aged 12-17 who demonstrate confirmed aided
awareness of the campaign slogan from 0 percent
in January 2009 to 65 percent in July
2009. Indicator Percentage of youth aged 12-17
who demonstrate confirmed awareness of at least
one paid media message Data Source Survey,
Legacy Media Tracing Survey
36
Inputs
Activities
Short-term Outcome
Long-term Outcome
Outputs
IntermediateOutcome
Goal
Intermediate Outcome
Short-term Outcome
Decrease the percentage of youth who think that
young people who smoke have more friends.
Increase the percentage of youth who are aware of
the campaign slogan.
Performance Objective Decrease the percentage of
youth aged 12-17 who think that young people who
smoke have more friends from 50 percent in
January 2009 to 25 percent by January 2010.
Indicator Proportion of youth who think the
majority of people their age smoke
cigarettes. Data Source Legacy Media Tracking
Survey
37
Inputs
Activities
Short-term Outcome
Long-term Outcome
Outputs
IntermediateOutcome
Goal
Long-term Outcome
Intermediate Outcome
Decrease the percentage of youth who smoke.
Decrease the percentage of youth who think that
young people who smoke have more friends.
Performance Objective Decrease the proportion of
high school youth who report trying a cigarette
from 60 percent in 2009 to 50 percent in
2010. Indicator Proportion of high school youth
who report trying a cigarette. Data Source Youth
Tobacco Survey
38
Special Notes

• While logic models list behavioral outcomes,
  results occur through a combination of
  interventions.
• Example a media literacy program would not be
  expected to result in a reduction in youth
  smoking unless other components of the
  counter-marketing program were also influencing
  these youth.
• Different interpretations of short-term,
  intermediate, ad long-term outcomes occur.The
  logical sequence is the most more important
  outcome.

39
CDCs Designing and Implementing an Effective
Tobacco Counter-Marketing Campaign
Program Planning
Advertising
Target Audience Insights
Specific Populations
PublicRelations
Counter- Marketing Campaign
Managing Implementing
MediaLiteracy
Media Advocacy
Evaluation
GrassrootsMarketing
40
Framework for Program Evaluation in Public Health
41

Evaluation of Media Campaigns

• Resources
• New CDCs Tobacco Counter-Marketing Outcome
  Evaluation Manual
• Designing and Implementing an Effective Tobacco
• Counter-Marketing Campaign
• Telephone Quitlines A Resource for Development,
  Implementation, and Evaluation
• Conclusion
• Rebecca Murphy-Hoefer, PhD, MPH
  (rmurph27_at_kennesaw.edu)
• Assistant Professor, Department of Communication
• Kennesaw State University