FOCUS on Evaluating Health Promotion Programs

1
FOCUS on EvaluatingHealth Promotion Programs

• Day 2

2
Review of FOCUS Provincial Program Record Form

• Questions?
• Comments?
• Suggested changes?

3
Your FOCUS Evaluation Toolbox

• Part II Focus Groups

4
Focus Group Definition

• A formal, facilitated discussion with 8-12
  participants on a specific topic.
• Purpose of a focus group is to collect in-depth
  information from a group of people representative
  of your population of interest (e.g, seniors,
  adolescents)

5
Advantages of Focus Groups

• Allows for in-depth discussion and probing on an
  issue
• Provides large volume of information at
  relatively low cost (compared to other data
  collection methods)
• Opinions of more people can be obtained within a
  shorter time frame (compared to
  surveys/interviews)

6
Limitations of Focus Groups

• Potential for participants to influence opinions
  of other participants
• Do not provide quantifiable information about a
  population
• Number of questions that can be asked is limited
  (more participants fewer questions)

7
Limitations of Focus Groups

• Quality of information is very dependent on
  skills of the moderator
• May not be appropriate for some
  populations/topics due to strong influence of
  social context on responses
• Can be difficult to conduct with populations with
  hearing, cognitive or communicative impairments

8
Steps in Conducting a Focus Group

• Clarify purpose of group
• Assess resources to conduct group
• Decide on methods and procedures
• Write moderators guide
• Recruit focus group participants
• Coordinate logistics
• Facilitate focus group
• Analyze focus group data
• Interpret and disseminate results
• Take action

9
1. Clarify purpose of group

• Identify needs/priorities of population of
  interest
• Pilot test new materials
• Pilot test other data collection instruments
  (e.g., questionnaire)
• Assess beliefs, attitudes opinions of population
  of interest
• Evaluate impact of program or activity

10
2. Assess Resources to Conduct Group

• Budget facilities, equipment, honoraria for
  participants, data analysis expenses.
• Staff availability and skill level
• Facilities and equipment
• Time

11
3. Methods and Procedures

• Need to determine
• Number of groups
• Size of groups
• Composition of groups

12
4. Write moderators guide

• Moderators guide is the outline of the
  discussion to be carried out by the focus group.
• Writing the guide involves deciding on the
  topic(s) to be discussed and translating the
  topics into logically ordered questions for
  discussion

13
Format of Moderators Guide

• Introduction/purpose of focus group, ground rules
• Warm-up (general questions that set the stage for
  the discussion
• Key content section (more specific questions and
  related probes)
• Summary (clarification, additional information)
• Closing (explain next steps, how information will
  be used)

14
Tips for developing questions

• Number of questions usually limited to 10-12 for
  a two hour focus group
• Refer back to reason for wanting to conduct focus
  group (step 1)
• Focus on need to know, not nice to know
• Questions should be general in nature, with
  suggested probes to stimulate discussion when
  necessary

15
Probes

• Short comments or questions that can be used to
  stimulate a more in-depth discussion of
  topics/sub-topics
• Probes used only when moderator feels that
  participants require further direction or
  encouragement in their discussion

16
Exercise 4developing moderators guide

• Develop a brief (6-8 questions plus probes)
  moderators guide on one of the following topics

17
Developing Moderators GuideSuggested Topics

• A focus group of HR managers, health and safety
  reps and union stewards assembled to identify
  priority issues and suggested activities for a
  workplace substance abuse prevention program
• A focus group of teens pilot-testing a pamphlet
  on Club drugs.
• Any other topic youd like to address

18
Facilitating Focus Group Sessions

• Role of Moderator
• Creates a non-threatening, supportive atmosphere
  that encourages all participants to share their
  views
• Builds rapport with participants
• Facilitates interaction among participants
• Covers all topics/questions, interjecting with
  probes and summaries when necessary

19
A good moderator

• Doesnt need (or pretend to be) an expert on the
  topic(s) under consideration
• Can build a rapport with participants
• Can probe issues without reacting to, or
  influencing, participants
• Can lead the discussion without becoming
  side-tracked
• Can control opinionated participants and
  encourage input from reluctant participants
• Keeps the discussion flowing with minimal
  response time
• Promotes discussion among, not between,
  participants

20
Exercise 5Facilitating Focus Groups

• Using the moderators guide provided, groups
  discuss and identify emerging priorities in FOCUS
  communities
• Record the participants responses on the
  flipcharts provided.

21
Steps 8-9Analyzing Evaluation Data and
Presenting Results
22
Step 8 Process the DataQuantitative Data

• Quantitative information is easier to analyze
  when it has been entered as a number. In other
  words all response categories should be assigned
  a number
• Use a computer
• Verify 10 of the data entry to eliminate errors.
  Most errors are systematic
• What if you dont have a computer?

23
Step 8 Process the DataQualitative Data -
Coding Data

• Familiarize yourself with the questionnaire and
  topic
• Read through 15 of the responses for each
  open-ended question writing down all the unique
  responses
• When no new responses are found, re-write codes
  and assign a number to each
• Write the corresponding code number beside each
  open-ended question
• Repeat for each open-ended question
• Can enter the code number into your data set to
  quantify

24
Step 8 Process the DataQualitative Data - Focus
Groups

• 1. Transcribe the discussion
• 2. Segregate and organize the data into logical
  and meaningful segments
• 3. Read through all material and formulate the
  different themes that are evident
• 4. Coding - Read through again comment by comment
  and assign a theme(s)

25
Step 8 Process the DataQualitative Data - Focus
Groups

• Focus group 1
• Area 1 What Safety Issues are discussed with
  kids
• comment 1
• comment 2
• etc.
• Area 2 What problems do parents face with
  enforcing safety
• comment 1
• comment 2
• etc.
• Focus group 2
• Area 1 What Safety Issues are discussed with
  kids
• comment 1
• comment 2
• etc.
• Area 2 What problems do parents face with
  enforcing safety
• comment 1
• comment 2
• etc.

26
Task Identifying themes for Qualitative Data

• Objectives
• 1. Read through comments
• 2. Individually theme each comment
• 3. Discuss and agree on what the main themes as a
  group
• 4. Write the main themes down on a flip chart

27
Step 8 Analyze the DataQualitative Data - Focus
Groups

• Theme each group independent of each other
• Ideally you should have more than one person
  theme the groups independently
• You are looking for themes which emerge across
  all groups
• Look for patterns and trends and avoid using a
  he said.she said kind of analysis
• Consider the following
• In how many groups did each theme appear?
• Are there common trends/concerns across the
  groups?
• Do not ignore themes that emerge in one or two
  groups.

28
Step 8 Analyze the DataQuantitative Data

• For most surveys or tracking methods simple
  descriptive statistics (frequencies, means,
  ranges, etc.) are all that is needed.
• More complex analysis may be required when
  comparisons are needed between subgroups of the
  population or to test changes over time.

29
Step 8 Analyze the DataUse of Statistical
Analysis

• Statistical analysis aims to show that your
  results are not just due to chance or the luck
  of the draw.
• It provides a way to determine the repeatability
  of any differences observed. If the same outcome
  is found when a study is repeated over and over
  again, we really dont need a statistical
  analysis.
• Similarly when we study a sample of the
  population, statistical analysis is used to help
  us decide whether it is likely that these same
  differences would be found if we repeated the
  experiment in multiple samples or in the entire
  population.

30
Step 8 Analyze the DataGuidelines

• Combine statistical expertise with stakeholder
  interpretation.
• Keep your original purpose/research questions in
  mind.
• Avoid getting tide up in detailed analysis that
  may not help to answer your research questions.

Link to SPSS results
31
Exercise 6 Analyzing evaluation data

• Trade your focus group notes with another group
• Review the notes to identify the key themes and
  issues for each question.
• Record the key themes and issues on the
  flipcharts provided.

32
Step 9 Presenting Evaluation Data

• It is easy to become overwhelmed with too much
  information, so focus on the evaluation questions
  and only present the information which answers
  those questions.
• Choose a format which will highlight the key
  results.
• Keep it simple.
• Pictures are worth a thousand words.

33
Response Rates for Each Province
Number Received and Used in analysis 117 113 72 7
0 54 52 23 22 7 6 2
Number Sent 155 173 99 112 90 71 31 31 8 8 7
Response Rate 75.5 65.3 80 62.5 60 73 74 71 87.
5 75 28.6
Ontario Quebec British Columbia Manitoba Nova
Scotia Alberta Saskatchewan New Brunswick Prince
Edward Island Newfoundland North West Territories
34
Example Line GraphThe Effect of the Number of
Media Advertisements andCommunity Events on the
Number of Initial CallsJuly 1994 - January 1997
Initial Calls
7 15 13 8 8 9 6 13
8 10 16 13 22 25 19 11
24 10 16 13 14 14 22 11 17
15 14 15 18 24 21
2 2
1 3 8 6 2 5 6
4 1 3 3 1 12 26 7
4 2 2 3 1 1
Promotional events
35
Example Pie ChartFigure 4 Percentage Reporting
a Need for an Alliance that Provides Services and
Networking Opportunities
N200
36
Example Stacked Bar GraphFigure 13
Effectiveness of Communication Channels for
Sharing With Other Alliance Members
Percent
37
Example Collapsing Response CategoriesOverall
Quality of Care and ServicesBenchmarking Data
Percent
Missing data have been excluded
9 sites, n975 Hospital A, n119
38
Example Misleading ResultsThings Respondents
Dislike About Organization
Percent
n622
n1,237
39
Step 9 Interpret and Disseminate Results

• Results need to be interpreted within the context
  of the purpose of the project.
• Keep your audience in mind when preparing report.
  What do they need and want to know?
• Consider the limitations of your design and
  instruments
• Possible biases (selection, non-response,
  measurement, etc.)
• Validity of results
• Reliability of results
• Generalizability of results

40
Step 9 Interpret and Disseminate Results

• Are there alternative explanations for your
  results?
• How do your results compare to other similar
  programs?
• Are different data collection methods used to
  measure your program showing similar results?
• Are your results consistent with theories which
  have been supported through previous research?
• Are your results similar to what you expected?
  If not, why do you think they may be different?

41
Dissemination of Results

• Goal Share the information in order to empower
  people to make changes.
• What is needed to do this?
• Who do you disseminate to?
• What format do you use? What has been
  effective?
• What information do you give them?

42
Step 10 Take Action

• Involve your stakeholders in interpreting and
  taking action on your results
• Revisit your original goals of data collection.
• Write a list of recommended actions
• Prioritize those changes which are most important
  and most feasible
• Set up an action plan to implement changes
• Implement!

43
Step 10 Take Action

• Look for opportunities to improve your program
• Consider improving one or two areas that will
  have the largest positive impact
• What are some of the programs strengths? How can
  you build on these areas?

44
Disclaimer

• The Health Communication Unit and its resources
  and services are funded by the Community Health
  Promotion Branch, Ontario Ministry of Health and
  Long-term Care. The opinions and conclusions
  expressed in this paper are those of the
  author(s) and no official endorsement by the
  Ministry of Health and Long-term Care, Community
  Health Promotion Branch is intended or should be
  inferred.