Measuring the Difference: Strategies for Improving and Evaluating Health Information Outreach Progra

1

• Measuring the Difference Strategies for
  Improving and Evaluating Health Information
  Outreach Programs
• Cathy Burroughs
• Maryanne Blake
• NN/LM, PNR
• May 22, 2004
• Annual Meeting
• Medical Library Association
• Washington D.C.

2
Outreach Evaluation Resource Center

• http//nnlm.gov/evaluation
• NN/LM, Pacific Northwest Region
• University of Washington, Box 357155
• Seattle, WA 98195-7155
• cburroug_at_u.washington.edu
• --206-543-9261
• blakema_at_u.washington.edu
• --206-221-3405

3
Workshop Objectives

• Identify where and how evaluation fits in program
  development
• Select theories to strengthen outreach
  strategies
• Determine what to consider when planning outreach
  evaluation

4
Mission of Outreach

• To affect capacities of target communities to
  address issues of health and health care through
  effective access, use, and exchange of health
  information resources.

5
Outreach Activities

• Promotional exhibits, brochures, fact sheets
• Logistical providing equip, facilitating
  connections, developing local resources,
  intermediated search services, ILL
• Educational training classes, demos
• Collaborative partnerships

6
Why Evaluate?

• Benefits
• Justify program with management or funder
• Improve the outreach program
• Other??

7
Why Not Evaluate?

• Fears
• Anxiety about time and resources
• Unreasonable expectations
• Discourages program ideas that may fail
• Others?

8
Best Practices in Outreach

• What Works?
• How Do We Know?

9
Behavior Change Theories

• Success in attaining outreach
• objectives can be challenging
• Changing behavior patterns requires outreach
  activities that provide more than just
  information
• Strategies are needed to help motivate,
  facilitate, and reinforce change

10
OutcomesResults or Changes

• By individual/organization/community
• Awareness
• Knowledge
• Attitudes and perceptions
• Skills
• Behavior
• Priorities, policy, support

11
Factors Affecting Behavior

• Predisposing provide the motivation or reason
  behind a behavior (knowledge, attitudes, beliefs)
• Enabling make it possible for a motivation to be
  realized (available resources, skills, info
  services, ability to obtain resources)
• Reinforcing reward a behavior (opinion leaders,
  community level participation)

12
4 Theories

• Social Learning Theory
• Stages of Change Model
• http//nnlm.gov/pnr/eval/witte.html
• Diffusion of Innovations Model
• http//nnlm.gov/pnr/eval/rogers.html
• Community Organization

13
Social Learning Theory

• Albert Bandura (1970s), also called social
  cognitive theory
• Factors that affect behavior change include
• - behavioral capability
• - outcome expectations
• - self-efficacy
• - observational learning

14
Stages of Change Model

• 5-stage framework for explaining how behavior
  change occurs
• People at different points in the process of
  change can benefit from
• different interventions, matched
• to their stage at that time

15
5 Stages

• Precontemplation Not thinking of it
• Contemplation Thinks about it
• Preparation Makes plans
• Action Uses it
• Maintenance Continues new behavior

16
Diffusion of Innovations

• Everett Rogers (1940s)
• Developed the theory of how new ideas or products
  spread within a society or from one society to
  another

17
Key Diffusion Principles

• People adopt an innovation because people they
  respect endorse it (not due to experts)
• Innovators are first to adopt (2.5 of pop)
• Early Adopters are second (13.5 of pop)
• Early Majority (34)
• Late Majority (34)
• Laggards (16)

18
People are more likely to adopt an innovation if
it has

• Advantage
• Compatibility
• Complexity
• Trialability
• Observability

19
Community Organization

• Empowerment
• Community competence
• Participation and relevance
• Issue selection

20
Phases of Evaluation

• Community Assessment
• Audience Assessment
• Process Evaluation
• Summative Evaluation

21
Planning and Evaluating Outreach
Stage One
Stage Two
Stage Three
Identify Target Community
Conduct Community Profile
Establish Goals Objectives
Develop Activities and Strategies
Stage Six
Conduct Audience Assessment to Tailor Strategies
Share Results and Modify Program as Needed
Develop a Plan to Implement Activities
Carry out Plan to Implement Outreach
Activities
Carry out Plan for Summative Evaluation to Assess
Progress and Impact
Stage Five
Stage Four
Analyze Results for Answers to Evaluation
Objectives

• Establish Evaluation Objectives (Process or
  Summative)
• Select Design and Data Collection Methods
• Develop Evaluation Instruments

Carry out Plan for Process Evaluation to Monitor
and Improve Activities
22
Evaluation Starts with Assessing and
UnderstandingCommunity Needs
23
Planning and Evaluating Outreach
Stage One
Stage Two
Stage Three
Identify Target Community
Conduct Community Profile
Establish Goals Objectives
Develop Activities and Strategies
Conduct Audience Assessment to Tailor Strategies
Stage Six
Share Results and Modify Program as Needed
Develop a Plan to Implement Activities
Stage Five
Carry out Plan to Implement Outreach
Activities
Carry out Plan for Summative Evaluation to Assess
Progress and Impact
Stage Four
Analyze Results for Answers to Evaluation
Objectives

• Establish Evaluation Objectives (Process or
  Summative)
• Select Design and Data Collection Methods
• Develop Evaluation Instruments

Carry out Plan for Process Evaluation to Monitor
and Improve Activities
24
Identify Target Community

• Community group of individuals who share
  certain characteristics, e.g., occupation,
  culture, geographic location

25
Types of Communities

• Rural or urban underserved areas
• Tribal community
• Rural primary care settings
• AIDS workers
• Public health workers
• K-12 nurses, librarians, teachers

26
Identify Target Communities

• What populations would you like to reach?
• Who can you best reach and influence?
• What groups could benefit most from health
  information access?
• What community is ready for outreach?
• What community is most in need?
• Are there potential partners to work with you?

27
Topics of Interest

• Demographic, economic, social and health issues
• Current information needs
• Barriers to accessing health information
• Cultural characteristics
• Resources available

28
Conduct Research

• Use secondary data from national/local sources
• Use the literature
• Obtain user input

29
Rigorous Approach

• To make generalizations
• Survey typical instrument
• Reliability, validity, random sampling

30
Practical Approach

• To gain an overall understanding of targeted
  group and their environment
• Key informant interviews
• Focus groups with stakeholders
• Open-ended survey questions

31
Utilizing the Results

• Understand causes for information problems and
  needs
• Reach a shared vision
• Develop hoped for outcomes
• Look for measures to pre-assess

32
Case Scenario

• Outreach to School Nurses

33
Planning and Evaluating Outreach
Stage One
Stage Two
Stage Three
Identify Target Community
Conduct Community Profile
Establish Goals Objectives
Develop Activities and Strategies
Stage Six
Conduct Audience Assessment to Tailor Strategies
Share Results and Modify Program as Needed
Develop a Plan to Implement Activities
Stage Five
Stage Four
Carry out Plan to Implement Outreach
Activities
Carry out Plan for Summative Evaluation to Assess
Progress and Impact

• Establish Evaluation Objectives (Process or
  Summative)
• Select Design and Data Collection Methods
• Develop Evaluation Instruments

Analyze Results for Answers to Evaluation
Objectives
Carry out Plan for Process Evaluation to Monitor
and Improve Activities
34
Developing Goals and Objectives

• If you dont know where youre going, youll end
  up somewhere else.
• Yogi Berra

35
Setting Goals

• What effect will the program have?
• What overall improvement is needed?
• What are the goals of my targeted audience?

36
Sample Goal

• To improve the quality of health care provided by
  school nurses via effective access and use of
  credible health information resources.

37
Objectives

• Steps important to reaching the goals
• Define what will be done or the outcomes that
  will hopefully result
• Should be measurable

38
Two Types of Objectives

• Process objective
• Specifies what outreach staff will do is a
  process statement
• Outcome objective
•  Includes statements of change such as what is
  hoped to change on the part of outreach
  participants or in the community

39
Sample Process Objective

• Within six months, project staff will develop and
  maintain a website for school nurses.

40
Making Process Objectives Measurable

• Process objectives list specific items that can
  be tracked or measured, such as
• What The action, product, service to be
  accomplished, developed, or maintained.
  Sometimes called the output.
• When A date or time period
• Target population To or for whom

41
OutcomesResults or Changes

• By individual/organization/community
• Awareness
• Knowledge
• Attitudes and perceptions
• Skills
• Behavior
• Priorities, policy, support

42
Outcome Objective

• Focuses on the intended results of the programs
  activities
• Statement of intended benefits to people
  achievements or changes in skill, knowledge,
  attitude, behavior, condition, or life status for
  program participants

43
Types of Outcomes Objectives

• Change Statements are outcomes that identify the
  change or desired effect in what or by whom.
• Targets are change statements with specific
  levels of achievement
• Benchmarks make comparisons to other time
  periods or organizations

44
Example Types of Outcomes

• Change statement More school nurse participants
  use the school nurse website for health
  information
• Target 75 more school nurse participants use
  the school nurse website for health information
• Benchmark Compared to school nurses in the 2
  county area, 80 more school nurse participants
  prefer to use the school nurse website

45
To Measure an OutcomeIdentify its Indicator!

• When figuring out what to measure, determine the
  desired effect (change) stated in the outcome
  objective
• Then, gather data on indicators that provide some
  type of evidence that the change has occurred.

46
Example Indicator

• Outcome More school nurses use the school nurse
  website for health information
• Indicator(s) for use
• Visits to school nurse website
• Number of nurses who have bookmarked the school
  nurse website
• Frequency of visits to website

47
Drafting the Objectives

• Review Community Assessmentlook at data to
  determine what you think needs to change.
  Remember to consider what the community wants to
  get from an outreach program

48
Drafting the Objectives, cont.

• Then, select priority outcomes the hoped for
  changes in knowledge, behavior, attitude, or
  skill by program participants-- write out the
  outcomes objectives
• Think about what activities, products, or
  services are needed to effect that changethen
  write out the process objectives

49
BREAKOUT GROUPS -- EXERCISE 1
50
Planning and Evaluating Outreach
Stage One
Stage Two
Stage Three
Identify Target Community
Conduct Community Profile
Establish Goals Objectives
Develop Activities and Strategies
Conduct Audience Assessment to Tailor Strategies
Stage Six
Share Results and Modify Program as Needed
Develop a Plan to Implement Activities
Stage Five
Carry out Plan to Implement Outreach
Activities
Carry out Plan for Summative Evaluation to Assess
Progress and Impact
Stage Four
Analyze Results for Answers to Evaluation
Objectives

• Establish Evaluation Objectives (Process or
  Summative)
• Select Design and Data Collection Methods
• Develop Evaluation Instruments

Carry out Plan for Process Evaluation to Monitor
and Improve Activities
51
Planning for Activities

• Review what you will do e.g. your process or
  activities
• Think about ways that theories may help you be
  more effective (Stage 3 Tool Kit, page 37 )
• Review the literature for other best practices
  (Appendix K)

52
Think Through Theory Strategy Outcome

• Reach outcomes using strategies based on theory
  and best practice
• See Appendix I

53
Implementation Plan

• A blueprint to clarify, document, and schedule
  your objectives and plans

54
Creating an Implementation Plan

• Describe the overall community and its needs
• List program goals
• List hoped for outcomes
• List objectives
• Describe strategies to be used
• Specify activities
• Include a timeline

55
Introducing the Logic Model

• Framework for program and evaluation
• Fluid
• Changes over time
• Makes evaluation easier

56
Logic Models May Change Over Time
57
Plan BackwardImplement Forward

• Outcomes
• Activities
• Resources

58
Parts of the Logic Model

• Inputs or Resources
• What will you need to carry out your activities?
• Activities
• What will you do?
• Outputs
• How many of what will your activities produce?
• Outcomes
• So what the difference your program makes, the
  benefits that accrue because of your program

59
Outcomes

• Outcomes should be
• specific
• measurable
• action oriented
• realistic
• timed

60
Outputs

• Outputs are what the program puts out!
• They are measures of accomplishment
• E.g.-- counts of services or programs delivered,
  produced or implemented
• Outputs are sometimes confused with outcomes
  (which are different behaviors, decisions,
  actions, attitudes or other changes by outreach
  participants)

61
Sample Logic Model
62
Planning and Evaluating Outreach
Stage One
Stage Two
Stage Three
Identify Target Community
Conduct Community Profile
Establish Goals Objectives
Develop Activities and Strategies
Stage Six
Conduct Audience Assessment to Tailor Strategies
Share Results and Modify Program as Needed
Develop a Plan to Implement Activities
Stage Five
Carry out Plan to Implement Outreach
Activities
Carry out Plan for Summative Evaluation to Assess
Progress and Impact
Stage Four
Analyze Results for Answers to Evaluation
Objectives

• Establish Evaluation Objectives (Process or
  Summative)
• Select Design and Data Collection Methods
• Develop Evaluation Instruments

Carry out Plan for Process Evaluation to Monitor
and Improve Activities
63
How Do I Evaluate My Program?

• Different evaluation phases address different
  questions (Figure 11, page 43)

64
Basics of an Evaluation Plan

• Questions or issues you will address in the
  evaluation
• What you will measure, how, when?

65
Consider the Evaluation Interests of Stakeholders

• Funding agency
• Targeted community
• Outreach staff

66
Interests of Funder

• Were objectives reached?
• What else happened re benefits, changes or
  impact (positive or negative) ?
• What worked the best ?
• What was least successful?
• How satisfied were participants?
• How much time and resources were involved?

67
Interests of Outreach Staff

• Were process objectives realistic?
• If not, why not?
• What worked the best ?
• What was least successful?
• How successful was the program from the
  perspective of the community?

68
Interests of Community

• How successful is the program, product, service
  for me or my community?
• What are the benefits of participating in this
  program or using this service?
• What improvements will be made?

69
Process (Formative) Evaluation

• How are we doing in reaching our objectives?
• Is outreach working as intended?
• How can it be improved?

70
Summative (Outcome) Evaluation

• Helps determine what outreach accomplished
• Evaluation questions address
• Were process objectives met?
• What difference (outcomes) resulted?

71
Choosing the Important Evaluation Questions

• Prioritize by figuring out I need to know
  ____because I need to decide_____.
• Consider the benefits and costs of answering each
  question

72
What Will You Measure How?

• Decisions about data collection, such as
• What indicators and outputs need to be measured
• What type of data will be gathered
• When and from whom (or what) measurements are
  gathered

73
Quantitative Data

• Quantitative--precoded responses
• Produce hard data such as counts, ratings or
  scores
• For example
• numbers of outreach participants
• Percentage of users satisfied with class
  instruction
• Number of website hits by school nurses
• Percentages of users who indicate increased use
  in a follow-up survey

74
Qualitative Data

• In-depth, open-ended interviews
• Direct observation
• Open-ended survey questions, diaries, records
• Develops an understanding--what is happening
• E.g. How is the school nurse website being used?
• What kind of impact does its use have in the
  school nurse program?

75
What Are the Data Sources?

• Self Report
• Written questionnaires
• Interviews
• Observations
• Focus groups
• Meetings
• Records

76
Quality Control

• Validitydoes the instrument measure what was
  intended
• Reliabilityis the answer always consistent
• Cultural appropriateness

77
When to Collect Data?

• Go back to the questions the evaluation will
  address
• Will you document or track process and progress?

78
Will you determine the amount or types of
difference outreach is making?

• Pre-activity/post-activity
• Post-activity only

79
BREAKOUT GROUPS -- EXERCISE 2
80
Evaluation Issues

• Outreach goals and objectives
• Evaluation questions to be addressed
• Types of info/data (quantitative, qualitative)
• Design when from whom collect data
• Data collection what and how collect data
• Resources
• Timeline

81
Planning and Evaluating Outreach
Stage One
Stage Two
Stage Three
Identify Target Community
Conduct Community Profile
Establish Goals Objectives
Develop Activities and Strategies
Stage Six
Conduct Audience Assessment to Tailor Strategies
Share Results and Modify Program as Needed
Develop a Plan to Implement Activities
Stage Five
Carry out Plan to Implement Outreach
Activities
Carry out Plan for Summative Evaluation to Assess
Progress and Impact
Stage Four
Analyze Results for Answers to Evaluation
Objectives

• Establish Evaluation Objectives (Process or
  Summative)
• Select Design and Data Collection Methods
• Develop Evaluation Instruments

Carry out Plan for Process Evaluation to Monitor
and Improve Activities
82
Making Sense of the Data

• Data analysis should be thought about before
  collecting data
• Be sure to leave enough time
• If needed, limit the evaluation goals and reduce
  the number of data collected

83
Where to Get Help

• Handbooks, and websites
• Continuing education courses, workshops, courses
  in other departments
• Collaborate with faculty at your local university

84
Resources

• Rule of thumb
• 5 or more of a programs budget should be
  allotted to evaluation activities
• Different designs require different levels of
  resources

85
Resource Factors

• of participants
• Frequency of data collection travel
• Length of time
• and type of data collection instruments
• Availability of existing data
• Staff expertise with research methods
• Ease of administering instruments
• Willingness of outreach participants

86
Planning and Evaluating Outreach
Stage One
Stage Two
Stage Three
Identify Target Community
Conduct Community Profile
Establish Goals Objectives
Develop Activities and Strategies
Stage Six
Conduct Audience Assessment to Tailor Strategies
Share Results and Modify Program as Needed
Develop a Plan to Implement Activities
Stage Five
Carry out Plan to Implement Outreach
Activities
Stage Four
Carry out Plan for Summative Evaluation to Assess
Progress and Impact

• Establish Evaluation Objectives (Process or
  Summative)
• Select Design and Data Collection Methods
• Develop Evaluation Instruments

Analyze Results for Answers to Evaluation
Objectives
Carry out Plan for Process Evaluation to Monitor
and Improve Activities
87
Using Reporting Results
88
ReassessGoals Objectives

• Has anything changed with your audience or your
  organizations mission to require revisions in
  the original goals and objectives?
• Are some objectives not being met? Why?
• Are there strategies or activities that did not
  succeed? Why?

89
Reporting

• Oral or written
• Summary of results
• Provide purpose, background, methodology of the
  evaluation
• Conclusions and recommendations
• Attachments

90
Disseminating Results

• Identify audiences (funding agency, targeted
  community, staff, professional colleagues)
• Look for other publishing, presentation, or
  promotion opportunities
• Professional meetings and activities, websites,
  listservs, print and e-journals

91
Planning and Evaluating Outreach
Stage One
Stage Two
Stage Three
Identify Target Community
Conduct Community Profile
Establish Goals Objectives
Develop Activities and Strategies
Stage Six
Conduct Audience Assessment to Tailor Strategies
Share Results and Modify Program as Needed
Develop a Plan to Implement Activities
Stage Five
Carry out Plan to Implement Outreach
Activities
Carry out Plan for Summative Evaluation to Assess
Progress and Impact
Stage Four
Analyze Results for Answers to Evaluation
Objectives

• Establish Evaluation Objectives (Process or
  Summative)
• Select Design and Data Collection Methods
• Develop Evaluation Instruments

Carry out Plan for Process Evaluation to Monitor
and Improve Activities
92
Web Resources

• CDC Evaluation Working Group Resources
  http//www.cdc.gov/eval/resources.htm
• Institute on Museum Library Services Outcomes
  Based Evaluation
• http//www.imls.gov/grants/current/crnt_obe.htm
• National Cancer Institutes Usability Site
  http//usability.gov/
• Outreach Evaluation Resource Center
• http//nnlm.gov/evaluation