Evaluation 101: Introduction to the Logic Model

1
Evaluation 101 Introduction to the Logic Model

• Holly S. Ruch-Ross, ScD

2
The speaker in this session has no relevant
financial relationship with the manufacturer of
any commercial product and/or provider of
commercial services discussed in this CME
activity. The speaker will not discuss or
demonstrate pharmaceuticals and/or medical
devices that are not approved by the FDA and/or
medical or surgical procedures that involve an
unapproved or "off-label" use of an approved
device or pharmaceutical.
3
Objectives Participants will

• Understand the use of a logic model for program
  planning, implementation and evaluation
• Understand process and outcome evaluation
• Identify the components of a logic model
• Develop a logic model for your project using the
  process and steps presented

4
Defining Evaluation

• While evaluation is not research, when conducting
  an evaluation we use research methods to gather
  information that will be actively used for
• Program development and improvement
• Program replication
• Resource allocation
• Policy decisions

5
Reasons to Evaluate

• Checking Your Process Are you doing what you
  said you would do?
• Determining Your Impact Are you having the
  desired effect in the target population?
• Building Your Base of Support Can you generate
  information and evidence to share with funders
  and other stakeholders?
• Replication Justification Is there evidence to
  support replication of this program?

6
Review The Evaluation Cycle
START
Step 2 Implement program and begin to collect
evaluative data
Step 4 Adjust program as data suggests adjust
evaluation
Step 3 Review data. Are you doing what you
planned? Are you impacting the need you
identified?
7
Review Types of Evaluation

• Process Evaluation
• Is the program being implemented the way it was
  designed?

• Outcome Evaluation
• Is the program having the intended effect?

8
Process Evaluation

• Describe the program and implementation, who
  participates in the program, what services are
  received.
• Information such as number served, patient
  characteristics, number of contacts with a
  program, number of trainings, number of referrals.

9
Outcome Evaluation

• Detect whether the intervention made a
  difference, what changes can be measured
  (knowledge, attitude, behavior, health status,
  incidence, prevalence)
• Longer term outcomes may need to be assessed
  using shorter term indicators.

10
Goals and Objectives

• Goal broad statement of what the program would
  like to accomplish for a specific target
  population.
• Objective measurable step toward the
  achievement of a goal.
• Who will do
• What by
• When

11
A GOOD OBJECTIVE IS SMART

• Specific
• Measurable
• Achievable
• Realistic for the program
• Time specific

12
What is a logic model?If you dont know where
youre going, you might wind up someplace
else.-Yogi Berra

• Adapted from a business model
• A brief (preferably one page) summary of your
  program
• A snapshot of the logical sequence of steps
  connecting resources to intended results
• Diverse templates, but the central idea remains
  the same

13
How is a logic model useful for my program?

• Summary that is easy to share with new staff,
  boards, funders
• Helps stakeholders keep focus on what is most
  important for program
• Facilitates program planning, implementation and
  evaluation
• Process of developing LM is itself useful for
  building consensus by planning together and for
  identifying gaps or problems in your planning.

14
The Logic Model
15
Logic Model Example Sarah and the Prevention
First Program

• Sarah is the program director of Prevention First
• Large multi-agency collaborative
• Community has
• High mobility
• Low income
• Limited/no English
• Various immigrant groups

16
Logic Model Example Sarah and the Prevention
First Program

• The program intends to
• bring together the diverse resources and
  expertise present in the collaborative
• facilitate the use of preventive health care by
  this community
• increase public awareness of the many free,
  non-emergency health and dental services
  available in the community.

17
Prevention First Goals and Objectives

• Goals
• Immigrant families will understand the importance
  of prevention.
• Immigrant families will use preventive health
  services
• Objectives
• Within the first 6 months of the project, we will
  conduct a focus group with immigrant parents to
  explore possible barriers to the use of
  prevention services.
• By the end of year 1, we will have made
  presentations to staff of at least 4 agencies
  serving immigrant families to promote preventive
  health services and encourage referrals.
• By the end of year 1, participating immigrant
  families will schedule and complete an increased
  number of well-child visits over base line.

18
Logic Model Example STEP ONEPrevention First
Who does the program serve? What problem does
the program address? Be as specific as possible.
19
Logic Model Example STEP TWOPrevention First
What resources are available to the
program? Consider resources other than the
immediate dollars and staff specified for the
program.
20
Logic Model Example STEP THREEPrevention First
What does the program do in its day to day
operations?
21
Logic Model Example STEP FOURPrevention First
What do participants get or do as a part of
their program participation?
22
Logic Model Example Prevention First
23
Logic Model Example Outputs vs Outcomes

• Outputs are the direct products of program
  activities (number of participants attending,
  number of sessions, materials produced and
  distributed).
• Outputs reflect program implementation, and are
  usually measured as a part of process evaluation.

• Outcomes are changes in the target population
  that result from the program (changes in
  knowledge, attitude, behavior, health status.
• Outcomes reflect program impacts, and are usually
  measured as a part of outcome evaluation.

24
Logic Model Example Where are my Goals and
Objectives?

• Your logic model can be seen as an expression of
  your goals and objectives.
• Goals are most often reflected in the target
  population and outcomes columns.
• Objectives are most often reflected in the
  activities, outputs and outcomes columns.

25
Getting Started on Your Logic Model

• Think very specifically about one program.
• Keep in mind that your logic model will be useful
  for planning, implementation, and evaluation.
• Consider who your audiences might be.
• Work as a team if possible.
• Dont make it too hard the goal is a snapshot
  of your program dont worry too much about the
  jargon.
• Its okay to work backwards if that is easier
  for you beginning with the end in mind.

26
Your Logic Model STEP ONE
Who does the program serve? What problem does
the program address? Be as specific as possible.
27
Step One - Questions and Reflections

• Was this step easy or difficult?
• What do you need more information about?
• Whose help do you need?

28
Your Logic Model STEP TWO
What resources are available to the
program? Consider resources other than the
immediate dollars and staff specified for the
program.
29
Step Two - Questions and Reflections

• Was this step easy or difficult?
• What do you need more information about?
• Whose help do you need?

30
Your Logic Model STEP THREE
What does the program do in its day to day
operations?
31
Step Three - Questions and Reflections

• Was this step easy or difficult?
• What do you need more information about?
• Whose help do you need?

32
Your Logic Model STEP FOUR
What do participants get or do as a part of
their program participation?
33
Step Four - Questions and Reflections

• Was this step easy or difficult?
• What do you need more information about?
• Whose help do you need?

34
Pre-Step FiveIdentifying Outcomes

• An outcome is a measurable change that occurs in
  your target community or population beyond the
  point of service or intervention. Outcomes
  define the logical and desired results of the
  services your program provides.
• Measuring outcomes is a way to detect whether
  your program is making a difference.
• Long term outcomes may need to be assessed using
  shorter term indicators.
• Limit the number of outcomes so that you can
  focus resources.
• Use standard outcomes when appropriate.

35
Your Logic Model STEP FIVE
36
Taking it Home What comes next?

• Who should you share/work on this with? When?
  Why?
• How could this be used to guide program planning
  and implementation?
• How might it inform you evaluation planning and
  your evaluator?

37
Where to find more information

• Evaluation Resources on the AAP web site
  http//www.aap.org/commpeds/resources/evaluation.h
  tml
• You will find information on
• Logic models
• Outcomes
• Questionnaire Design
• Evaluation measures instruments
• Also on the web site, you will find
• Evaluating Your Community-Based Program, Part 1
  Designing Your Evaluation
• Evaluating Your Community-Based Program, Part 2
  Putting Your Evaluation Plan to Work (Fall 2008)

38
Were lost, but were making good time.-Yogi
Berra