The Logic and Illogic of Logic Models

1
The Logic and Illogic of Logic Models

• Jay A. Mancini
• March 17, 2008
• ISCE SIG on Evaluation Research

2
Evaluation Research Processes Inquiry and
Discourse

• Asking questions
• Considering elements
• Logic model serves as vehicle for focusing on
  connections

3
The Search for Explanatory Tools

• For clarifying ideas and procedures
• A visual way to Take them by the hand
• We have more experience with words, less so with
  symbols
• A picture, if not worth a thousand words, is at
  least worth a couple of hundred

4
The Essence of Logic Models

• A framework for examining fit
• Examples
• Intervention goals and supportive activities
• Parts and sequences
• Cumulative thinking and procedures
• Tools for determining whether our inquiry and
  discourse are
• Plausible
• Doable
• Testable
• Ultimately knowing how well our words and deeds
  are aligned

5
Two Very General Types of Logic Models

• Project activity model
• Such as service delivery process
• Theory and research model
• Structural model of concepts and measures
• Both seek to avoid
• Theory failure (bad ideas)
• Implementation failure (bad procedures)

6
Results Management Logic Model

• Particular spin on program and evaluation
  research
• Manage results rather than activities
• But one example of using logic models
• Source Mancini, J.A., Huebner, A.J., McCollum,
  E., Marek, L. (2005). Program evaluation
  science and family therapy. In D.H. Sprenkle
  F.P. Piercy (Eds.), Research methods in family
  therapy (2nd Ed.) (pp. 272-293. NY Guilford.

7
Logic Model Template
Note Monitoring and evaluation activities
permeate all aspects of the logic model.
8
Step 1 Identified Needs and Assets

• The first step is to identify needs and assets in
  the community which allow program personnel to
  clearly identify and understand the issues.
• Because this is the basis for the entire program,
  it is important to use accurate and valid
  information rather than uninformed hunches.
• Be sure to look at needs and assets at many
  levels, from the community systems such as
  health, education, legal, etc. to families and to
  individuals.

9

• Needs and assets analysis is a systematic way of
  gathering information that helps set priorities
  for action while also recognizing the assets that
  already exist in a community.
• In the logic model process the information that
  evaluators are gathering to understand severity
  of a problem also assists them in determining
  program priorities, and results that accrue from
  program activities.
• Census data, observations, records and existing
  data bases, and literature reviews are sources of
  existing information to identify needs and
  assets.

10
Step 2 Desired Results

• The second step in the logic model process is to
  articulate the desired result or change you
  expect to see because of your program.
• This is arguably the most important aspect of a
  logic model because it provides the focus that
  all successful programs need.
• At the completion of the program, what will be
  different?

11

• Results should be defined as short-term and long-
  term.
• Specifying results is useful because it enables
  program professionals to have a more realistic
  sense of how parts of the program theory fit, and
  what can reasonably be expected to change over a
  specific time period.

12
Short-Term Results

• Changes in knowledge, attitudes, and/or behaviors
  that are anticipated as a result of participating
  in programs can be considered short-term results.
• More proximal than distal, expected to occur
  sooner rather than later may also be precursors
  to other expected results.

13
Long-Term Results

• System change or changes in organizations,
  community capacity, program support, and/or
  program sustainability may be considered
  long-term results.
• Also longer-term changes in individuals,
  families, and communities Behaviors, attitudes,
  knowledge.

14
Step 3 Indicators

• Indicators flow logically from the desired
  results. How will you know if the results have
  been achieved, with what evidence?
• By determining program indicators, program
  activities and results can be measured and
  tracked. Consequently notions, hunches, hearsay,
  and anecdotes are not good indicator candidates.

15

• Indicators can be far-ranging, including results
  from surveys and focus groups, agency
  administrative data, population data, in-depth
  conversations, observations, as well as other
  measures.
• You must specify program indicators for each of
  your program goals.

16
Step 4 Activities

• Activities describe programs that are implemented
  to achieve the desired results.
• Program activities become important only when
  they are linked to desired results.
• Attention to the choice of programs, research
  upon which it is based, population to be served
  and critical elements of successful programs must
  all be considered.

17
Step 5 Resources

• Resources include the people, stakeholders,
  curricula, spaces, approvals, funds and anything
  else needed to conduct the activities to achieve
  the desired results to meet the identified needs
  and capitalize on the available assets.

18
Logic Model Template
Identified Needs and Assets
Indicators Short Term Long-term
Results Short Term Long-term
Activities
Resources
Note Monitoring and evaluation activities
permeate all aspects of the logic model.
19
Figure 1 Logic model of Children and Divorce
Seminar

• Desired Results
• Short Term
• Parents and children
• gain knowledge on
• improving
• relationships
• Mid-range
• Parental
• communication and
• conflict management
• will improve
• Long-term
• Child adjustment
• post-divorce will
• improve
• Custody and
• visitation court
• disputes will

• Indicators
• N of referrals
• Attendance/
• completion rates
• Pre- and post-
• test parental
• functioning
• scores
• Youth functioning
• scores
• Childrens school
• success
• Rates of returning
• to court to settle
• disputes

• Identified Needs
• Child problems post-
• divorce
• Legal system
• clogged with
• custody and
• visitation disputes
• Parental stress from
• post-divorce conflict

• Activities
• Train referral
• sources
• Provide
• educational
• sessions for
• parents and
• children

• Resources
• Mental Health
• Center
• Judges Court
• Service Workers
• Volunteers

Note Monitoring and evaluation activities
permeate all aspects of the logic model.
20
Figure 1 Logic model of Children and Divorce
Seminar

• Desired Results
• Short Term
• Parents and children gain knowledge on
• improving relationships
• Mid-range
• Parental communication and conflict
• management will improve
• Long-term
• Child adjustment post-divorce will improve
• Custody and visitation court disputes will
• decrease

• Identified Needs
• Child problems post-
• divorce
• Legal system
• clogged with
• custody and
• visitation disputes
• Parental stress from
• post-divorce conflict

Note Monitoring and evaluation activities
permeate all aspects of the logic model.
21
Figure 1 Logic model of Children and Divorce
Seminar

• Indicators
• Number of referrals
• Attendance/completion
• rates
• Pre- and post-test parental
• functioning scores
• Youth functioning
• scores
• Childrens school
• success
• Rates of returning to court
• to settle disputes

• Activities
• Train referral
• sources
• Provide
• educational
• sessions for
• parents and
• children

• Resources
• Mental Health
• Center
• Judges Court
• Service Workers
• Volunteers

Note Monitoring and evaluation activities
permeate all aspects of the logic model.
22
Figure 2 Logic Model of a Youth Development
Community Intervention

• Desired Results
• Short Term
• Class participants will
• demonstrate knowledge
• gain
• Increase in mental
• health resources for
• youth
• Mid-range
• Increased number of
• teens in need will
• receive counseling or
• other treatment
• Long-term
• Incidence of adolescent
• depression will
• decrease

• Indicators
• Completion rate of
• class participants
• Pre and post-test
• measures of
• knowledge gains
• Changes in number
• of teens who
• receive referrals
• Number of
• programs for youth
• Depression
• screening results on
• instrument annually
• in school settings

• Identified Needs
• Lack of information
• about signs
• symptoms of
• depression
• Lack of available
• mental health resources
• Easy access to drugs
• and alcohol
• Poor parental
• monitoring
• Boredom and lack of
• alternative activities for
• teens

• Activities
• Provide school-
• based mental
• health services
• Provide classes
• for teens,
• teachers and
• parents

• Resources
• Community
• Services Board
• Schools
• Parents
• Teens
• City Officials

Note Monitoring and evaluation activities
permeate all aspects of the logic model.
23
Figure 2 Logic Model of a Youth Development
Community Intervention

• Desired Results
• Short Term
• Class participants will demonstrate
• knowledge gain
• Increase in mental health resources
• for youth
• Mid-range
• Increased number of teens in need
• will receive counseling or other treatment
• Long-term
• Incidence of adolescent depression
• will decrease

• Identified Needs
• Lack of information
• about signs symptoms of
• depression
• Lack of available mental
• health resources
• Easy access to drugs and
• alcohol
• Poor parental monitoring
• Boredom and lack of
• alternative activities for teens

Note Monitoring and evaluation activities
permeate all aspects of the logic model.
24
Figure 2 Logic Model of a Youth Development
Community Intervention

• Indicators
• Completion rate of class
• participants
• Pre and post-test measures
• of knowledge gains
• Changes in number of teens
• who receive referrals
• Number of programs for
• youth
• Depression screening results
• on instrument annually in
• school settings

• Activities
• Provide school-
• based mental
• health services
• Provide classes
• for teens,
• teachers and
• parents

• Resources
• Community Services
• Board
• Schools
• Parents
• Teens
• City Officials

Note Monitoring and evaluation activities
permeate all aspects of the logic model.
25
Theories of Change and Logic Models

• Articulation of connections
• Explicit rather than tacit
• Elevation of assumptions
• Following example shows steps in developing a
  successful youth employment initiative

26
Micro-Step Assumptions that Make Up Change
Theory

• Training for attractive occupations is (or can
  be) provided in accessible locations
• Information about its availability will reach the
  target audience.
• When young people hear of the programs
  availability, they will sign up for it.
• They will attend regularly.
• Where necessary, stipends (and perhaps child
  care) will be available to youth while they are
  in training.
• Trainers will offer quality training and they
  will help youth learn marketable skills.
• Trainers will attend regularly and provide
  helpful and supportive counsel.

• Youth will learn the lessons being taught about
  work habits and work skills.
• Youth will internalize the values and absorb the
  knowledge.
• Having attained the knowledge and skills, the
  youth will seek jobs.
• Jobs with adequate pay will be available in the
  areas in which training was provided.
• Employers will hire the youth to fill the jobs.
• The youth will perform well.
• Employers will be supportive.
• Youth will remain on the job and they will become
  regular workers with good earnings.

Carol Weiss Nothing as Practical as Good Theory
27
Research Logic Models

• Two examples from social gerontology
• Study one (R03 funding)
• Study two (R01 proposal)

28
Conceptual Framework  Community Ecology and
Retention of Long-Term Care Employees
RETENTION-RELATED OUTCOMES
ECOLOGICAL LAYERS
COMMUNITY RESULTS
Individual Age Health level Race Gender Psycholog
ical well-being Employment aspirations Related
work history Attitudes toward aging
Retention intentions Job satisfaction Job
competence or readiness Job commitment
Family Marital status Family size Family
supportiveness Family financial needs Quality of
relationships Family responsibilities
Retention
Community Current position/ time in
position Job/position demands Distance between
home and work Closeness to residents Feeling
appreciated at workplace Feeling
rewarded/benefited at work Conflict at
workplace Affinity/liking for coworkers Sense of
community/belonging Time spent with coworkers
off-site Sense of shared responsibility Sense of
collective competence
29
Figure 1 Proposed Ecological Model of
Retention-Related Outcomes, Retention Intentions,
and Employment Outcomes
Ecological Layers
Employment-Related Outcomes
Job Competence Job Commitment Job
Satisfaction Job Match Interactions with Elderly
INDIVIDUAL Psychological Well-Being, Health
Well-Being, Locus of Control, Optimism
EMPLOYMENT STATUS Length of Employment Employed
vs. Not Employed
Retention Intentions
FAMILY AND RELATIONSHIPS Family Responsibilities
and Worries, Relationship Quality and Support,
Competing Family and Work Demands, Family
Structure
COMMUNITY CONNECTIONS Work Community Capacity,
Supervisor Support, Affinity For Coworkers,
Connections with Coworkers (time spent off
site), Work Worries
NOTE Exogenous demographic variables are
gender, race/ethnicity, socioeconomic status, and
age the validity of the model with be examined
within these contexts. Though not represented by
vectors in the model both direct and indirect
effects will be examined across study categories.
30
Program Procedure Logic Model

• Oriented around service delivery elements
• Actually a set of job descriptions
• Simple at first glance but reflect numerous
  complex processes

31
(No Transcript)
32
Program and Research Logic Model

• Teen parent education
• Expected positive outcomes model

33
(No Transcript)
34
Multiple Trails Mediation and Moderation

• Variability in idea complexity
• Degrees of descriptiveness

35
(No Transcript)
36
Program Discharge Logic Model

• Depicts paths taken according to service
  utilization
• Shows ebb and flow between ongoing support and
  little support post-hospitalization

37
(No Transcript)
38
The Plus Side of Logic Models

• Shows how we are conceiving a process or a
  project
• Forces an examination of fit
• How logical are we?
• Is the line-up correct?
• Helps crystallize thinking and procedures
• Visual aid for explanation to others

39
Logic Model Cautions

• Model is only as good as our thinking
• May be overly simplistic
• Theory itself should be simple, but not too
  simple
• May be overly complex
• On its own, does it tell a story?
• If not considered malleable may eventually not
  reflect program and research realities
• Logic model is an approximation and probably
  incomplete
• Boxes, circles, and arrows can be juxtaposed in
  various ways, depending on who is drawing
• Be mindful of discerning association from
  causation
• Model is only as good as our thinking

40
Boxes, Ovals, and Arrows Unleashing Would-be
Artists on Program Processes
ABCDEF
Results
41
References

• Bowen, G.L., Orthner, D.K., Martin, J.A.,
  Mancini, J.A. (2001). Building community
  capacity A manual for U.S. Air Force Family
  Support Centers. Chapel Hill, NC A Better
  Image Printing.
• Mancini, J.A., Huebner, A.J., McCollum, E.,
  Marek, L. (2005). Program evaluation science and
  family therapy. In D.H. Sprenkle F.P. Piercy
  (Eds.), Research methods in family therapy (2nd
  Ed.) (pp. 272-293. NY Guilford.
• Orthner, D.K., Bowen, G.L. (2004).
  Strengthening practice through results
  management. In A.R. Roberts K. Yeager (Eds.),
  Handbook of practice based research (pp.
  897-904). NY Oxford.
• Rossi, P.H., Lipsey, M.W., Freeman, H.E.
  (2004). Evaluation A systematic approach (7th
  ed.). Thousand Oaks, CA Sage.
• Weiss, C.H. (1995). Nothing practical as good
  theory. In J.P. Connell, A.C. Kubisch, L.B.
  Schorr, C. H. Weiss (Eds.), New approaches to
  evaluating community initiatives Vol. 1,
  Concepts, methods, and contexts (pp. 65-92).
  Washington, DC Aspen.

42
Jay A. Mancini

• Jay A. Mancini is the Senior Research Fellow at
  Virginia Techs Institute for Society, Culture,
  and Environment, and also a Professor in the
  Department of Human Development (mancini_at_vt.edu)