Logic%20Model%20of%20Program%20Performance

1
Logic Model of Program Performance

• Ensuring our actions - (INPUTS) are
  resulting in desired (OUTCOMES)

2
The Big Five

1. View evaluation as learning-integrated into the
   way we work
2. Build evaluation in upfront
3. Ask tough questions
4. Make measurement meaningful
5. Be accountable for highest professional standards

3
Why the Logic Model?

• Shows difference between what we do and impact we
  are having
• Provides a common vocabulary
• Focus on quality and continuous improvement

4
Why Measure?

• What gets measured gets done
• If you dont measure the results, you cant tell
  success from failure
• If you cant see success, you cant reward it
• If you cant reward success, youre probably
  rewarding failure
• If you cant see success, you cant learn from it
• If you cant recognize failure, you cant correct
  it
• If you can demonstrate results, you can win
  public support

5
The logic model contains six components with
Inputs-Outputs-Outcomes being central to the
model

1. Situation the context and need that gives rise
   to a program or initiative logic models are
   built to respond to existing and projected
   situations.
2. INPUTS the resources, contributions and
   investments that are made in response to the
   situation.
3. OUTPUTS the activities, products, methods,
   services that are implemented.
4. OUTCOMES the targeted and measurable results and
   benefits for individuals and/or the school
   community.
5. Environment Where the program exists and what
   influences the implementation and success of the
   initiative, including politics, economic and
   human resource factors, etc.
6. Assumptions the beliefs we have about the
   program, the participants and the way we expect
   the program to operate the principles that guide
   our work. Assumptions have a significant and
   direct impact on expected outcomes

6
Logic Model
INPUTS
OUTPUTS
OUTCOMES
SITUATION
7
Everyday Logic Model
Get Food
Eat Food
Feel Better
HUNGER
8
Logic Model What Is It?

• Graphic representation of the program theory or
  action
• Relationship between inputs, outputs and outcomes
• Logical chain of if-then relationships
• This is at the core of effective program
  development

9
Logical LinkagesSeries of If-Then Relationships
IF THEN IF THEN
IF THEN IF THEN
Program invests time money
Students will know what is available
Students know/ access services available
Resource inventory can be developed
Students will have needs met
INPUTS OUTPUT OUTCOMES
10
Logic Model Program Performance Framework
INPUTS What we invest Staff District
support Students Parents Volunteers Money Material
s Equipment Technology
OUTPUTS Activities
Participation What we do
Who we reach Workshops
Students Meetings
Parents Counseling
Staff Facilitation
Community Assessments Product Dev.
Media Work Teaching
OUTCOMES Short Medium Long Term What the
What the What the short term medium term
impact(s) Is results are results are




Learning Action Conditions
Awareness Behavior Social Knowledge
Practice Economic
Attitudes Decisions Civic Skills Policies
Opinions Socials action Aspirations Mo
tivations
SITUATION
ENVIRONMENT Influential Factors
11
A K-12 Education ExampleAfter School Academic
Recovery Program

• The School/District invests time and resources
• A variety of educational activities are provided
  for students
• Students who participate gain knowledge and
  change personal learning practices resulting in
  improved academic achievement

12
LOGIC MODEL Program Performance
INPUTS OUTPUTS
OUTCOMES
Activities
Participation
Short
Medium
Long Term
Program investments
What we What we Who we
What results Invest do
reach
13
INPUTSWhat were putting in . . .

• Staff
• Money
• Time
• Volunteers
• Partners
• Equipment
• Technology

14
OUTPUTSWhat were producing . . .
WHAT WE DO WHO
WE REACH
ACTIVITIES
PARTICIPATION
Workshops Counseling Educational
Research Facilitation Program Development Curricul
um Design Training Conferences Media
Students Parents Staff Community
15
OUTCOMESWhat results for students?
SHORT Learning
MEDIUM Action
LONG TERM Conditions
Awareness Knowledge Attitudes Skills Opinion Aspir
ations Motivation
Behavior Practice Decisions Policies
Academic Social Emotional Civic
16
INPUTS
OUTPUTS
OUTCOMES
Students increase knowledge of Study Skills
Designed Curriculum
Staff
Students use new learning practices
Improved Student Achievement
Targeted audience attends
Money
Materials
Provide program
Students learn new ways to allocate study time
17
ASSUMPTIONS

• Beliefs about the program
• -the participants
• -the way the program will operate
• -how resources, staff will be engaged
• -the theory of action
• Assumptions are often the reason for poor results
• Check and test assumptions
• -Identify potential barriers for each if-then
  sequence

18
OUTCOMES vs. ACTIVITIES

• BE OUTCOME DRIVEN,
• NOT ACTIVITY DRIVEN

19
WHAT ARE OUTCOMES?
Outcomes are the benefits or results of a
program. They are changes or improvements for
students, staff, parents, or the organization
that occur during or after the program. Outcomes
represent the difference between the original
condition and that which exists after
implementation.
Be outcome driven, not activity-driven.
ACTIVITY DRIVEN
OUTCOME DRIVEN

• Examples
• To provide after school study sessions
• To teach algebra concepts to students
• To engage students in best-practice study methods

• Examples
• Students will have increased capacity to
  successfully complete homework
• Student algebra test scores will improve
• Students will demonstrate the ability to match
  study skill with learning task

20
CHAIN OF OUTCOMES
Outcomes often fall along a continuum from
shorter to longer-term results.
IMMEDIATE OUTCOMES SHORT TERM LEARNING
FINAL OUTCOMES LONG TERM IMPACTS
INTERMEDIATE OUTCOMES MEDIUM TERM ACTION
TIME
EXAMPLE
Immediate
Intermediate
Final
Students know how to develop a learning
plan Students increase their knowledge of and
ability to complete algebraic equations Students
are more aware of school services Students learn
to develop a college-prep plan
Students access needed class offerings Student
enrollment in advanced math courses
increases Student services performance
improves Admissions to desired colleges increases
Students use a learning plan Student algebra
grades improve Students access appropriate school
services Students use and follow a college-prep
plan
21
HOW FAR OUT THE OUTCOME CHAIN DO WE GO?

• What is logical?
• What is realistic?
• What is meaningful?

22
WHICH OUTCOME?
The questions often arises Which outcome in the
chain will be the point where the value of the
program will be assessed and judged? What is
good enough?

1. The school will develop and facilitate an after
   school academic recovery program.
2. Small (3-5) student learning groups will be
   established and function with the direction and
   instruction of a teacher.
3. Homework completion will increase through the use
   of the student learning groups
4. Student achievement will improve with the
   assistance of the student learning groups that
   are facilitated by the teacher

23
When Selecting Outcomes think about

• Importance Which outcomes are the most
  important?
• Meaningfulness Which outcome or benefit is
  meaningful for the participants and stakeholders?
• Realistic What is realistic given the nature of
  the problem and what we can expect to influence?
• Reasonable Which outcomes are reasonable
  considering our investment and what we did and
  whom we reached?
• Outcomes must realistically reflect the changes,
  benefits, and improvements that the program can
  influence.

24
Issues in Defining Outcomes

• There is no right number of outcomes. The number
  of outcomes selected by your program will depend
  upon the nature and purpose of the program,
  resources, size and number of constituencies
  represented, (i.e. students, staff, parents,
  etc.).
• There may be more than one outcome track. The
  program may have several chain of events,
  usually linked to different target groups or
  programming components.
• In some cases, immediate outcomes may seem like
  outputs. The misinterpretation occurs because
  initial outcomes may not represent major change.
  Critically ask yourself, Is this an outcome or
  an output? This is similar to instructional vs.
  learning objectives.
• The more immediate the outcome, the more
  influence the program has over its achievement.

25
Issues in Defining Outcomes Cont..

• Conversely, the longer term the outcome, the less
  direct influence the program has over its
  achievement and the more likely other extraneous
  forces are to intervene. The timing of the
  measurement strategy is a critical design
  decision, influencing the reliability and
  validity of the measurement strategy and
  resulting data.
• Because other forces affect an outcome doesnt
  mean it shouldnt be included. Despite the
  influence of other factors, you will want to
  measure and track these outcomes in order to
  understand what effect the program has and what
  might be done to achieve the desired results.
• Long-term outcomes, however, should not go beyond
  the program purpose or target audience. Think
  about what the program is designed to do-where
  its influence is likely to be felt and focus the
  outcome measurement at that level. Likewise,
  keep the the outcome measures focused on the
  targeted audience.

26
What does a logic model look like?

• Often it is a graphic display of boxes or columns
  arranged in a horizontal or vertical display
• Arrows may be used to depict causal
  relationships, sequence of events
• Level of detail in the logic model depends upon
  your purpose and scope of the project
• SIMPLE logic models are useful when communicating
  with externals.
• More DETAILED logic models are useful when
  developing consensus internally among design team
  members.

27
SAMPLE LOGIC MODEL
OUTPUTS
OUTCOMES
INPUTS












28
SAMPLE LOGIC MODEL
OUTPUTS
OUTCOMES
INPUTS














29
SAMPLE LOGIC MODEL
INPUTS
OUTPUTS
OUTCOMES






30
LIMITATIONS OF LOGIC MODEL

• Logic model only represents reality, it is not
  reality
• Programs are not linear
• Programs are dynamic interrelationships that
  rarely follow sequential order
• Logic model focuses on expected outcomes also
  need to pay attention to unintended or unexpected
  outcomes positive, negative, neutral
• Program is likely to be just one of many factors
  influencing outcomes
• Consider other factors that may be affecting
  observed outcomes

31
BENEFITS

• Brings detail to broad goals
• Shows the chain of events that link inputs to
  outcomes
• Builds understanding and consensus
• Identifies gaps in logic and uncertain
  assumptions
• Signals what to evaluate and when
• Summarizes a complex program to communicate with
  externals

32
BUILDING A LOGIC MODEL

• New program
• Existing program
• Team and Collaboration
• Involvement of others
• Keep it dynamic

33
CREATING A LOGIC MODEL Where to Start?
The process of developing a logic model should
bring all key stakeholders to a shared
understanding of what the program is and what it
will do.

• AN EXISTING PROGRAM
• You might start by asking, What is it that we
  do? What are we hoping to accomplish? What
  does our program consist of? Who are we
  reaching? What results are we seeking?
• A NEW PROGRAM
• If you are in the planning stage of a new
  program, you might start with the long-term
  expected end resultthe impactand work
  backwards. What is our long-term desired
  result? What will be different as a result of
  this program? What must happen in each
  preceding step to get us there?
• A logic model is dynamic. It will change as the
  program changes.

34
LOGIC MODEL WORKSHEET
Program__________________________________
Goal__________________________________
OUTPUTS
INPUTS
OUTCOMES-IMPACT
Activities
Participation
Short
Medium
Long-Term
35
CHECK YOUR LOGIC MODEL

• Are the outcomes really outcomes?
• Is the longest-term outcome
• -meaningful?
• -logical?
• -realistic
• Does it represent the programs purpose response
  to the situation?

36
HOW GOOD IS YOUR LOGIC MODEL?

• Ask Yourself
• Is each listed outcome truly an outcome? Does
  the logic model clearly separate outcomes from
  outputs, or are the distinctions blurred?
• Does the highest-level outcome represent a
  meaningful benefit or value to the target
  audience? Can it be associated with the program?
• Is the model truly logical? Do the relationships
  among the program elements make sense? Are the
  casual relationships supported?
• (1) Start at inputs, ask why? at each level
  why do we need these inputs? Why do we need to
  conduct these activities?
• (2) Starting at the output level ask, how are we
  going to produce these outcomes by looking at the
  items immediately preceding it.

37
HOW GOOD IS YOUR LOGIC MODEL Continued . . .

• Ask Yourself
• Are the resources realistic? Is what you intend
  to do even possible given your resources?
• How valid are the assumptions? Are they based on
  experience and research, or are they your best
  guesses?
• Does the logic model reflect the opinions and
  support of key stakeholders? Were any
  stakeholders left out?

38
EVALUATION PLAN
39
EVALUATION QUESTIONS-What do you want to know?

• QUESTION TYPE OF EVALUATION
• Need? Needs assessment
• Process or implementation? Process
  evaluation
• Outcomes or impact? Performance
  analysis
• Costs and efficiencies? Cost-benefit/
  analysis
• ROI return on
• 
  investment

40
INDICATORS- HOW WILL YOU KNOW IT?

• The evidence or measures that indicates what you
  wish to know or see
• Often multiple indicators are necessary
• May be quantitative or qualitative

41
INDICATORS (Data)

• An indicator is the evidence or information that
  represents the phenomena you are asking about.
• For example
• EXAMPLE
• Indicator of academic achievement Improved
  CSTs
• Indicator of improved performance Improved
  student projects
• Indicators help you know something. They are
  measurable or observable they can be seen
  (e.g., observed behavior), heard (e.g.,
  participant response), or read (i.e., student
  work).
• For each aspect you want to measure, ask
  yourself What would it look like? If I were a
  visitor to the program, what would I see? Invite
  others to give their perspectives. Check your
  ideas with others.

42
INDICATORS continued . . .

• Indicators should be
• Sensible Indicators need to make sense in
  relation to that which you are asking about.
• Direct An indicator should measure as directly
  as possible what it is intended to measure. For
  example, if the outcome being measured is
  improved attendance, then the best indicator is
  the schools attendance records. The number and
  percent of students that receive attendance
  improvement education does not directly measure
  the desired result.
• Proxy Often, however, we do not have direct
  measures or we may be constrained by time and
  resources. Then, we use proxy measures as our
  best guesses. For example, a proxy measure of
  student satisfaction with educational programs
  might include enrollment patterns and changes in
  classroom performance.

43
INDICATORS CONT..

• Specific Clearly define the indicator so that
  anyone would understand it in the same way would
  collect the same data. Example Number and
  percent of students who adopted a best learning
  practice in the past year-which learning
  practices, which students, what is the time
  period, what constitutes adoption???
• Useful Indicators need to help us understand
  what it is we are measuring
• Practical Indicators need to be practical that
  is we need to be able to collect the data in a
  timely manner at reasonable cost. Is it
  reasonable to be able to collect the data given
  our existing or potential resources?
• Adequate There is no correct number or type
  of indicators. The number of indicators you
  choose depends upon the result being measured,
  the level of information you feel that you need,
  and the resources available. Often more than one
  indicator is necessary to capture a concept.
  However, too many is problematic also. A large
  number of indicators may mean that the result is
  too complex or not understood well enough.

44
INDICATORS CONT..

• Quantitative-qualitative Often indicators are
  expressed as a number or percent that shows
  attainment number of. percent of ratio
  ofincidence of.. proportion of..
• Indicators do not always have to be a number
  Qualitative indicators may be important.
  Example indicators are frequently qualitative
  proficient, and/or mastery level performances.
• Not everything that counts can be counted
• Comprehensive We should include indicators that
  express all possible aspects of what we are
  wanting to measure, i.e., possible negative or
  detrimental aspects as well as the positive. What
  might be some negative effects or spin-offs of
  the program and include indicators for these.

45
INDICATORS Worksheet
OUTCOME
INDICATORS







46
INDICATOR REVIEW

• Check your indicators according to the criteria
  below.
• Do the indicators make sense?
• Are the indicators directly related to the
  outcome? Do they provide evidence of what you
  want to know?
• Are the indicators specific and clearly define
  what information will be used?
• Do the indicators provide useful information?
• Is it practical to think that the data can be
  collected in a timely fashion with the resources
  available?
• Do the indicators adequately measure the concept?
• Are the indicators comprehensive and consider
  possible negative or detrimental effects?

47
INDICATORS-CONSIDERATIONS

• Direct (proxy)
• Sensible, understandable
• Reliable, trustworthy
• Available
• Useful, credible

48
SOURCE AND METHOD OF DATA COLLECTION

• Source of Information Method of
  Collecting
• Information
• Participants/Students Survey
• Parents Interview
• Staff/District Observation
• Community End-of-Program
• Questionnaire
• Focus group
• Specific school
• data

49
EVALUATION PLAN

FOCUSING THE EVALUATION
COLLECTING THE INFORMATION  
 
50
EVALUATION PLAN CONT

• How will the data be analyzed and interpreted? By
  whom and when?
• How will the results be communicated? By whom and
  when?
• Who is the receiving audience?

51
The Big Five

• View evaluation as learning-integrate into the
  way we work
• Build evaluation in upfront
• Ask tough questions
• Make measurement meaningful
• Be accountable for highest professional standards