A Hierarchy for Targeting Outcomes and Evaluating Their Achievement

1
A Hierarchy for Targeting Outcomes and Evaluating
Their Achievement http//deal.unl.edu/TOP/
S. Kay Rockwell Professor and Evaluation
Specialist Agricultural Leadership, Education and
Communications Department University of Nebraska
2
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE
Outcomes
Conditions
Practices
Practices
KASA
KASA



Reactions
Reactions
Participation
Participation
Activities
Activities
K Knowledge
S Social
Resources
Resources
A Attitude

E Economic
S Skill

E Environment

A Aspirations

3
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE
Outcomes
Conditions
Practices
Practices
KASA
KASA


Reactions
Reactions
Participation
Participation
Activities
Activities
K Knowledge
S Social
Resources
Resources
A Attitude

E Economic
S Skill

E Environment
A Aspirations

4
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
Program
Development
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE
Outcomes
Conditions
Practices
Practices
Assess Needs
KASA
KASA



Reactions
Reactions
Participation
Participation
Activities
Activities
K Knowledge
S Social
Resources
Resources
A Attitude
E Economic
S Skill
E Environment
A Aspirations
5
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE
Outcomes
Conditions
Practices
Practices
KASA
KASA




Reactions
Reactions
Reactions
Participation
Participation
Participation
Activities
Activities
K Knowledge
S Social
Resources
Resources
A Attitude

E Economic
S Skill

E Environment
A Aspirations

6

• Stakeholder
• a person, inside or outside the organization, who
  has
• a real, active interest in the organization and
  its programs
• an investment in the organization/program (time,
  mental/emotional energy, money) and
• a commitment to the organization's/program's
  success.

7
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE
Outcomes
Conditions
Practices
Practices
KASA
KASA



Reactions
Reactions
Participation
Participation
Activities
Activities
K Knowledge
S Social
Resources
Resources
Resources
A Attitude

E Economic
S Skill

E Environment

A Aspirations

8
Program vs. Activity
Print Media
Multi-media Courses
Direct Teaching
Publications
Video Conferencing
Programs
Tours
Home Study Courses
Demonstration Sites
9
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE
Outcomes
Outcomes
Conditions
Practices
Practices
Practices
KASA
KASA



Reactions
Reactions
Participation
Participation
Activities
Activities
K Knowledge
S Social
Resources
Resources
A Attitude

E Economic
S Skill

E Environment


10

• Inputs
• Staff and Volunteers
• Financial Resources
• Organization/Agency Commitments
• Grant Dollars

11
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE
Outcomes
Conditions
Practices
Practices
KASA
KASA


Reactions
Reactions
Assess Opportunities
Participation
Participation
Activities
Activities
K Knowledge
S Social
Resources
Resources
A Attitude

E Economic

S Skill

E Environment
A Aspirations

12
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE
SEE
Outcomes
Outcomes
Conditions
Practices
Practices
Practices
KASA
KASA


Reactions
Reactions
Participation
Participation
Activities
Activities
K Knowledge
S Social
Resources
Resources
A Attitude

E Economic
S Skill

E Environment
A Aspirations

13
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE
Outcomes
Outcomes
Conditions
Practices
Practices
KASA
KASA


Reactions
Reactions
Participation
Participation
Activities
Activities
K Knowledge
S Social
Resources
Resources
A Attitude

E Economic

S Skill

E Environment
A Aspirations

14
OUTPUTS Products or services generated for
programming

• Newsletters
• Workshops
• Displays
• Demonstration Projects
• Etc.

• Coalitions developed
• Curricula developed
• Consultations
• Etc.

15
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE
SEE
Outcomes
Outcomes
Conditions
Practices
Practices
Practices
KASA
KASA



Reactions
Reactions
Reactions
Participation
Participation
Participation
Activities
Activities
K Knowledge
S Social
Resources
Resources
A Attitude

E Economic
S Skill

E Environment

A Aspirations

16
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE
SEE
Outcomes
Outcomes
Conditions
Practices
Practices
KASA
KASA


Process evaluation
Reactions
Reactions
Reactions
Participation
Participation
Participation
Activities
Activities
Activities
K Knowledge
S Social
Resources
Resources
Resources
A Attitude

E Economic
S Skill

E Environment

A Aspirations

17
Process Evaluation
Aim provide information for program improvement
modification management (Developmental or
Formative)
18
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE
Outcomes
Outcomes
Conditions
Practices
Practices
KASA
KASA


Reactions
Reactions
Participation
Participation
Activities
Activities
K Knowledge
S Social
Resources
Resources
A Attitude

E Economic
S Skill

E Environment
A Aspirations

19
Outcomes
The effects that the products and services
(outputs) have on customers or clientele
Behavior changes of individuals, groups,
organizations, or communities. Attributed in
part to the efforts and influences of the agency,
program or project (Kibel, 1999)
20
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE
Outcomes
Conditions
Long-term
Practices
Practices
Intermediate
KASA
KASA


OUTCOMES
Reactions
Immediate
Reactions
Participation
Participation
Activities
Activities
K Knowledge
S Social
Resources
Resources
A Attitude

E Economic
S Skill

E Environment
A Aspirations

21
Impacts

• Lasting and generalized changes that occur
  because of the..
• Activity(ties)
• Program(s)
• Grant(s)

22
Impact Evaluation
Form of outcomes evaluation Assess net effect of
a program Compare
Estimate of what would have happened in the
absence of the program
Program Outcomes
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
Indicators
Specific items of information that trace a
programs success
They Describe .observable characteristics .meas
urable characteristics .or changes that
represent achievement of an outcome
27
Objective Indicators

• Direct observation of what people overtly do or
  receive, as well as observations of natural
  phenomena
• Statistical records
• Objective tests
• Third party observations
• Etc.

28
Subjective Indicators

• Self-report processes by participants
  themselves and/or other who may be affected by
  the program
• Example 1 Public satisfaction with personal
  health economic status and cleanness of public
  air, land, and water
• Example 2 reports/ratings by program teams or
  program participants of their adoption and use of
  recommended practices and technologies.

29
(No Transcript)
30
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE
Outcomes
Conditions
Practices
Practices
KASA
KASA


Reactions
Reactions
Participation
Participation
Activities
Activities
K Knowledge
S Social
Resources
Resources
A Attitude

E Economic

S Skill

E Environment

A Aspirations

31
Targeting Outcomes of Programs (TOP)
http//deal.unl.edu/TOP/




Program
P
Development
Program
Program
Performance
Performance
Feedback
Feedback
SEE
SEE

Outcomes
Conditions
Practices
Practices
KASA
KASA



Reactions
Reactions
Participation
Participation
Activities
Activities
K Knowledge
S Social
Resources
Resources
A Attitude

E Economic
S Skill

E Environment
A Aspirations

32
Why Assess needs and evaluate?

• Help establish goals and objectives
• Test if met goals and objectives
• Identify outcomes/impact on..
• Individuals
• Community
• Provide documentation for stakeholders
• Document accountability

33
A Hierarchy for Targeting Outcomes and Evaluating
Their Achievement http//deal.unl.edu/TOP/
S. Kay Rockwell Professor and Evaluation
Specialist Agricultural Leadership, Education and
Communications Department University of Nebraska