Assessment Tools for Program Evaluation

1
Assessment Tools for Program Evaluation

• California Higher Education
• Foster Youth Consortium
• October 27, 2009

2
Session Outcomes

• Describe how planning assessment are linked.
• See value in program assessment.
• Identify basic elements of a program logic model
  (inputs, outputs, outcomes).
• Describe how a program logic model guides
  assessment.
• Use an evaluation plan worksheet to practice
  designing an assessment of a financial literacy
  program for former foster youth.

3
Definitions

• Assessment
• Any effort to gather, analyze, interpret
  evidence which describes institutional,
  departmental, divisional, or agency
  effectiveness. p.18
• Evaluation
• Any effort to use assessment evidence to improve
  institutional, departmental, divisional or agency
  effectiveness. p. 19

Upcraft, M.L. Schuh, J.H. (1996). Assessment in
student affairs A guide for practitioners. San
Franciso Jossey-Bass.
4
Basic Assessment Questions

• What is our destination? (outcomes)
• How will we get there? (activities)
• What resources will we need to get there?
  (inputs)
• What will tell us were going the right way
  have arrived? (assessment)

5
Reasons Why We Wouldnt Want to Do This

• We already do this in our own, personalized way.
• It takes time theres already too much to do.
• It takes us away from delivering services,
  programs to students.
• Its not clear how it will lead to increased
  effectiveness.
• It is impossible to precisely determine success.
• Its hard. Its not fun. We dont like it. We
  dont know how.

6
Why Assessment Still Makes Sense

• Helps us clarify focus on outcomes
• Builds understanding of our programs
  accomplishments
• Makes our assumptions explicit
• Helps us determine progress effectiveness
• Helps prioritize allocate resources
• Helps us to gain support resources
• Helps to guide improvement
• Helps us respond to accountability / transparency
  expectations

7
Assessment Planning

• Are linked intertwined
• Start with the end in mind

8
How do we begin?

• A logic model road map for planning
  assessment
• Picture of your program showing what it will do
  and what it is to accomplish
• Graphic representation of the theory of action
  what is invested, what is done, what results
• Framework to guide your work that integrates
  program planning implementation evaluation

Taylor-Powell, E. Henert, E. (2008). Developing
a logic model Teaching and training guide.
University of Wisconsin-Extension. Board of
Regents of University of Wisconsin System.
9
Simplest Form
Inputs What we invest
Outputs What we do
Outcomes What results
Taylor-Powell, E. Henert, E. (2008). Developing
a logic model Teaching and training guide.
University of Wisconsin-Extension. Board of
Regents of University of Wisconsin System
10
Logic Model Can Be Applied to

• A process
• Small program
• Large multi-component program
• Organization
• Everyday experiences

Taylor-Powell, E. Henert, E. (2008). Developing
a logic model Teaching and training guide.
University of Wisconsin-Extension. Board of
Regents of University of Wisconsin System
11
Everyday Logic ModelDirty Laundry
Inputs What we invest
Outputs What we do
Outcomes What results
detergent washing machine water time person
sort laundry select wash cycle start machine put
detergent in machine put dirty laundry in machine
Clean laundry
12
So, to plan an effective program all I need to do
is figure out outcomes, outputs, inputs?

• Not quite..........
• Need to examine assumptions
• Beliefs we hold about program how it will
  work.
• All programs have assumptions often basis for
  failure or less than expected results.
• Logic model helps make assumptions explicit.
• Assumptions in laundry model?
• Need to align program with mission, needs,
  priorities

13
Mission, Needs, Assumptions Everyday Logic
Model
Inputs What we invest
Outputs What we do
Outcomes What results
Mission Priorities To meet safety health needs
of family
Problem / Needs No food in house Dirty laundry
detergent washing machine water time person
sort laundry set wash cycle start machine add
detergent add laundry
clean laundry
wool sweater shrunk undies stuck in machine
agitator white tee-shirts turned pink

• Assumptions
• All laundry is
• washable.
• Washing machine
• works properly.

Taylor-Powell, E. Henert, E. (2008). Developing
a logic model Teaching and training guide.
University of Wisconsin-Extension. Board of
Regents of University of Wisconsin System.
14
Where does evaluation fit? From beginning to end
activities
outcomes - impact
inputs
situation
Evaluation Focus-Collect Data- Analyze
Interpret-Report
15
Evaluation Plan Worksheet

• FOCUS What will we evaluate?

Taylor-Powell, E. Henert, E. (2008). Developing
a logic model Teaching and training guide.
University of Wisconsin-Extension. Board of
Regents of University of Wisconsin System.
16
Evaluation Plan Worksheet

• FOCUS What will we evaluate? Effectiveness of
  program to clean dirty laundry.

Taylor-Powell, E. Henert, E. (2008). Developing
a logic model Teaching and training guide.
University of Wisconsin-Extension. Board of
Regents of University of Wisconsin System.
17
Goal Improve Higher Education Outcomes for
Students from Foster Care
Inputs What we invest
Outputs What we do Who we reach
Outcomes Impact Short-term Medium-term
Long-term learning
action conditions
Six Core Elements Program Development
Six Baseline Services
1 Designated Leadership
Learning Outcomes
1 Year-Round Housing Basic Needs
Transfer Out
Students Served
Degree Certification Attainment
Persistence begin /end term cont to next
Aspirations
2 Internal External Champions
2 Full-Financial Aid
students receiving financial aid
Awareness
Knowledge
3 Community Partners
3 Academic Advising, Career Counseling, Supp.
Support
Credits Enrolled Completed / Term
Skills
enrolled EOP/EOPS
4 Personal Guidance Counseling, Supp. Support
Course Enrollment / Completion Math
Employment
4 Data-Driven Decisions
5 Staff Peer Support Prof Dev.
5 Ops for Student Engagement Leadership
Course Enrollment / Completion English
Life Success
6 Sustainability Planning
6 Planned Transition to betw Colleges to
Employment
GPA

Evaluation
What do you want to know?
How will you know it?
18
You must be kiddingNow were really overwhelmed

• Getting started
• Determine purpose of logic model who will use
  it, for what?
• Involve others (group process)
• Set boundaries
• Multiple models may be needed to depict complex
  initiatives

19
Youth Financial LiteracyNeed / problem Youth
lack basic skills in management of personal
finances. Many are unable to budget and have
little knowledge of basic principles of earning,
spending, saving, and investing. Many fail in
managing their first consumer credit experience
and establish poor financial management habits
that follow them through and after college.
Outcomes short med
long learning action
conditions
Inputs What we invest
Outputs What we do Who
we reach
Staff
Develop test curriculum
Student increase confidence in ability to live
within budget
Community Partners
Dev. participation incentive
budget track spending
Students
Time
Recruit students
Sound financial habits
Budget
Teach 6 session curriculum Develop
budget Monitoring spending Checking savings
accts Accessing credit wisely Minimize manage
debt Plan for the future
Student increase skills Develop budget Monitor
spending Use chck saving acct Access credit
wisely Reduce / manage debt Plan for future
Pay bills on time
Technology
Adhere to debt mgmt plan
Space
Research base
Save money monthly
Evaluation
20
Evaluation Plan Worksheet

• FOCUS What will we evaluate?

Taylor-Powell, E. Henert, E. (2008). Developing
a logic model Teaching and training guide.
University of Wisconsin-Extension. Board of
Regents of University of Wisconsin System.
21
Evaluation Plan Worksheet

• FOCUS What will we evaluate? Short-term outcomes
  of financial literacy program

Taylor-Powell, E. Henert, E. (2008). Developing
a logic model Teaching and training guide.
University of Wisconsin-Extension. Board of
Regents of University of Wisconsin System.
22
Assessment is not an activity.
Assessment is a state of mind, a way of thinking.