Some Practical Tips for Measuring Financial Success Dr. Angela Lyons University of Illinois

1
Some Practical Tips for Measuring Financial
Success Dr. Angela LyonsUniversity of Illinois
Program Evaluation II Creating Your Evaluation
Toolkit Presented by Dr. Angela Lyons University
of Illinois October 2009
2
Identifying the ideal approach to program
evaluation

• Evaluation methods and measures vary widely
  across programs and academic disciplines.
• Wide variation in financial outcomes across
  programs.
• Significant differences in financial needs across
  consumers.
• Some participants unable to implement certain
  financial behaviors.

3
Then you need to decide.
To evaluate or not to evaluate, that is the
question?

• To evaluate or not to evaluate,
• that is the question?

4
Questions to ask yourself .

• At the end of the day, what are you trying to
  show?
• What is the purpose of the evaluation?
• Who will use the information and how?
• What information do you want to collect?
• Who is your target audience?
• What is your primary delivery method?
• What are your available resources (i.e., time,
  money, and staff)?
• What is your timeline?
• What is your expertise and evaluation capacity?
• Who are your partners, funders, and stakeholders?

5
Common survey methods used to collect impact data

• Post evaluation only
• Retrospective pre-test (RPT)
• Pre and post evaluation
• Follow-up
• Stages to Change (TTM)
• Control groups and longitudinal studies
• Key question to ask
• What is the length of your program?

6
Post evaluation only

• When to use Short programs that are less than 2
  hours
• Advantages
• Only need to survey group once.
• Good for limited-resource audiences and groups
  that are transient.
• Relatively inexpensive and less time intensive.
• Can document participants levels of knowledge,
  skills, and planned behaviors at the end of the
  program.
• Disadvantages
• With no pre assessment, its difficult to
  document potential and actual changes in
  knowledge, attitudes, and behavior.
• Retrospective pre-tests (RPTs) The Post-Then-Pre
  Evaluation

7
Retrospective pre-test (RPTs)

• When to use Any program, but typically 2 hours
  or less
• Advantages
• Only need to survey group once.
• Good for limited-resource audiences and groups
  that are transient.
• Controls for response shift bias.
• Can document relative change.
• Disadvantages
• Potential for respondent bias (social
  desirability factor).
• Self-assessment measures are subjective.

8
Pre and post evaluations

• When to use Programs that are 2 hours or longer
• Advantages
• Can compare pre and post responses and document
  changes in knowledge, attitudes, and behavior.
• Can be used to document immediate changes in
  knowledge, skills and planned behaviors following
  the program.
• Disadvantages
• More time intensive.
• Identification numbers are needed to match pre
  and post surveys.
• May be difficult to show actual behavior change.
• May be difficult to show that the intervention
  caused the change.
• Doesnt account for other possible reasons for
  change.
• Transient populations may lead to low unmatched
  evaluations.

9
Follow-ups

• When to use
• Program is comprehensive enough to potentially
  result in intermediate and long-term impact.
• Must have adequate resources and evaluation
  capacity.
• Usually administered three to six months after
  the program.
• Can document changes in actual financial
  behaviors, ability to achieve financial goals,
  and overall financial position.

10
Stages to Change (TTM)

• When to use Programs that have multiple
  sessions
• Advantages
• Can document intermediate and long-term change.
• Easier to measure actual behavior change and to
  control for other factors that may lead to change
  over time.
• Can identify stage at which individual is ready
  and able to change behavior.
• Behaviors can be recorded at the beginning,
  middle, and end of the program so that changes in
  actual behavior can be observed.
• Disadvantages
• Time and resource intensive.
• May require additional progress reporting and
  long-term follow-up.
• Can only be used with multi-session programs.

11
Train-the-trainer evaluations

• Similar to pre and post evaluation, but more
  content specific.
• Covers subject material in more detail to ensure
  that trainers have an adequate level of knowledge
  to teach the program to others.
• Can be used to document changes in both the
  instructors teaching skills and personal
  financial behaviors.
• Follow-ups can document how the curriculum
  materials are being used and identify additional
  programming needs.

12
Designing the evaluation instrumentKey survey
content

• General reactions to the session
• Changes in knowledge
• Changes in motivation, confidence, and abilities
• Intended changes in behavior
• Actual changes in behavior
• Future programming needs and preferences
• Demographics
• Qualitative / open-ended responses

13
General reactions to the session

• Please rate the instructor(s), materials, and the
  overall program
• by checking the box that best applies.

Not Helpful Somewhat Helpful Helpful Very Helpful
Instructor(s) ? ? ? ?
Educational materials ? ? ? ?
Overall program ? ? ? ?
14
Measuring changes in knowledge

• Testing Knowledge
• Please circle your answer to each of the
  following statements.

1. Fixed expenses are expenses that typically change from month to month such as food, clothing, and utilities. True False Not Sure
2. Financial experts recommend having an emergency fund that is equal to 3-6 months worth of living expenses. True False Not Sure
3. Goals should only be made for long-term plans such as homeownership, college tuition, or retirement. True False Not Sure
4. Gross income is defined as income after taxes and other withholdings have been subtracted from net income. True False Not Sure
5. Credit reports can affect an individuals ability to get a job, purchase a home, and obtain home and auto insurance. True False Not Sure
15
Measuring changes in knowledge (conti.)

• Format can be True/False or multiple choice.
• True/False is reliable indicator for low literacy
  audiences and youth.
• The more questions you ask, the greater the
  reliability measure.
• May include a dont know option to control for
  guessing.
• Post-test 10 questions (established standard)
• Pre- and post-test 10-20 questions
• Train-the-trainer 10-25 questions

16
Changes in motivation, confidence, and abilities

• Building Skills/Confidence Indicators
• Please check the box that best describes your
  confidence to do
• the following

Your confidence to Not Confident A little confident Somewhat confident Confident Very confident
Set short and long-term financial goals. ? ? ? ? ?
Save money regularly. ? ? ? ? ?
Use a spending plan to track income and expenses. ? ? ? ? ?
Pay bills on time each month. ? ? ? ? ?
17
Changes in motivation, confidence, and abilities
(conti.)

• Recording Participants Attitudes
• Please check the box that best describes how much
  you agree
• with the following statements.

Statement Strongly Disagree Disagree Undecided Agree Strongly Agree
Saving money regularly is important to me. ? ? ? ? ?
Keeping track of spending is a good habit. ? ? ? ? ?
Planning my personal budget is a priority. ? ? ? ? ?
18
Intended changes in behavior

• Please indicate how often you plan to do each of
  the following
• financial practices. There is no right or
  wrong answer. (Choose only one)

Financial Practice Never Rarely Sometimes Usually Always
Set short and long-term financial goals. ? ? ? ? ?
Save money regularly. ? ? ? ? ?
Use a spending plan to track income and expenses. ? ? ? ? ?
Pay bills on time each month. ? ? ? ? ?
As a result of this program, you plan to No Maybe Yes Already doing this Does not apply
Set short and long-term financial goals. ? ? ? ? ?
Use a spending plan to track income and expenses. ? ? ? ? ?
Pay bills on time each month. ? ? ? ? ?
19
Actual changes in behavior

• Please indicate how often you are currently doing
  each of the following
• financial practices. There is no right or
  wrong answer. (Choose only one)

Financial Practice Never Rarely Sometimes Usually Always
Setting short and long-term financial goals. ? ? ? ? ?
Saving money regularly. ? ? ? ? ?
Using a spending plan to track income and expenses. ? ? ? ? ?
Paying bills on time each month. ? ? ? ? ?
20
Capturing behavior change in follow-ups and
stages
Financial Progress Indicators Please check the
box that best describes how your financial
position has changed since completing the
program. Then, indicate specifically how your
financial position has changed.
Decreased Stayed about the same Increased Amount of change, if any ( or )
Monthly income ? ? ?
Monthly expenses ? ? ?
Amount saved monthly ? ? ?
Current credit card debt ? ? ?
21
Capturing behavior change in follow-ups and
stages

• Progress Reporting
• Please record your financial position based on
  your current progress in
• the program.

Financial Position At the beginning of the program In the middle of the program At the end of the program
How much credit card debt do you owe? ()
How many credit cards do you have? ()
What is the highest interest rate on your credit cards? ()
How much do you have in savings? ()
22
A few words about train-the-trainer programs.

• Testing knowledge
• Building teaching skills
• Shaping personal skills
• Taking action for teaching
• Taking action for personal financial success
• Follow-ups

23
Qualitative / Open-Ended Questions (common
examples)

• Post Evaluation Only and Pre and Post
  Evaluation
• What did you like the most about this program?
• What did you like the least about this program?
• How could this program be improved?
• Would you recommend this program to others?
• Stages to Change Evaluation
• What has made it easier for you to improve your
  financial practices?
• What has prevented you from improving your
  financial practices?
• With respect to the overall program, what did you
  like the most?
• What did you like the least?
• How could this program be improved?
• Have you shared what you learned with others?
• Would you recommend this program to others?

24
Qualitative / Open-Ended Questions (conti.)

• Train-the-Trainer Evaluation
• What was the most helpful information you
  received during this training program?
• How could this training program be improved?
• How do you plan to share this information with
  your target audience(s)?
• What information and materials from this training
  do you plan to share with your target
  audience(s)?
• Will you share what you learned with other
  instructors and colleagues?
• Would you recommend this training program to
  other instructors and colleagues?

25
Demographic Questions

• Age
• Gender
• Race, Ethnicity, and Language
• Marital Status
• Education
• Employment
• Family Structure
• Health Status
• Income, Assets, and Debts
• Region/Location

• Financial Experience
• Students/Youth
• Instructors/Educators

26
Rigor vs. Reality

• Longitudinal data?
• Control groups?
• Randomized experiments?

27
NEFE Financial Education Evaluation Toolkit
http//www2.nefe.org/eval/intro.html
28
NEFE Financial Education Evaluation Toolkit

• Database
• Post evaluation only with option for follow-up
• Pre and post evaluation with option for follow-up
• Stages to Change Evaluation
• Train-the-Trainer
• Testing Knowledge
• Building Skills
• Taking Charge
• Manual
• How-to-guide for grass-roots level organizations
• Examples (survey instruments, executive summary,
  reports)
• Guidance on how to organize and present impact
  data

29
Manual http//www2.nefe.org/eval/manual.html
30
Manual http//www2.nefe.org/eval/manual.html Par
t I Financial Education Overview Part II
Understanding Program Evaluation Part III The
Evaluation Planning Process Part IV Using the
Evaluation Database Part V Reporting Program
Impact
31
Part I Financial Education Overview
32
Part II Understanding Program Evaluation
33
(No Transcript)
34
(No Transcript)
35
Part III The Evaluation Planning Process
36
(No Transcript)
37
Part IV Using the Evaluation Database
38
Part V Reporting Program Impact
39
(No Transcript)
40
(No Transcript)
41
Appendix Sample Evaluation Instruments
42
Database http//www2.nefe.org/eval/index.php
43
Step 1 Program Info and Follow-up
44
Step 2a Knowledge Indicators
45
Step 2b Customizing Questions
46
Step 3 Confidence and Behavior Indicators
47
Step 4a Selecting Statements
48
Step 4b Customizing Statements
49
Step 5 Qualitative Info
50
Step 6 Demographics
51
(No Transcript)
52
Step 7 Follow-Up Financial Progress Indicators
53
Step 8 Follow-Up Personal Achievements
54
Step 9 Follow-Up Demographics
55
Step 10 Finalizing Evaluation
56
ACTIVITY Putting It All Together (Evaluation
Action Plan Part B)
57
Think about your signature program, what is the
most appropriate evaluation method?

• Post-test only
• Retrospective pre-test
• Pre and post-test
• Follow-up survey
• Stages-to-change
• Focus groups

• Interviews
• Case studies
• Observations
• Stories/anecdotal evidence
• Tests of ability
• Other

58

• What types of questions will
• your evaluation seek to answer?

I would really like to know.
1.
2.
3.
59
What types of indicators will you use to document
this impact?

• Changes in satisfaction levels
• Changes in knowledge
• Changes in skills and confidence levels
• Changes in attitudes
• Changes in aspirations
• Anticipated or intended changes in behavior
• Actual changes in behavior
• Socio-economic changes
• Other

60
List some specific indicators.
1.
2.
3.
4.
61
U of I Center for Economic and Financial
Education http//www.cefe.illinois.edu/
62
Educational Tools for Evaluation http//www.cefe.
illinois.edu/tools/evaluation.html
63
Checklist of evaluation resources

• Program Planning Guide
• Evaluation Action Plan
• Evaluation Road Map
• Evaluation Reading List
• Sample Evaluations
• NEFE Financial Education Evaluation Toolkit
• http//www.cefe.illinois.edu/tools/evaluation.html
  

64
University of Wisconsin-Extension http//www.uwex
.edu/ces/pdande/evaluation/index.html
65
Cornell University Extension http//staff.cce.cor
nell.edu/administration/program/evaluation/evalref
s.htm
66
Penn State Extension http//www.extension.psu.edu
/evaluation/
67
Contact Information

• Dr. Angela Lyons
• Associate Professor
• Director, U of I Center for Economic and
  Financial Education
• University of Illinois
• Phone [phone]
• E-mail anglyons_at_illinois.edu

68
Questions