Measuring Program Outcomes

1
Measuring Program Outcomes

• Or . . .

2

• How to torture under-paid,
• over-worked, under-appreciated,
• over-utilized, under-funded, and chronically
  over-wrought non-profit professionals.

3
UWA Survey, 2000

• 86 of respondents initiated outcome measurement
  because United Way required it
• 61 at the direction of agencys professional
  leadership
• 25 because of government funders

4
UWA Survey, 2000

• Implementing program outcomes has helped us to
• Communicate program results to stakeholders 88
• Focus staff effort on common goals 88
• Identify effective practices of the program 86

5
UWA Survey, 2000

• Successfully compete for resources/funding 83
• Improve the service delivery of the program 76
• Share effective practices with programs in other
  agencies 72

6
UWA Survey, 2000

• Assess staff performance 64
• Identify staff training needs 64
• Increase program participants investment in
  achieving positive outcomes 55

7

• On balance, implementing program outcome
  measurement has had a positive impact on our
  programs ability to serve clients effectively.
• 74

8

• Program outcome information should be used in
  making decisions about program funding.
• 74

9
What, exactly, is an outcome?

• Benefit to a participant of a program
• May be during or after the program
• May be initial, intermediate, or long term

10
An outcome is a change or improvement in
participants

• Knowledge
• Skills
• Attitudes
• Values

• Behaviors
• Condition or
• Status

11
For instance

• A teen mother participates in parenting education
  program which makes her more knowledgeable of how
  to care for her child (initial) which leads her
  to provide proper care to her child
  (intermediate) which leads to her baby achieving
  appropriate 12-month milestones for physical,
  motor, verbal and social development (long-term).

12

• Outcomes answer the so what? question.

13
Teen Parenting Program

• Teen mother participates in parenting education
  program.
• So what?

14
Teen Parenting Program

• So, she acquires knowledge of proper care,
  feeding and social interaction with babies.
• So what?

15
Teen Parenting Program

• So, she provides proper care to her child.
• So what?

16
Teen Parenting Program

• So, baby achieves appropriate 12 month milestones
  for physical, motor, verbal, and social
  development.

17
Outcomes are all about

• Inputs
• Activities
• Outputs
• Outcomes

18
The Program Outcome Model

• Inputs
• Resources dedicated to or consumed by the program
• (money, staff staff time, supplies)

19

• Activities
• What the program does with the inputs to fulfill
  its mission
• (feed the homeless, provide job training, counsel
  pregnant teens)

20

• Outputs
• The direct products of program activities
• (Number of classes taught, number of hours of
  service delivered, number of participants served)

21
Outcomes Benefits for participants during and
after program activities

• New knowledge
• Increased skills
• Changed attitudes or values

• Modified behavior
• Improved condition
• Altered status

22
Inputs, activities, outputs, or outcomes?

• Agency hires ½ time staff person to implement
  parenting program for pregnant high school
  students.
• So what?

23
Inputs, activities, outputs, or outcomes?

• Pregnant students attend program 3 times per week
  for 12 weeks.
• So what?

24
Inputs, activities, outputs, or outcomes?

• 37 students complete 12 weeks of parenting
  program.
• So what?

25
Inputs, activities, outputs, or outcomes?

• Teens gain knowledge.
• Teens deliver healthy babies
• Teens know how to care for their babies.
• Babies achieve appropriate 12-month milestones
• So what?

26
TA DA!!!!!!

• The world is saved all because YOU cared enough
  to measure and evaluate the success of your
  program!!

27

• 8 steps to outcome measurement

28
1. Get ready.

• Assemble a work group
• Decide which program to measure
• Develop a timeline
• Share your game plan with key players

29
2. Choose the outcomes you want to measure.

• Gather ideas for program outcomes
• Construct a logic model for your program
• Select the outcomes that are important to measure
• Get feedback

30
Specify indicators for your outcomes

• Specify one or more indicators for each outcome
• Decide what factors could influence participant
  outcomes
• Use indicators you can influence

31
Uh, whats an indicator?

• The specific items of information that track a
  programs success
• For instance, a program that seeks to result in
  healthy babies could define healthy baby as one
  that scored 7 or above on the Apgar scale
• Number or of babies in program who achieve the
  7 score is an indicator of how well the program
  is doing with respect to that outcome

32
4. Prepare to collect data on your indicators

• Identify data sources for your indicators
• Design data collection methods
• Pretest your data collection instruments and
  procedures

33
5. Try out your outcome measurement system

• Develop a trial strategy
• Prepare the data collectors
• Track and collect outcome data
• Monitor the process

34
6. Analyze report your findings.

• Enter the data and check for errors
• Tabulate the data
• Analyze the data
• Provide explanatory information related to your
  findings
• Present your data in clear understandable terms

35

• Do your findings seem reasonable?
• Are they presented clearly?
• What questions do they raise that are not
  answered in the report?
• Can you explain?
• What other tables or charts would be helpful?

36
7. Improve your outcome measurement system

• Review your trial-run experience
• Make necessary adjustments
• Start full-scale implementation
• Monitor and review your system periodically

37
8. Use your findings internally

• Provide direction for staff
• Identify training technical assistance needs
• Identify program improvement needs and strategies
• Support annual long range planning
• Guide budgets justify resource allocation
• Suggest outcome targets
• Focus attention on policy programmatic issues

38
8. Use your findings externally

• Recruit talented staff volunteers
• Promote your program to participants referral
  sources
• Identify partners for collaborations
• Enhance your programs public image
• Retain increase funding!

39

• My work is done.
• When your work is done, you will have
• Better programs
• More successful participants
• More money, and
• The undying love and devotion of a grateful
  nation.

40

• All because you asked the burning question
• So what?