Measuring Success Using Performance Measurement

1
Measuring Success Using Performance Measurement

• NAEH Conference on Ending Family Homelessness

Michelle Abbenante Brooke Spellman February 8,
2008
2
What is Performance Measurement?

• Performance measurement is a process that
  systematically evaluates whether your efforts are
  making an impact on the clients you are serving
  or the problem you are targeting.

3
Multiple Levels of Performance Measurement
State-wide Report or Performance Measurement Tool
3. State Level
NAEH Assessment of National Progress or Federal
GRPA and PART Reviews
4. National Level
4
Why Should Programs be Interested in Performance
Measurement?

• We are all in the business of helping people,
  which means we need to
• understand whether current activities are
  working to achieve intended results.
• drive program improvement and share information
  on effective practices with others.
• acknowledge that high-performing programs are
  more likely to receive funding through
  competitive funding processes.

5
Building Blocks of Performance Measurement

• Inputs include resources dedicated to, or
  consumed by, the programe.g., money, staff and
  staff time, volunteers and volunteer time,
  facilities, equipment and supplies.
• Activities are what the program does with the
  inputs to fulfill its mission, such as providing
  shelter, feeding the homeless, or providing job
  training.
• Outputs are the direct products of program
  activities. They usually are presented in terms
  of the volume of work accomplishede.g., number
  of participants served and the number of service
  engagements.
• Outcomes are benefits or changes among clients
  during or after participating in program
  activities. Outcomes may relate to change in
  client knowledge, attitudes, values, skills,
  behaviors, conditions, or other attributes.

6
Performance Measurement Process
Should we adjust how we spend our resources?
How do we document our efforts?
Should we add or change use of resources to
expand our impact?
What did our efforts achieve?
7
Outputs vs. Outcomes

• Whereas, an outcome is

• An output is

• Focused on what the program will do to achieve
  the outcome.

• Focused on what the participant will gain from
  the program.

• A way to measure the client-level impact with
  clear targets and methods for measuring change.

• A way to quantify the frequency and intensity of
  the activity.

• Specific to the activity described for the
  program.

• Attributable (a result of) to that program.

• Meaningful and attainable.

• Feasible and attainable.

If outcomes show the program works outputs are
needed to understand how to replicate results
8
Group ExerciseOutcome, Output, or Neither?
Answers
Examples

• 150 clients received prevention counseling and
  one-time financial assistance.
• 90 of persons will obtain employment by
  completion of program.
• 75 of program staff will be trained in crisis
  management techniques.
• Met 40 (50) of Permanent Supportive Housing
  goal.
• 65 of clients with chronic medical condition
  will improve physical health

Output
Outcome
Activity
Output
Outcome
9
Achieving Your Outcomes
Achieving your outcomes can be a progression.
Monitoring this progression requires data that
are collected at different intervals

• Short-term outcomes What change will the client
  experience within a month of his/her involvement
  in the program? How will you measure this?
• Intermediate outcomes What change will the
  client experience within a year of being involved
  in the program? How will you measure this?
• Long-term outcomes What is the long-term (e.g.,
  3-year) impact of the program on clients? Has it
  been sustained? How will it be measured?

10
Example Employment Programs Impact Over Time
39 of participants (75 of those who get a job)
will retain their jobs for gt 12 months.
All of those who get a job (52 clients) will
receive weekly check-up calls and job counseling,
as needed.
All of those who complete the training classes
(95 clients) will be referred to jobs and
receive job placement counseling.
52 of participants (55 of those who complete
the job training class) will obtain full-time
employment
93 of participants (97 of people who completed
the job training class) will show improved job
skills
95 participants will complete job training
classes.
100 people expected to participate in the
program annually
11
Framework for Converting Program Goals into
Outcomes

• How do I convert program goals into measurable
  outcomes?
• What do I need to calculate the outcomes?

12
Converting Program Goals into Outcomes Example

• Program Goal Supporting participants in stable
  housing at least 6 months

Step 3
Step 2
Step 1
20 people are still in stable housing (6mo) or
exited after being in housing for 6 mo
Base Persons who have been enrolled gt 6 mo or
have exited (n40 people)
Goal remain housed gt 6 mo
50 remain in stable housing at least 6 months
13
Look Out for Ambiguous Concepts
Developing and measuring performance outcomes
often invites ambiguous concepts into the process.
For example, what do we mean by

• obtaining stable housing?
• obtaining employment?
• increasing income?
• accessing services?
• becoming more self-sufficient?

Which data elements and responses will count?
14
HMIS Data Elements Are the Building Blocks of
Performance Measurement
Universal Data Elements Program-Specific Data Elements
Name Income Sources
Social Security Number Non-Cash Benefits
Date of Birth Physical Disability
Ethnicity Race Developmental Disability
Gender HIV/AIDS
Veteran Status Mental Health
Disabling Condition Substance Abuse
Residency Prior to Entry Domestic Violence
Zip Code of Last Permanent Add Services Received
Entry Date Destination
Exit Date Reasons for Leaving
Person, Program, HH ID
These data elements are collected at entry and exit. These data elements are collected at entry and exit.
15
Wherever Possible Use HMIS to Define Your Concepts
Based on the Destination HMIS data element, we
can define stable housing (narrowly) using the
following response categories

• Emergency shelter

• Apartment or house that you own

• Transitional housing

• Staying/living with family

• Permanent housing

• Staying/living with friends

• Substance abuse facility or detox center

• Hotel or motel voucher paid for without ES voucher

• Hospital (non-psychiatric)

• Foster care home or group home

• Jail, prison or juvenile detention center

• Place not meant for human habitation

• Room, apartment, or house that you rent

• Other

• Dont Know

• Refused

16
Group Exercise
Employment Program The goals of the program are
to help unemployed clients obtain employment and
help employed clients get better jobs. During
the past year, the program served 6
(unduplicated) persons
Client ID Entry Date Exit Date Employment Entry Employment Exit
1 1/31/07 9/15/07 Unemployed Employed
2 3/15/07 6/28/07 Unemployed Unemployed
3 7/11/07 -- Unemployed --
4 7/7/07 9/18/07 Employed Same Employment
5 8/2/06 5/12/07 Employed Higher Paying Job
6 11/7/06 8/2/07 Unemployed Employed
17
Define the Base Population for Each Goal
Is everyone part of the target population?E.g.,
Do you expect to calculate an outcome for
everyone?
Goal 1 Achieve employment at exit
Goal 2 Obtain better employment at exit
Client ID Entry Date Exit Date Employment Entry Employment Exit
1 1/31/07 9/15/07 Unemployed Employed
2 3/15/07 6/28/07 Unemployed Unemployed
3 7/11/07 -- Unemployed --
4 7/7/07 9/18/07 Employed Same Employment
5 8/2/06 5/12/07 Employed Higher Paying Job
6 11/7/06 8/2/07 Unemployed Employed
18
Calculate the Outcome for Goal 1

• Program Goal 1 Obtain Employment at Exit

Step 3
Step 2
Step 1
All unemployed persons at entry who exited (N 3)
Achieve employment
2 persons achieved employment
67 achieved employment
19
Calculate the Outcome for Goal 2

• Program Goal 2 Improved Employment at Exit

Step 3
Step 2
Step 1
Persons who were employed at entry and exited (N
2)
Improve employment
1 person increased earnings
50 gained better employment
20
Exercise 2 and the Performance Measurement Process
Activities
Job Training Classes Interview Assistance Job
Placement Services
Outputs
6 enrolled in weekly services 6 employment
assessments Referred to av. 4jobs each
21
Using Outcomes to Inform Future Program Operations
22
Step 1 Reviewing Outcomes with Managers
Whats Going On?

• Program director and managers should review
  outcomes collaboratively to understand what the
  outcomes are suggesting.
• Break down the outcomes to understand the
  underlying forces

• What are we doing right? What activities
  contributed to our ability to meet/exceed our
  benchmarks?
• Where do we need to improve? What activities fell
  short of producing the desired outcomes?
• What else might be contributing to our outcomes?
  How can we influence or mitigate these external
  forces to further our positive outcomes?

23
Step 2 Developing Action Steps and Timelines
Reinforcing the Good and Adjusting the Bad

• Outcomes that were achieved/exceeded Continue to
  support the activities that led to our positive
  performance.
• Outcomes that were not achieved Allocate our
  inputs differently to support different
  types/levels of activities.
• Set target dates for reviewing all outcomese.g.,
  3-month intervals.
• Collaborate with other service providers to
  control the external impacts on the program.

24
Step 3 Implementing the Action Steps
Getting Buy-In Through Information Sharing

• You cant implement what you dont understand
  program directors, managers and front-line staff
  must understand the reasons for making changes in
  program operations.
• Information sharing promotes the idea that we
  are all in this together.
• Information sharing is fluid program directors,
  managers and front-line staff can learn from one
  another its not a one-way (top-down) process.

25
Step 4 Regular Monitoring
Its Easier to Adjust Program Operations
Incrementally than Wholesale

• Monitor your progress by generating your
  performance outcomes at different periods of
  timee.g., 3-month intervals.
• Adjust your approach as needed, but usually
  incrementally.
• Important to acknowledge that clients needs may
  shift, and thus program goals and approach may
  also need to shift.

26
Comparing Program Results

• You can compare results from one program to
  another to see which programs are working best
  and which are working least well
• With limited dollars, you want to fund the
  programs that are most effective.
• You can use program results to identify best
  practice programs and those that need TA
• You can use results from multiple programs to
  help set a community expectation or standard of
  performance

27
Case Study How Washington, D.C. Uses Program
Results

• Outputs/Efficiency Measures
• Clients Served
• Chronically Homeless Served
• Occupancy the rate at which program was used
• Interim Outcome Measures
• Permanent Housing
• Positive client destinations at exit (TH
  programs)
• Retain clients for 6 months (PSH programs)
• Income the amount of income or sources obtained
• Self-Sufficiency change in substance use,
  education, mental illness or employment
• Measures apply differently to each program type
  and are supplemented with qualitative data for
  ranking purposes

28
Washington, DCFY 07 DHS Performance Measures
Self Sufficiency Temporary, Transitional and
Permanent Supportive Housing Programs required to
submit a Self Sufficiency Indicator will have to
choose from Substance Abuse, Education, Mental
Illness Assistance or Employment.
29
Apples to Apples Risk Adjustment

• Problem Comparing program results can encourage
  programs to cream to ensure strong results
• Solution Risk adjustment allows you to account
  for differences in client populations when
  comparing results across programs
• Results can be adjusted on the basis of
• Client characteristics, such as demographics,
  family size, disability
• Client history, such as past eviction, criminal
  background,
• Client functionality or level of
  engagement/commitment to change

30
Risk Adjustment Requires Expertise

• To adjust for client differences, programs need
  to collect consistent data on clients to use
  during analysis of program results
• Agree on these standards beforehand
• Develop an analysis plan for how you intend to
  adjust for client differences
• Engage a researcher to help develop the plan
• Even if you dont formally adjust results,
  acknowledge that different programs may have
  different outcome expectations based on
  differences in clients targeted and/or served

31
Simplified Illustration of Dissecting Client
Outcomes on Increased Earned Income
Program A
Program B
By establishing targets, programs can be compared
against CoC expectations in the future to
determine if program performance is higher or
lower than expected.
32
System Performance Measurement

• Are your actions achieving your intended goals at
  the system level?
• Does the system work?
• If yes, what makes it work?
• If no, what part doesnt work, and how do you fix
  it to make it work?
• Note that you may have system goals that only
  relate to certain types of clients or parts of
  the system (e.g., different goals for severely
  disabled persons)

33
Sample Impact Measures

• Incidence of homelessness - Is homelessness
  declining?
• Incidence of street or CH - Is street or chronic
  homelessness declining?
• Length of stay in system, across all homeless
  programs - Do people stay homeless for shorter
  periods of time?
• Prevention Are fewer people experiencing
  homelessness for the first-time?
• Rates of Recidivism Are repeat occurrences of
  homelessness avoided or declining?
• Cross-tabulate results by core characteristics to
  understand if/how results vary for different
  subpopulations

34
Steps to Calculate System Length of Stay
Client ID Prog ID
1 A
1 B
2 A
3 C
Entry Exit Date
Date 5/8/06 5/30/07 6/1/07
9/01/07 3/1/07 5/21/07 2/1/06
12/7/06
LOS 229281309
35
Some notes of caution

• There is more to performance measurement than
  conducting the analysis
• Educate, train, obtain buy-in
• Be careful about how you interpret and use the
  data
• Jump in, but dont be careless in how you use the
  results
• Look at the results within the context of all the
  outputs, interim measures and impact measures to
  validate the interpretation thats being made
• Vet the results before publicly releasing
  anything
• Appropriately caveat the limitations of the data
  and analysis

36
Summary of System Performance Measurement
37
Questions?

• Contact us for more information or assistance
• Michelle Abbenante, michelle_abbenante_at_abtassoc.co
  m
• Brooke Spellman, brooke_spellman_at_abtassoc.com