High Quality Performance Measures

1
High Quality Performance Measures

• What is Performance Measurement?
• What makes a high quality performance measure?

2
Learning Objectives

• By the end of this module, you will be able to
• Describe basic performance measurement concepts
• Describe how to strengthen performance measures
  through
• Alignment with theory of change
• Alignment of outputs and outcomes
• Selection of a meaningful outcome

3
Module Overview

• Performance Measurement
• Concept and definitions
• Comparison to impact evaluation
• Strengthening Performance Measures through
• Alignment with the Theory of Change
• Alignment of outputs and outcomes
• Selection of a meaningful outcome

4
What is Performance Measurement?

• Systematic process of regularly measuring
• Outputs
• Amount of service provided (e.g. people served,
  products created, or programs developed)
• Outcomes
• Reflect the changes or benefits that occur as the
  result of the service
• Can reflect changes in individuals,
  organizations, communities, or the environment
• Address changes in attitudes/beliefs,
  knowledge/skills, behavior, or conditions

5
Performance Measurement and Impact Evaluation

• Similarities
• Systematic data collection
• Measurement of progress
• Collect data on
• What took place (intervention)
• Outputs (amount of service)
• Outcomes (changes)

6
Performance Measurement and Impact Evaluation
Differences Key Areas Performance Measurement Impact Evaluation
Causality Assumes causality does not prove theory of change Seeks to show causality and prove theory of change
Implementation Tracks outputs and outcomes on a regular, ongoing basis May occur from time to time, but not on a regular, ongoing basis
Time Focus Shorter term focus what can be measured within one year Longer term focus
7
Performance Measurement and Impact Evaluation
(continued)
Differences Key Areas Performance Measurement Impact Evaluation
Design Balances rigor with practicality measurement implemented by program staff with limited resources and expertise Uses most rigorous evaluation design and methods that are right for program (e.g. quasi-experimental design)
Capable of producing valid (accurate) results.
8
Learning Exercise
Check the best answer Performance Measurement Impact Evaluation Both
1. Seeks to provide evidence of the existence of a causal relationship between the intervention and the observed outcomes, that is, it attempts to prove the theory of change
2. The information collected can be used for decision-making on programmatic improvement or expansion
3. Usually planned and implemented by program staff
4. Careful and systematic and quality data collection is very important
9
Learning Exercise
Check the best answer Performance Measurement Impact Evaluation Both
1. Seeks to provide evidence of the existence of a causal relationship between the intervention and the observed outcomes, that is, it attempts to prove the theory of change X
2. The information collected can be used for decision-making on programmatic improvement or expansion X
3. Usually planned and implemented by program staff X
4. Careful and systematic and quality data collection is very important X
10
Ensuring Quality

• Performance Measure
• Intervention
• Output
• Outcome

Review for meaningful outcome
11
Performance Measurement and Alignment

• Alignment in Performance Measurement Logical
  flow and connection between the different
  elements
• Alignment within the theory of change
• Alignment between output and outcome

12
Alignment within Theory of Change
A theory of change defines a cause-and-effect
relationship between a specific intervention and
an intended outcome.
13
Mentoring Example Alignment within Theory of
Change

• Which outcome is aligned with the intervention?

• NEED Documented high rates of chronic absence
  from school lead to other problems, e.g. low
  school engagement, poor academic performance
  (Social Work Practices, 2010).
• INTERVENTION Weekly individual and group
  mentoring to 6th grade students with attendance
  problems over 10 months. Focus on promoting
  re-engagement with school through social
  development and leadership skills.
• OUTCOME ________

• Students participate in the mentoring program
• Students increase academic performance
• Students improve school attendance

Descriptions in examples are abbreviated.
14
Mentoring Example Alignment within Theory of
Change

• Which outcome is aligned with the intervention?

• NEED Documented high rates of chronic absence
  from school lead to other problems, e.g. low
  school engagement, poor academic performance
  (Social Work Practices, 2010).
• INTERVENTION Weekly individual and group
  mentoring to 6th grade students with attendance
  problems over 10 months. Focus on promoting
  re-engagement with school through social
  development and leadership skills.
• OUTCOME ________

• Students participate in the mentoring program
• Students increase academic performance
• Students improve school attendance

15
Mentoring Example Alignment within Theory of
Change

• NEED Documented high rates of chronic absence
  from school lead to other problems, e.g. low
  school engagement, poor academic performance
  (Social Work Practices, 2010).
• INTERVENTION Weekly individual and group
  mentoring to 6th grade students with attendance
  problems over 10 months. Focus on promoting
  re-engagement with school through social
  development and leadership skills.
• OUTCOME Students improve school attendance.

16
Learning Exercise

• NEED 35 of young veterans (18-24 year olds) are
  unemployed (Department of Veteran Affairs, 2011).
  Economists cite a lack of marketable civilian
  skills and the need for education degrees,
  vocational certifications and experience in
  professional settings.

• Intervention
• Veterans participate in a drop-in job clinic
  offering job fairs, resume writing classes and
  skills in internet job searches.
• Veterans supported in completing training
  programs through assistance in finding
  appropriate programs and financial aid, with
  tutoring resources, in internship placement.
• Military families receive job placement
  counseling.

• Outcome
• Veterans report increased confidence in using the
  internet for a job search.
• Veteran family member report completion of skill
  classes.
• Veterans find or are placed in jobs.

17
Learning Exercise (continued)Select the outcome.

• NEED 35 of young veterans (18-24 year olds) are
  unemployed (Department of Veteran Affairs, 2011).
  Economists cite a lack of marketable civilian
  skills and the need for education degrees,
  vocational certifications and experience in
  professional settings.

• Intervention
• Veterans participate in a drop-in job clinic
  offering job fairs, resume writing classes and
  skills in internet job searches.
• Veterans supported in completing training
  programs through assistance in finding
  appropriate programs and financial aid, with
  tutoring resources, in internship placement.
• Military families receive job placement
  counseling.

• Outcome
• Veterans report increased confidence in using the
  internet for a job search.
• Veteran family member report completion of job
  skills classes.
• Veterans find or are placed in jobs.

18
Learning Exercise (continued)Select the
intervention.

• NEED 35 of young veterans (18-24 year olds) are
  unemployed (Department of Veteran Affairs, 2011).
  Economists cite a lack of marketable civilian
  skills and the need for education degrees,
  vocational certifications and experience in
  professional settings.

• Intervention
• Veterans participate in a drop-in job clinic
  offering job fairs, resume writing classes and
  skills in internet job searches.
• Veterans supported in completing training
  programs through assistance in finding
  appropriate programs and financial aid, with
  tutoring resources, in internship placement.
• Military families receive job placement
  counseling.

• Outcome
• Veterans find or are placed in jobs.

19
Learning Exercise (continued)Summary Alignment
within Theory of Change

• NEED 35 of young veterans (18-24 year olds) are
  unemployed (Department of Veteran Affairs, 2011).
  Economists cite a lack of marketable civilian
  skills and the need for education degrees,
  vocational certifications and experience in
  professional settings.

• Intervention
• Veterans supported in completing training
  programs through assistance in finding
  appropriate programs and financial aid, with
  tutoring resources, in internship placement.

• Outcome
• Veterans find or are placed in jobs.

20
Alignment of Outputs and Outcomes

• Output results from the intervention
• Outcome is likely to result from output
• Output and outcome measure the same intervention
  and beneficiaries

21
Housing Example Output-Outcome Alignment

• INTERVENTION Construction teams assess housing
  request, plan and execute housing upgrades and
  repairs and complete inspection requirements for
  individuals with disabilities.
• OUTPUT Individuals with disabilities receive
  housing service (upgrades and repairs).
• OUTCOME Construction team members improve
  construction skills.

• Do the intervention and output align?
• Yes
• No
• Why or why not?

22
Housing Example Output-Outcome Alignment

• INTERVENTION Construction teams assess housing
  request, plan and execute housing upgrades and
  repairs and complete inspection requirements for
  individuals with disabilities.
• OUTPUT Individuals with disabilities receive
  housing service (upgrades and repairs).
• OUTCOME Construction team members improve
  construction skills.

• Do the intervention and output align?
• Yes
• No
• Why or why not?
• Logical to assume if well-designed and implemented

23
Housing Example Output-Outcome Alignment

• INTERVENTION Construction teams assess housing
  request, plan and execute housing upgrades and
  repairs, and complete inspection requirements for
  individuals with disabilities.
• OUTPUT Individuals with disabilities receive
  housing services (upgrades and repairs).
• OUTCOME Construction team members improve
  construction skills.

• Do the output and outcome align?
• Yes
• No
• Why or why not?

24
Housing Example Output-Outcome Alignment

• INTERVENTION Construction teams assess housing
  request, plan and execute housing upgrades and
  repairs and complete inspection requirements for
  individuals with disabilities.
• OUTPUT Individuals with disabilities receive
  housing service (upgrades and repairs).
• OUTCOME Construction team members improve
  construction skills.

• Do the output and outcome align?
• Yes
• No
• Why or why not?
• 2 different groups of beneficiaries

25
Learning ExerciseOutput-Outcome Alignment

• INTERVENTION National Service participants
  provide one-on-one tutoring to preschool children
  in math readiness activities, 5 times a week over
  9 months.

• Output
• Children complete participation in the early
  childhood program.
• Teachers complete training with expert math
  teachers.
• Children complete participation in preschool art
  group.

• Outcome
• Children demonstrate gains in school readiness in
  terms of literacy skills.
• Children demonstrate gains in school readiness in
  terms of numeracy (math) skills
• Teachers implement new math curriculum in the
  classroom.

26
Learning ExerciseOutput-Outcome Alignment

• INTERVENTION National Service participants
  provide one-on-one tutoring to preschool children
  in math readiness activities, 5 times a week over
  9 months.

• Output
• Children complete participation in the early
  childhood program
• Teachers complete training with expert math
  teachers.
• Children complete participation in preschool art
  group.

• Outcome
• Children demonstrate gains in school readiness in
  terms of literacy skills.
• Children demonstrate gains in school readiness in
  terms of numeracy (math) skills
• Teachers implement new math curriculum in the
  classroom.

27
Learning ExerciseOutput-Outcome Alignment

• INTERVENTION National Service participants
  provide one-on-one tutoring to preschool children
  in math readiness activities, 5 times a week over
  9 months.

• Output
• Children complete participation in the early
  childhood program .

• Outcome
• Children demonstrate gains in school readiness in
  terms of literacy skills.
• Children demonstrate gains in school readiness in
  terms of numeracy (math) skills
• Teachers implement new math curriculum in the
  classroom.

28
Examples of Aligned Output-Outcome National
Performance Measures
Focus Area Output Outcome
Education Number of children that completed participation in CNCS-supported early childhood education programs. (ED21) Number of children demonstrating gains in school readiness in terms of social and/or emotional development.(ED23)
Healthy Futures Number of individuals receiving support, services, education and/or referrals to alleviate long-term hunger (H11) Number of individuals that reported increased food security of themselves and their children (household food security) as a result of CNCS-supported services (H12)
National performance measures at the Resource
Center www.nationalserviceresources.org/npm/home
29
Identifying a Meaningful Outcome

• The Outcome should
• Be central to your theory of change
• Match the type of change (attitude, knowledge,
  behavior, or condition) you want to measure
• Be ambitious but manageable

30
Central to the Theory of Change

• How is the proposed outcome supported by the
  theory of change?
• Evidence for intervention should support the
  outcome
• Evidence supports cause-and-effect relationship
  between intervention and outcome

31
Central to the Theory of Change

• How is the proposed outcome central to the
  theory of change?
• Addresses the need/problem
• Likely change given the intervention
• A key, not preliminary or peripheral, outcome

32
Mentoring Example Central to Theory of Change

• Which outcome is central?

• NEED Documented high rates of chronic absence
  from school lead to other problems, e.g. low
  school engagement, poor academic performance
  (Social Work Practices, 2010).
• INTERVENTION Weekly individual and group
  mentoring to 6th grade students with attendance
  problems over 10 months. Focus on promoting
  re-engagement with school through social
  development and leadership skills.

• Families are satisfied with the mentoring program
• Students improve engagement with peers
• Students improve school attendance

33
Type of outcome Accurate and Clear

• Is the outcome getting at what you wish to
  measure?
• Articulates the question you really want to
  answer
• Has the outcome type been identified attitude,
  knowledge/skills, behavior or condition?
• Outcome type should be same as theory of change
• Outcome type will inform instrument

34
Veteran Example Type of Outcome

• Which type of outcome?

• NEED 35 of young veterans (18-24 year olds) are
  unemployed (Department of Veteran Affairs, 2011).
  Economists cite a lack of marketable civilian
  skills and the need for education degrees,
  vocational certifications
• INTERVENTION National service participants
  support veterans in completing training programs
  by assisting in locating appropriate programs,
  securing financial aid, and by providing tutoring
  resources and internship placements.

1. Veterans report increased confidence about
   finding employment. (attitude)
2. Veterans demonstrate new technical skills.
   (knowledge/skills)
3. Veterans are placed in jobs. (condition)

35
Ambitious and Manageable

• Is the outcome meaningful AND manageable?
  Consider
• Staff time and expertise
• Reporting constraints

36
Ambitious and Manageable

• Is the outcome too modest? Consider
• Outcome should reflect the main change expected
  from the intervention
• Is the outcome too ambitious? Consider
• Program timeframe
• Scope of the intervention
• Severity of the problems being addressed

37
Mentoring Example Ambitious and Manageable

• Which outcome is meaningful and manageable?

• NEED Documented high rates of chronic absence
  from school
• INTERVENTION National Service participants
  provide individual and group mentoring to 6th
  graders with documented attendance problems

• Student increase outdoor teamwork skills
• Students increase graduation rates from high
  school
• Students improve school attendance

38
Mentoring Example Ambitious and Manageable

• Which outcome is meaningful and manageable?

• NEED Documented high rates of chronic absence
  from school
• INTERVENTION National Service participants
  provide individual and group mentoring to 6th
  graders with documented attendance problems

• Student increase outdoor teamwork skills
• Students increase graduation rates from high
  school
• Students improve school attendance

39

• Learning Exercise

Intervention Year long mentoring project with
4th graders literacy focus.

• Outcomes

• Students know location of the library
• Student improve reading ability
• Students graduate from high school
• Students increase school attendance

1. Which outcome is too ambitious? (select one)
2. Which outcome is too modest? (select one)
3. Which outcomes look ambitious but manageable?
   (select two)

40

• Learning Exercise

Intervention An afterschool program for children
ages 6-12 is expanded to two new sites.

• Outcomes

• Parents are interested in learning more about the
  program.
• There is reduced gang violence in the area.
• Donations for sustaining activities at the two
  sites are secured.
• Trained community volunteers are serving at the
  new sites.

1. Which outcome is too ambitious? (select one)
2. Which outcome is too modest? (select one)
3. Which outcomes look ambitious but manageable?
   (select two)

41

• Learning Exercise

Intervention 6-month nutrition and health
program for high school students.

• Outcomes

• Youth develop long-term healthy eating habits.
• Youth increase healthy food intake.
• Youth improve cooking skills.
• Youth are motivated to eat more vegetables.

1. Which outcome is too ambitious? (select one)
2. Which outcome is too modest? (select one)
3. Which outcomes look ambitious but manageable?
   (select two)

42

• Learning Exercise

Intervention Year-long riparian (river)
restoration project involving youth volunteers.

• Outcomes

1. Which outcome is too ambitious? (select one)
2. Which outcome is too modest? (select one)
3. Which outcomes look ambitious but manageable?
   (select two)

• Youth increase knowledge of native stream bank
  plant species.
• 250,000 residents in the county benefit from
  restored watershed.
• Youth increase interest in protecting the
  environment.
• Water quality is improved over 6 mile clean-up
  area.

43
Learning Exercise
INTERVENTION National Service participants
assess the housing up-grade requests from the
community members and address dilapidated, unsafe
or inaccessible housing. They plan and then
execute housing remodels, upgrades and repairs
for economically disadvantaged individuals and
individuals with disabilities.

• The most meaningful and manageable outcome for
  this intervention is
• Number of individuals satisfied with quality of
  the work.
• Number of individuals securing mortgage loan
  approval for house purchase within 5 years.
• Number of individuals transitioned into safe
  housing.

44
Summary of Key Points

• Performance measurement is a systematic process
  of measuring progress (outputs and outcomes)
• Performance measurement does not seek to prove
  a theory of change, but can provide snapshots.
• Impact evaluation can determine if results
  occurred because of the intervention.
• Strong performance measures align with the theory
  of change (need, intervention and outcome)
• The intervention is based on evidence that
  supports a cause-and-effect relationship between
  the intervention and an intended outcome.
• The theory of change helps identify an
  appropriate outcome to measure.

45
Summary of Key Points

• Strong performance measures are aligned
• Output results from the intervention
• Outcome is likely to result from output
• Output and outcome measure the same intervention
  and beneficiaries
• Strong outcomes should
• Be central to the theory of change
• Match the type of change you want to see
  (attitude, knowledge/skills, behavior, condition)
• Be ambitious but manageable

46
Resources

• CNCS Priorities and Performance Measures
  http//www.nationalservice.gov/resources/npm/home
  
• Program Specific Notices of Funding Opportunities
  and Application Instructions http//http//www.na
  tionalservice.gov/for_organizations/funding/nofa.a
  sp
• Link to Evaluation