Targeting Criminogenic Need: Results from Meta-Analyses

1
Planning for Performance Measurement
Development of the OJJDP Performance Measurement
System Presenters Greg Thompson, OJJDP Dennis
Mondoro, OJJDP Heidi Hsia, OJJDP
2
Day 1 Objectives

• Enhance planning for performance measurement (PM)
  at the State level
• Review development of the Performance measurement
  system
• Review logic models and measures
• Incorporate PM into the RFP
• Understand reporting guidelines

3
Day 2 Objectives

• Overcoming challenges in implementing
  performance measurement and being able to train
  subgrantees to
• Identify data sources
• Share information
• Monitor and track effectively
• Understand GMS reporting system and
• how to do performance reporting on GMS

4
The Federal Climate

• Governmentwide move toward accountability
• Government Performance and Results Act
• Presidents Management Agenda
• Office of Management and Budget
• Program Assessment Rating Tool
• Paperwork Reduction Act of 1995

Programs need to show effectiveness to justify
funding.
5
Funding and Information Flows
Congress

OJJDP
Information
State
Subgrantee
6
Information Example
Congress
OJJDP
State
State
Subgrantee
Subgrantee
Subgrantee
Subgrantee
7
Development Methodology

• 1. Categorize Formula Grants Program Areas
• System Improvement
• Core Requirements
• Prevention
• Intervention
• 2. Develop logic models.
• 3. Define common indicators.
• 4. Apply prevention and selected intervention
  models to Title V
• Analysis possible by or across Program Area

8
Sources for Indicator Development

• 1. Federal guidelines and best practices
• 2. Literature from the field
• 3. Examples from other agencies at the
  Federal, State, and local levels
• JJ Specialist/Title V Coordinator input
• Written feedback
• Conference calls

9
Planning for Performance Measurement Reviewing
the Logic Models and Program Areas
Presenters Marcia Cohen, DSG, Inc. Heidi Hsia,
OJJDP
10
What Is Performance Measurement?
It is directly related to program goals and
objectives. It measures progress
quantitatively. It is not exhaustive. It provides
a temperature readingit may not tell you
everything you want to know but provides a quick
and reliable gauge of selected results.
11
Measurement vs. Evaluation
Impact evaluations are broader and assess the
overall or net effectsintended or unintendedof
the program as a whole.
Impact evaluation
Scope
Evaluation
Outcome evaluations investigate whether the
program causes demonstrable effects on
specifically defined target outcomes.
Outcome evaluation
Process evaluations investigate the process of
delivering the program, focusing primarily on
inputs, activities, and outputs.
Process evaluation
Performance Measurement
Program Monitoring
Time

Evaluation definitions excerpt from Trochim,
William M. The Research Methods Knowledge Base,
2nd Edition. Internet WWW page, at
URL lthttp//trochim.human.cornell.edu/kb/index.ht
mgt (version current as of Aug. 02, 2000).
12
Outputs Versus Outcomes

• Outputs are products of program
  implementation/activities.
• Outcomes are benefits or changes as a result of
  the program. There are two types of outcomes
• Short-term The first benefits or changes
  experienced and the ones most closely related to
  program outputs.
• Long-term Link a programs short-term and
  long-term outcomes. Often they are changes in
  practice, policy, decision-making, or behavior.

13
Outputs Versus Outcomes (contd.)

• With regard to the Formula Grants and Title V
  performance measures, to summarize
• Outputs are at the micro level and reflect
  program-level activity.
• Outcomes are at the macro level and, when
  aggregated, will reflect Federal outcomes.
• A good performance measurement system
  should be results oriented and focus on desired
  outcomes, less on outputs.

14
Outcome Measure Definitions
Short-term Occurs during the program or by
the end of the program. Long-term Occurs 6
months to 1 year after program completion.
15
Logic Model Template
A graphic representation that clearly lays out
the logical relationships between the problem to
be addressed, program activities, outputs, and
outcomes.
Outcomes
Activities
Outputs
Problem

• Short term
• Long term

16
Overall Formula Grants and Title V Programs Logic
Model
OUTCOMES
Problems
Juvenile Delinquency
Outcome measures are for illustrative purposes
only and are not comprehensive. To see a
comprehensive list of outcomes, refer to the
individual program area logic models.
17
Title V Program Areas
Title V programs are for keeping at-risk youth
from offending or first-time, nonserious youth
out of the JJ system. Title V has 18 Prevention
and Early Intervention Program Areas 3, 4, 9,
10, 11, 12, 13, 15, 16, 18, 20, 21, 22, 25, 26,
27, 32, 34
18
Formula Grants Program Areas
Three Formula Grants Program Area categories
Prevention and Intervention Program Areas 1,
2, 3, 4, 5, 7, 9, 10, 11, 12, 13, 14, 15, 16, 18,
20, 21, 22, 24, 25, 26, 27, 29, 30, 32, 33,
34 JJ System Improvement Program Areas 10, 19,
23, 31, 33 Core Requirements Program Areas 6,
8, 10, 17, 28
19
Planning for Performance Measurement Reviewing
the Performance Measures
Presenters Gregory Thompson, OJJDP Heidi Hsia,
OJJDP Marcia Cohen, DSG, Inc.
20
Reporting Requirements

• Output All mandatory (bold) and two
  non-mandatory measures.
• Outcome All mandatory (bold) and two
  non-mandatory measures (may be either
  short- term or long-term).
• Except Core Requirements and JJ System
  Improvement require only one non-mandatory
  measure.

21
Key to Logic Models and Grid Charts

• Bold Mandatory measure
• Mandatory only for intervention programs
• Mandatory only for prevention programs
• Mandatory if applicable

22
Output Objective Categories

• Six types of output objectives
• Increase organizational/system capacity
•   Improve planning and development
• Improve program activities
• Improve program efficiency
• Improve the monitoring of compliance

23
Outcome Objective Categories

• Nine types of outcome objectives
• Reduce delinquency
• Increase accountability
• Improve system effectiveness
• Increase prosocial behaviors
• Increase program support
• Increase compliance with Core Requirements
• Reduce disproportionate minority contact
• Improve school climate
• Improve the management of the State
  Juvenile
• Justice and Delinquency Prevention
• program

24
Prevention and Intervention Program Areas
25
Mandatory Output Measures for Prevention and
Intervention Programs

• Title V or FG awarded for services
• The amount of Formula Grants and Title V funds in
  whole dollars that are awarded for delinquency
  prevention services during the reporting period.
  Program records are the preferred data source.

26
Mandatory Output Measures for Prevention and
Intervention Programs (contd)

• Number of program youths served
• An unduplicated count of the number of youths
  served by the program during the reporting
  period. Definition of the number of youth
  served for a reporting period is the number of
  program youth carried over from previous
  reporting period, plus new admissions
  during the reporting period.  In calculating the
  3-year summary, the total number of youth served
  is the number of participants carried over
  from the year previous to the first fiscal
  year, plus all new admissions during the 3
  reporting fiscal years. Program records are the
  preferred data source.

27
Mandatory Outcome Measures for Prevention
Programs

• Number and percent of program youths exhibiting
  desired change in targeted behaviors
• Substance use
• School attendance
• Antisocial behavior
• Family relationships
• Pregnancies

Note Double denotes mandatory for prevention
programs
28
Mandatory Outcome Measures for Intervention
Programs

• Number and percent of program youths who offend
  or reoffend
• The number of program youths who were rearrested
  or seen at juvenile court for a new delinquent
  offense. Appropriate for any youth-serving
  program.

Note Single denotes mandatory for
intervention programs
29
Mandatory Outcome Measures for Prevention and
Intervention Programs

• Number and percent of youth completing program
  requirements
• The number and percent of program youth who have
  successfully fulfilled all program obligations
  and requirements. Program requirements will vary
  by program but should be a predefined list of
  requirements or obligations that clients must
  meet prior to program completion. Program records
  are the preferred data source.

30
Example Probation Program Area
31
Sample Logic Model PA 24Probation
PROBLEM
Delinquency
Goals
To improve JJ systems by increasing compliance
with the Core Requirements and increasing the
availability and types of prevention and
intervention programs
Performance measures should report on those
activities funded by Title II (Formula Grants)
funds.
Outcome Measure Definitions Short TermOccurs
during the program or by the end of the
program Long TermOccurs 6 months to 1 year after
program completion
32
Sample Grid Chart PA 24Probation Output
Performance Measures
PROGRAM AREA 24. Probation OUTPUT PERFORMANCE
MEASURES
33
Sample Grid Chart PA 24Probation Outcome
Performance Measures
34
Example Delinquency Prevention Program Area
35
(No Transcript)
36
Sample Grid Chart PA 09Delinquency
Prevention Output Performance Measures
37
Sample Grid Chart PA 09Delinquency
Prevention Outcomes Performance Measures
38
Core Requirements Program Areas
39
Mandatory Output Measures forCore Requirements
Programs

• FG funds awarded
• The amount of Formula Grants money in whole
  dollars that is awarded for the separation
  requirement during the reporting period. Program
  records are the preferred data source.

40
Mandatory Output Measures for Core Requirements
Programs (contd)

• Number of programs implemented
• The number of programs created and/or implemented
  during the reporting period. Program records are
  the preferred data source.

41
Mandatory Outcome Measure for Core Requirements
Programs
Change in the number of violations The
change in the number of violations of the sight
and sound requirement from the previous
reporting period compared with the current
reporting period. Sight and sound compliance is
determined according to the definition in the
OJJDP Guidance Manual for Monitoring Facilities
Under the JJDP Act of 2002. The Annual Compliance
Monitoring Report is the preferred data source.
42
Example Separation Program Area
43
Sample Logic Model PA 28Separation
PROBLEM
Delinquency
Goals
To improve JJ systems by increasing compliance
with the Core Requirements and increasing the
availability and types of prevention and
intervention programs.
Performance measures should report on those
activities funded by Title II (Formula Grants)
funds.
Outcome Measure Definitions Reported Annually.
44
Sample Grid Chart PA 28Separation Output
Performance Measures
45
Sample Grid Chart PA 28Separation Outcome
Performance Measures
46
Example Compliance Monitoring Program Area
47
Sample Logic Model PA 06Compliance Monitoring
PROBLEM
Delinquency
Goals
To improve JJ systems by increasing compliance
with the Core Requirements and increasing the
availability and types of prevention and
intervention programs
Performance measures should report on those
activities funded by Title II (Formula Grants)
funds.
Outcome Measure Definitions Reported Annually.
48
Sample Grid Chart PA 06Compliance
Monitoring Output Performance Measures
49
Sample Grid Chart PA 06Compliance
Monitoring Outcome Performance Measures
50
Sample Logic ModelPA 10 Disproportionate
Minority Contact
PROBLEM
Delinquency
Goals
To improve JJ systems by increasing compliance
with the Core Requirements and increasing the
availability and types of prevention and
intervention programs
Performance Measures should report on those
activities funded by Title II (Formula Grants)
funds.
Outcome Measure Definitions Short-TermOccurs
during the program or by the end of the
program Long-TermOccurs 6 months to 1 year after
program completion

• Key
• system-level indicator
• individual-level indicator
• objectives
• Bold mandatory measure
• mandatory only if applicable (if not
  
  
  applicable,
  choose a different measure)
• mandatory for intervention programs
  only
• mandatory for prevention programs only

51
Sample Grid Chart PA 10Disproportionate
Minority Contact Output Performance Measures
52
Sample Grid Chart PA 10Disproportionate
Minority Contact Outcome Performance Measures
53
Juvenile Justice System Improvement Program Area

54
Mandatory Output Measure for JJ System
Improvement Programs

• Number of grants funded with Formula Grants
  funds
• The number of grants funded with Formula
  Grants funds during the reporting period.

55
Mandatory Output Measures for JJ System
Improvement Programs

• Number of programs implemented
• Number of program youth served
• Number of subgrants awarded
• Number of grant applications reviewed and
  commented upon

If applicable (if not applicable, select
another non-mandatory measure)
56
Mandatory Outcome Measures for JJ System
Improvement Programs

• Number/percent of youth completing program
• Number/percent of youth exhibiting desired
  changed in targeted behaviors
• Average length of time between initial appearance
  and disposition
• Number of programs funded directly in line with
  3-year Plan
• Number of Plan recommendations implemented
• Number of allegations of civil rights violations
  filed
• Number/percent of court hearings attended by an
  advocate

If applicable (if not applicable, select
another non-mandatory measure)
57
Sample Logic ModelPA 33 Youth Advocacy
PROBLEM
Delinquency
Goals
To improve JJ systems by increasing compliance
with the Core Requirements and increasing the
availability and types of prevention and
intervention programs.
Performance Measures should report on those
activities funded by Title II (Formula Grants)
funds.
Outcome Measure Definition Annual Measures
Measures of system change are conducted every 12
months.
58
Sample Grid Chart PA 33Youth Advocacy Output
Performance Measures
59
Sample Grid Chart PA 33Youth Advocacy Outcome
Performance Measures
60
Example State Advisory Group (SAG) Program
Area
61
Mandatory Output Measures for SAG
Program Area

• Number of grants funded with Formula Grants
  funds
• The number of grants funded with Formula
  Grants funds during the reporting period.

62
Mandatory Output Measures for
(SAG) Program Area

• Number of grant applications reviewed and
  commented on
• The number of grant applications reviewed and
  commented on to guide the development of
  juvenile justice programming in the State.

63
Mandatory Outcome Measure forSAG Program Area

• Number and percent of Plan recommendations
  implemented
• Number and percent of SAG recommendations for
  the State Plan implemented during the reporting
  period. The preferred data source is
• committee meeting minutes.

64
Performance measures should report on those
activities funded by Title II (Formula Grants)
funds.
Outcome Measure Definitions Reported Annually.
65
Sample Grid Chart PA 31State Advisory Group
Output Performance Measures
66
Sample Grid Chart PA 31State Advisory Group
Outcome Performance Measures
67
Formula Grants Performance Measurement
Implementation Timeline (First year)
April 1, 2005 Reporting period begins Sept. 30,
2005 Reporting period ends Dec. 30, 2005 Data
report due to OJJDP
68
Formula Grants Performance Measurement
Implementation Timeline (Second and Subsequent
Years)
Oct. 1, 2005 Reporting period begins Sept. 30,
2006 Reporting period ends Dec. 30, 2006 Data
report due to OJJDP
69
Title V Performance MeasurementImplementation
Timeline (First year)
  January 1, 2005 Reporting period begins
  Sept. 30, 2005 Reporting period ends
  Nov. 30, 2005 Data reports due to
OJJDP   March 31, 2006 FY 2005 OJJDP Report to
Congress due
70
Title V Performance MeasurementImplementation
Timeline(Second and Subsequent Years)
  October 1, 2005 Reporting period begins
  Sept. 30, 2006 Reporting period ends
  Nov. 30, 2006 Data reports due to
OJJDP   March 31, 2007 FY 2006 OJJDP Report to
Congress due
71
Performance MeasurementTools and Support

• OJJDP Web site
• www.dsgonline.com/performance_measures.htm
• Training
• Technical assistance
•