Measuring Changes in Reclaiming Futures Communities: National Evaluation Results

1
Measuring Changes in Reclaiming Futures
Communities National Evaluation Results
Annual Meeting of the Coalition for Juvenile
Justice Washington, DC June 11, 2007 Jeffrey
A. Butts Chapin Hall Center for
Children University of Chicago
Views expressed are those of the author and
should not be attributed to the University of
Chicago, Chapin Hall Center for Children, their
trustees or funders..
2

• Reclaiming Futures
• An initiative of Robert Wood Johnson Foundation
  
• Focused on substance abuse interventions in
  juvenile justice
• Principal strategy was organizational
  change/system reform
• Evaluation challenge Tracking change in 10
  completely different service systems with 10
  different change strategies

3

• Surveys of Reclaiming Futures Communities
• National evaluation tracked change with
  bi-annual surveys
• Surveys measured quality of systems as reported
  by 20 to 40 expert informants in each
  community
• Six surveys conducted in 2003 December
  2004 June and December 2005 June and
  December 2006 June
• Response rate averaged 70 percent and ranged
  from 63 (December 2003) to 73 (December
  2004)

4

• Surveys of Reclaiming Futures Communities
• 58 questions about quality and effectiveness
• All questions were brief statements
• Respondents indicated whether they strongly
  disagreed, disagreed, were neutral, agreed,
  or strongly agreed
• Answers coded so higher scores indicate positive
  opinions
• Responses scored -10 strongly negative
  -5 somewhat negative 0 neutral
  5 somewhat positive 10 strongly positive

5
Surveys of Reclaiming Futures Communities
6
Surveys of Reclaiming Futures Communities
The overall effectiveness of treatment was
measured with five statements In the past three
months, 1. The substance abuse treatment needs
of youth in my community were adequately
met. 2. The mental health needs of youth in my
community were adequately met. 3. Graduated
sanctions were used effectively to support
treatment goals for youth. 4. Youth-serving
agencies in my community generally did a good
job serving youth. 5. Youth-serving agencies in
my community were usually able to provide
youth with the range of services they needed.
7
Surveys of Reclaiming Futures Communities
8
Survey Results for Each Index
9
ADMINISTRATION INDICES
Access to Services
10
ADMINISTRATION INDICES
Data Sharing
11
ADMINISTRATION INDICES
Systems Integration
12
ADMINISTRATION INDICES
Resource Management
13
COLLABORATION INDICES
Client Information
14
COLLABORATION INDICES
Partner Involvement
15
COLLABORATION INDICES
Agency Collaboration
16
QUALITY INDICES
AOD Assessments
17
QUALITY INDICES
Treatment Effectiveness
18
QUALITY INDICES
Targeted Treatment
19
QUALITY INDICES
Cultural Integration
20
QUALITY INDICES
Family Involvement
21
QUALITY INDICES
Pro-social Activities
22
Change in scores across all Reclaiming Futures
communities between December 2003 and June 2006,
ranked by percentage of possible improvement
realized
Percent of Possible Improvement Realized
Rank
Survey Index
Statistically Significant?
Note The total improvement possible in any index
is limited by the score in Survey 1. Indices with
high values in Survey 1 have less room to
improve, given that 10 is the maximum possible
score. Possible improvement realized was
calculated by taking the total difference in
survey scores (Survey 6 Survey 1), and dividing
by the maximum possible improvement, or 10 minus
the first survey score. Change Indices are rank
ordered from largest to smallest change, based on
percentage of possible improvement realized. A
t-test was used to determine each if change was
significantly different from zero.
23
Significant increases in survey indices by RF
community, 2003-2006
Note Significant increases occurred when the
difference in mean index scores on the first
survey (December 2003) and last survey (June
2006) was statistically significant ( p lt .05).
An increase was considered to be linear when at
least 4 of 5 possible score changes between the
six surveys were in a positive direction.
24
None of the changes reported in Santa Cruz were
significant and linear, but scores in Santa Cruz
started out very high and thus had less room to
improve.
25
What About Bias?

• Answers to survey questions could be measures of
  self-interest or wishful thinking
• Regression analyses were performed to check on
  this possibility
• Analyses tested the influence of several factors
  on the trajectory of changes in individual
  survey scores
• Main variable of interest Did survey
  scores change more for RF Insiders -- the 4 or
  5 people in each community who were involved
  in planning the RF initiative and who were
  closer to RWJF?

26
Most Indices Were Not Affected by Bias
27
Treatment Effectiveness
The values differ, but the trajectories are
positive for both groups, indicating growing
confidence in the effectiveness of treatment.
28
Treatment Effectiveness
29
AOD Assessment
30
A Few Indices Were Affected by Bias
31
Cultural Integration
32
Partner Involvement
33
If we cross out the indices that were possibly
affected by bias
None of the changes reported in Santa Cruz were
significant and linear, but scores in Santa Cruz
started out very high and thus had less room to
improve.
34
None of the changes reported in Santa Cruz were
significant and linear, but scores in Santa Cruz
started out very high and thus had less room to
improve.
35
None of the changes reported in Santa Cruz were
significant and linear, but scores in Santa Cruz
started out very high and thus had less room to
improve.
36
None of the changes reported in Santa Cruz were
significant and linear, but scores in Santa Cruz
started out very high and thus had less room to
improve.
37
None of the changes reported in Santa Cruz were
significant and linear, but scores in Santa Cruz
started out very high and thus had less room to
improve.
38
Conclusions

• Respondent ratings of local juvenile justice
  systems improved significantly between 2003
  and 2006.
• Increases were statistically significant in 12
  of the 13 survey indices.
• Improvements were especially dramatic in the
  ratings for - treatment effectiveness, -
  the use of client information in support of
  treatment, - the use of screening and
  assessment tools, and - overall systems
  integration.
• These findings suggests that the coordination
  and quality of substance abuse interventions for
  youthful offenders improved during the RWJF
  Reclaiming Futures initiative.

39
For more information
Jeffrey A. Butts Research Fellow Chapin Hall
Center for Children University of Chicago 1313
East Sixtieth Street Chicago, IL 60637 (773)
256-5163 jabutts_at_uchicago.edu This presentation
can be accessed at www.jeffreybutts.net