Systemic Changes Reported by Reclaiming Futures Communities

1
Systemic Changes Reported by Reclaiming Futures
Communities March 2007 Jeffrey A.
Butts Chapin Hall Center for Children University
of Chicago John Roman Justice Policy
Center Urban Institute
Any views expressed herein are those of the
authors alone and should not be attributed to the
University of Chicago, Chapin Hall Center for
Children, the Urban Institute, or their trustees
and funders..
2
Introduction Reclaiming Futures (RF) is an
initiative of the Robert Wood Johnson Foundation
(RWJF). The initiative focuses on improving
substance abuse interventions for youth in the
juvenile justice system. The first phase of
Reclaiming Futures (2002-07) was a ten-site
demonstration effort that relied on
organizational change and system reform to
improve the quality and consistency of services
for youthful offenders. As part of a national
evaluation of Reclaiming Futures, the Urban
Institute and Chapin Hall Center for Children at
the University of Chicago conducted bi-annual
surveys in each community participating in
Reclaiming Futures.
The surveys measured the quality of juvenile
justice and substance abuse treatment systems as
reported by 20 to 40 expert informants in each
community. Only a handful of informants in each
community were directly involved in the planning
and management of the initiative. Most were
simply experienced professionals in the fields of
juvenile justice and substance abuse. The pattern
of survey responses between December 2003 and
June 2006 suggests that RF may be a promising
strategy for improving interventions for youth.
Most of the quality indicators measured by the
evaluation improved significantly during the
course of the RF initiative.
The material in this presentation is adapted from
a forthcoming report Changes Reported by
Communities Participating in Reclaiming Futures
2003-2006. Reclaiming Futures National
Evaluation Report. Portland, OR Reclaiming
Futures National Program Office, Portland State
University.
3

• The RF Initiative
• The ten initial Reclaiming Futures communities
  were
• Anchorage, Alaska
• Santa Cruz, California
• Chicago, Illinois
• Southeast Kentucky
• Marquette, Michigan
• State of New Hampshire
• Dayton, Ohio
• Portland, Oregon
• Sovereign Tribal Nation of Sicangu Lakota in
  Rosebud, South Dakota
• Seattle, Washington

• A multi-disciplinary team from each Reclaiming
  Futures community worked to enhance the
  availability and quality of substance abuse
  interventions for youth involved with the
  juvenile justice system.
• Each team focused on goals unique to its own
  community, but all of the RF projects worked to
  improve the system of care for youthful offenders
  with substance abuse issues by relying on
• enhanced judicial leadership
• improved court/community collaborations
• stronger performance management
• better treatment quality, and
• close inter-agency partnerships.

4
Evaluation Methods The RF evaluation used
participant surveys to measure systemic change.
The survey approach was selected largely for
practical reasons. It was a cost-effective means
of tracking the systemic changes that were
expected to occur during a highly complex,
multi-site, and multi-faceted organizational
reform initiative. The evaluation had to be
able to monitor the process of system change in
ten unique communities with ten different
approaches to system reform. The evaluation
design assumed that system change would be
non-linear, with multiple causal links across
many different domains of system functioning.
Measuring the effects of the initiative using
client-based, administrative data in all 10 RF
communities was considered to be impossible, or
at least highly expensive and inordinately
time-consuming. Thus, the evaluation chose to
rely in part on the reports of expert informants.
Based upon these reports, the efforts of the
Reclaiming Futures communities appear to have
generated important and significant changes in
the systems of care used to intervene with young
offenders, although the exact nature of these
changes varied across the sites. The efforts of
the ten RF communities will be described in other
forthcoming reports from the RF evaluation
project.
5
Surveys Surveys in each RF community were
conducted every six months between December 2003
and June 2006. The survey respondents were the
people in each community who were identified as
best qualified to assess the overall
effectiveness of the local juvenile justice and
substance abuse treatment systems. Across the
ten RF communities, the pool of survey
respondents included as few as 20 and as many as
50 people per site. The total response rate
averaged 70 percent, with the lowest rate of 63
percent occurring in the first administration of
the survey (December 2003), and the highest rate
of 73 percent occurring in the third
administration (December 2004).
Respondents answered 58 questions about the
quality and effectiveness of the juvenile justice
and substance abuse treatment systems in their
communities. Questions were asked in the form
of brief statements, and respondents indicated
whether they strongly disagreed, disagreed, were
neutral, agreed, or strongly agreed with the
statement. Responses were scored from -10, to -5,
0, 5, and 10, ranging from strongly negative
assessments to strongly positive assessments.
Some statements were worded negatively, but all
survey answers were coded so that higher scores
indicated more positive opinions.
6
Surveys (continued) The 58 individual survey
items were compiled into 13 multi-question scales
or indices.For example
The overall effectiveness of treatment was
measured with five statements In the past three
months, 1. The substance abuse treatment needs
of youth in my community were adequately
met. 2. The mental health needs of youth in my
community were adequately met. 3. Graduated
sanctions were used effectively to support
treatment goals for youth. 4. Youth-serving
agencies in my community generally did a good
job serving youth. 5. Youth-serving agencies in
my community were usually able to provide
youth with the range of services they needed.
The extent of family involvement was measured
using four similar questions In the past three
months, 1. Family input was used to define
service and treatment goals for
justice-involved youth. 2. Youth serving agencies
in my community did a good job involving
family members in delivering drug and alcohol
treatment services for adolescents. 3. Youth
serving agencies in my community did a good
job involving family members in developing
overall treatment goals for their children
and youth. 4. Youth serving agencies in my
community did a good job involving family
members in developing treatment service plans
for their children and youth.
7

• Surveys (continued)
• Respondent scores on each index were calculated
  as the numerical average of a persons answers to
  all of the questions making up the index.
• Scores on each index were compared over time to
  assess the direction and magnitude of system
  change in each RF jurisdiction.
• The 13 survey indices measured system changes in
  3 categories
• Administration
• Collaboration, and
• Quality

8
Survey Results for Each Index
9
ADMINISTRATION INDICES
Access to Services
Items in the index ask respondents to assess how
much their ability to serve clients is limited by
logistical difficulties and resource shortages.
Questions address issues such as lack of
transportation, poor location, waiting lists and
reductions in finding.
Note The scale of the vertical axis varies from
index to index due to their differing survey
ratings, but the range of all vertical axes is
5.0 points from low to high, enabling the
analysis to compare relative changes across the
indices.
10
ADMINISTRATION INDICES
Data Sharing
Questions ask respondents to rate the degree of
difficulty their community experiences in sharing
information across youth agencies. The index
measures each communitys ability to share
information, not only willingness to share.
Questions address issues such as legal and
regulatory limitations on information sharing, as
well as technological difficulties.
Note The scale of the vertical axis varies from
index to index due to their differing survey
ratings, but the range of all vertical axes is
5.0 points from low to high, enabling the
analysis to compare relative changes across the
indices.
11
ADMINISTRATION INDICES
Systems Integration
The index measures the level of coordination
among agencies in each RF community. Items ask
whether youth agencies work to include other
community organizations in service design and
delivery, and whether they work to ensure the
consistency of treatment goals across agencies.
Note The scale of the vertical axis varies from
index to index due to their differing survey
ratings, but the range of all vertical axes is
5.0 points from low to high, enabling the
analysis to compare relative changes across the
indices.
12
ADMINISTRATION INDICES
Resource Management
The index measures respondent impressions of how
effective each community is at generating,
utilizing and sharing resources. Resources are
defined as budgets, staff, and materials.
Questions in the index address such issues as
staff distribution and training, as well as grant
writing and financial management.
Note The scale of the vertical axis varies from
index to index due to their differing survey
ratings, but the range of all vertical axes is
5.0 points from low to high, enabling the
analysis to compare relative changes across the
indices.
13
COLLABORATION INDICES
Client Information
Index measures the extent to which inter-agency
sharing of client-specific information is an
integral part of service delivery. Questions in
the index address whether information sharing
improves service delivery, the quality of
information provided, and whether agencies
provide regular feedback and client status
updates.
Note The scale of the vertical axis varies from
index to index due to their differing survey
ratings, but the range of all vertical axes is
5.0 points from low to high, enabling the
analysis to compare relative changes across the
indices.
14
COLLABORATION INDICES
Partner Involvement

Index asks survey respondents to rate their
communitys success in building and maintaining
working relationships with key stakeholders.
Questions address efforts to recruit and gain
access to key partners, the coordination of
decision-making, and building support across key
sectors in the community.

Partner Involvement was the one survey index that
did not experience a statistically significant
increase between 2003 and 2006.
15
COLLABORATION INDICES
Agency Collaboration
The index measures each communitys success in
establishing positive working relationships
across organizations and minimizing inter-agency
tensions. Questions in the index address turf
issues, mutual trust, respect and shared
priorities.
16
QUALITY INDICES
AOD Assessments
Questions ask respondents to rate their
communitys success at using appropriate
screening and assessment tools for alcohol and
other drug (AOD) problems. Questions address the
routine use of assessment, the reliability of the
information gathered through assessment, and the
use of assessment results in matching youth to
appropriate services.
17
QUALITY INDICES
Treatment Effectiveness
The index asks respondents to rate the success of
their communities in meeting the substance abuse
and mental health needs of youth, the use of
graduated sanctions, the range of services
provided, and the overall performance of
youth-serving agencies.
18
QUALITY INDICES
Targeted Treatment
The index asks respondents to assess the
availability of treatment for various target
client groups. Questions address in-patient,
out-patient, intensive out-patient and mental
health services, as well services appropriate to
clients developmental status, gender and sexual
orientation.
19
QUALITY INDICES
Cultural Integration
Questions in the index assess the extent to which
cultural barriers interfere with service
delivery. Items address the availability of
resources necessary to provide access to
non-English speaking clients, and the
incompatibility between clients and providers on
religious grounds.
20
QUALITY INDICES
Family Involvement
Questions ask respondents to assess the extent to
which family members are involved in service
planning and delivery. The index addresses the
extent of family involvement in treatment
planning for individual children, as well as the
family role in developing treatment goals for the
broader community.
21
QUALITY INDICES
Pro-social Activities
Questions in the index ask respondents to assess
the availability of recreational, cultural and
other pro-social activities in the community, and
to rate their communitys success in linking
these activities to youth in the juvenile justice
system.
22
Overall Results
To judge the overall significance of the systemic
changes associated with Reclaiming Futures, all
of the changes that occurred in the 13 indices
between the six surveys (65 inter-survey changes
overall) can be analyzed together. If more
changes are positive than would be expected by
chance alone, this may be interpreted as evidence
that Reclaiming Futures had a significant effect
on the systemic changes reported by survey
respondents. Independent sample t-tests were
conducted on successive survey pairs (i.e.,
survey 1 versus survey 2, survey 2 versus survey
3, etc.), and the t-values from those tests were
plotted from lowest to highest to determine the
direction of improvement or deterioration in each
index.
Fifty-four out of the 65 (83) t-tests were
positive, indicating that the mean score on a
particular index was higher than its predecessor.
In 32 of the 54 instances where change was
positive, the difference between the first and
second score was statistically significant
(indicated by t values greater than 1.96). A
Wilcoxon Matched-Pairs Signed-Ranks test
conducted on the changes across successive
surveys indicated that the number of positive
changes was greater than would be expected by
chance (W3.97, p lt .001). Thus, the overall
change in juvenile justice systems as reported by
respondents to the RF surveys could be described
as positive and statistically significant. (see
figure on next page)
23
Nearly half of 65 possible changes in mean index
scores between successive surveys (5
survey-to-survey changes for 13 indices) were
positive and significant
The overall pattern of responses suggests that
system performance in the RF communities improved
significantly between 2003 and 2006.
Note This figure portrays all comparisons of
mean responses to each of the 13 survey indices
from one survey to the next (survey 1 to 2,
survey 2 to 3, etc.). There were five
survey-to-survey changes for each of 13 indices,
for a total of 65 comparisons. A t-test was used
to determine whether each change was
significantly different from zero. About half of
the comparisons were positive and significant (32
of 65), and 54 of all 65 possible changes were
positive.
24
Strongest Results
If an index had a particularly high score in the
first survey, it would be harder to observe
significant improvement on that index in
subsequent surveys, simply because there would be
less room to improve numerically. Conversely, it
would be easier to observe improvements in a
relatively low-scoring index because that index
would have more room to improve. To control for
this potential bias, an alternative measure can
be used to assess general trends in the change
survey indices.
The alternative measure examines the degree to
which an index realized its potential for
improvement, or the amount of positive change
realized by a community as a proportion of what
was possible, given that a 10 is the highest
possible score for all indices. Looking at
overall change using the alternative measure, the
Treatment Effectiveness and AOD Assessment
indices are still the two strongest performers in
terms of total change realized between the first
and sixth RF surveys. The indices for Client
Information and Family Involvement also realized
more than 20 percent of total possible
improvement. (see table on next page)
25
Change in scores across all Reclaiming Futures
communities between December 2003 and June 2006,
ranked by percentage of possible improvement
realized
Percent of Possible Improvement Realized
Rank
Survey Index
Statistically Significant?
Given where it started, the AOD Assessment index
moved 28 of the way toward a perfect score of
10.
Note The total improvement possible in any index
is limited by the score in Survey 1. Indices with
high values in Survey 1 have less room to
improve, given that 10 is the maximum possible
score. Possible improvement realized was
calculated by taking the total difference in
survey scores (Survey 6 Survey 1), and dividing
by the maximum possible improvement, or 10 minus
the first survey score. Change Indices are rank
ordered from largest to smallest change, based on
percentage of possible improvement realized. A
t-test was used to determine each if change was
significantly different from zero.
26
Change Among RF Communities
If the impact of Reclaiming Futures is assessed
by the statistical significance of changes in the
survey indices between 2003 and 2006, the
strongest system change appears to have occurred
in Dayton, Ohio. Every one of the evaluations
13 indices of systemic change increased
significantly in Dayton between the first and
last survey. Other strong performers include
Portland, Oregon (12 significant increases),
Southeastern Kentucky (10 significant increases),
and the RF projects in Anchorage, Alaska and
Chicago, Illinois (9 significant increases each).
In addition to the significance of the total
change between the first and last surveys, the
linear consistency of change can also be compared
across RF communities. Change is defined as
positive, linear and consistent as long as 4 of
the 5 inter-survey changes were positive. Using
the linear criterion to examine differences among
the 10 communities, the greatest degree of change
again appears to have occurred in Dayton, as well
as Southeastern Kentucky. Other communities
reporting large linear increases included the
projects in Chicago, New Hampshire, and Alaska.
(see figure and table on next two pages)
27
Significant increases in survey indices by RF
community, 2003-2006
In Alaska, 9 of the 13 survey indices grew
significantly, and 4 of these increased in a
linear or consistent fashion.
Note Significant increases occurred when the
difference in mean index scores on the first
survey (December 2003) and last survey (June
2006) was statistically significant ( p lt .05).
An increase was considered to be linear when at
least 4 of 5 possible score changes between the
six surveys were in a positive direction.
28
None of the changes reported in Santa Cruz were
significant and linear, but scores in Santa Cruz
started out very high and thus had less room to
improve.
Marquette seemed to have its largest success in
efforts related to pro-social activities for
youth and in data sharing arrangements among
youth agencies.
Seattle reported its strongest changes in the
indices that measured treatment quality.
In Portland, the strongest changes occurred in
the data sharing and cultural integration indices.
29
Conclusions

• Respondent ratings of local juvenile justice
  systems improved significantly between 2003
  and 2006.
• Increases were statistically significant in 12
  of the 13 survey indices.
• Improvements were especially dramatic in the
  ratings for - treatment effectiveness, -
  the use of client information in support of
  treatment, - the use of screening and
  assessment tools, and - overall systems
  integration.
• These findings suggests that the coordination
  and quality of substance abuse interventions for
  youthful offenders improved during the RWJF
  Reclaiming Futures initiative.

30
Jeffrey A. Butts, Ph.D. Research Fellow Chapin
Hall Center for Children University of
Chicago 1313 East Sixtieth Street Chicago, IL
60637 [phone] jabutts_at_uchicago.edu This
presentation can be accessed at www.jbutts.com
For more information